Information processing program, information processing device, and information processing method

Abstract

To allow for accurately estimating information on a plurality of fishes located in a crawl from an image.SOLUTION: An information processing program causes a computer to execute an acquisition procedure and an estimation procedure. The acquisition procedure acquires a plurality of images captured from each of a plurality of different directions from a plurality of imaging devices which capture a crawl in which a plurality of fishes are located from each of the plurality of different directions. The estimation procedure estimates first fish position information indicative of a position of each piece of fish captured to each captured image in a crawl per captured image on the basis of the distance from an imaging device corresponding to each captured image acquired by the acquisition unit to each piece of fish, consolidates a plurality of first fish position information corresponding to positions in a prescribed range from a prescribed position in the crawl to second fish position information indicative of a position of one piece of fish, and estimates information on the plurality of fishes on the basis of the second fish position information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing program, an information processing device, and an information processing method. [Background technology]

[0002] Conventionally, various techniques for improving fish farming techniques have been known. For example, a technique is known in which predetermined features that change according to the number of fish (also called the number of tails) are extracted from images of fish in a fish cage. Then, a learning model, which is data on the relationship between the features and the number of tails created by machine learning, is compared with the extracted features to detect the number of tails from the learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6787471 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology only detects the number of fish in the fish pen, and therefore it is not necessarily possible to accurately estimate information about the multiple fish in the pen from the image. [Means for solving the problem]

[0005] An information processing program according to an embodiment causes a computer to execute the following steps: an acquisition step of acquiring multiple captured images from multiple different directions from multiple imaging devices that capture images of a fish pen in which multiple fish are located, the multiple captured images being acquired from each of the multiple different directions; and an estimation step of estimating, for each captured image, first fish position information indicating the position of each fish captured in each captured image in the fish pen based on the distance from the imaging device corresponding to each captured image acquired by the acquisition step to each fish captured in each captured image, consolidating multiple pieces of first fish position information corresponding to positions within a specified range from a specified position in the fish pen into second fish position information indicating the position of a single fish, and estimating information regarding the multiple fish based on the second fish position information. Effect of the Invention

[0006] According to one aspect of the embodiment, it is possible to accurately estimate information about a plurality of fish present in a fish pen from an image. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram for explaining that the size of a fish detected by object detection differs from the tail length required by fish farmers. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of the arrangement of a plurality of imaging devices that capture images of a fish pen from a plurality of different directions. [Figure 4] FIG. 4 is a diagram showing an example of images of a fish pen in which a plurality of fish are present, taken from a plurality of different directions, and an arrangement of virtual objects corresponding to each of the taken images. [Diagram 5] FIG. 5 is a diagram showing a state in which virtual objects of fish seen from different directions are arranged overlapping each other in the same virtual space. [Figure 6] FIG. 6 is a diagram showing a state after the position information of the virtual fish objects viewed from each direction has been aggregated. [Figure 7]FIG. 7 is a diagram showing how virtual fish objects viewed from different directions are detected by object detection according to the first modified example. [Figure 8] FIG. 8 is a diagram showing the relationship between a skeletal model of a virtual fish object and the diagonal length of the fish's tail. [Figure 9] FIG. 9 is a diagram showing an example of a method for arranging a virtual fish object according to the second modified example. [Figure 10] FIG. 10 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the information processing program, the information processing device, and the information processing method according to the present application will be described in detail with reference to the drawings. Note that the information processing program, the information processing device, and the information processing method according to the present application are not limited to the embodiments. In addition, the same parts in the following embodiments are given the same reference numerals, and duplicated explanations are omitted.

[0009] (Embodiment) 1. Introduction Conventionally, there is known an object detection technique that performs position estimation and class classification of an object captured in a still image or a video image (hereinafter also referred to as an image). In object detection, the position of the object is detected as a rectangular frame (also referred to as a bounding box).

[0010] Object detection is performed using a machine learning model (hereinafter also referred to as a trained model) that has been trained in advance to estimate the position and class of an object captured in an image from an image. For example, an image of a school of fish swimming in a fish tank is input as input information of the trained model, and information on the position and number of fish belonging to the school of fish captured in the captured image is obtained as output information of the trained model.

[0011] FIG. 1 is a diagram for explaining that the size of a fish detected by object detection is different from the tail length required by fish farmers. The left side of FIG. 1 shows a captured image 1 of a school of fish swimming in a fish pen. The captured image 1 also includes a fish 4 detected by object detection and a bounding box 2 surrounding the fish 4, and a fish 3 and a bounding box 5 surrounding the fish 3. The right side of FIG. 1 shows the relationship between the total length, tail length, and body height of the fish. The information required by actual fish farmers is how many fish of shipping size are there. The shipping size is determined by the tail length. Fish farmers estimate the weight of the fish based on the tail length.

[0012] Here, the size of an object detected by object detection is the size of a bounding box. In the example shown in FIG. 1, a fish 4 detected from a captured image 1 is close to its shape when viewed from the side, but its head is toward the back of the image. Therefore, the length of one side of a bounding box 2 is different from the size of the fish 4 when viewed from the side (e.g., the total length of the fish). Also, a fish 3 detected from a captured image 1 has its head facing forward. Therefore, the length of one side of a bounding box 5 is different from the size of the fish 3 when viewed from the side (e.g., the total length of the fish).

[0013] As described above, what is detected using conventional object detection techniques is the size of a bounding box, which is different from the tail length required by fish farmers.

[0014] In contrast, the information processing device according to the present embodiment uses multiple imaging devices to obtain multiple captured images of multiple fish in the water of the fish tank from multiple different directions. Then, the information processing device estimates the distance between each fish captured in the captured image and the imaging device (hereinafter also referred to as the camera-fish distance). Then, the information processing device places virtual objects corresponding to each fish at positions corresponding to the camera-fish distance in a virtual space corresponding to the fish tank (hereinafter also referred to as the fish tank space). Then, when multiple virtual objects overlap at the same position in the fish tank space, the information processing device selects one virtual object from the multiple virtual objects and deletes virtual objects other than the selected virtual object, thereby consolidating position information of the virtual objects. Then, the information processing device adjusts parameters of the position, size, orientation, and number of tails of each of the multiple virtual objects so as to reduce an error between a virtual image of the fish tank space corresponding to each of the multiple captured images and a captured image corresponding to the virtual image. This allows the information processing device to accurately estimate the position, size, orientation, and number of fish in the fish tank from an image of the fish tank in which the fish are present. Therefore, the information processing device can accurately estimate information about the fish in the fish tank from the image.

[0015] In addition, there are various dimensions that express the size of a fish, such as the total length of the fish, standard body length (body length), fork length, body height, body width, etc. In this specification, when the term "size of a fish" is used, it is intended to include the concept of the size of a fish measured by any dimension, such as the total length of the fish, standard body length (body length), fork length, body height, body width, etc.

[0016] In the following, the case where the image is a moving image will be described.

[0017] 2. Configuration of Information Processing Device 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, an input unit 130, an output unit 140, and a control unit 150.

[0018] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network by wire or wirelessly, and transmits and receives information to and from, for example, an imaging device.

[0019] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various programs (an example of an information processing program). The storage unit 120 also stores camera identification information for identifying each imaging device in association with camera information regarding each imaging device. For example, the storage unit 120 stores, as the camera information, information regarding the three-dimensional position coordinates of the space of the fish pen included in the imaging range of each imaging device, and information indicating the inter-camera distance, focal length, and parallax of each imaging device which is a stereo camera.

[0020] (Input unit 130) Various operations are input by the user to the input unit 130. For example, the input unit 130 may receive various operations from the user via a display surface (e.g., the output unit 140) using a touch panel function. The input unit 130 may also receive various operations from buttons provided on the information processing device 100, or a keyboard or mouse connected to the information processing device 100.

[0021] (Output unit 140) The output unit 140 is a display screen realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. The output unit 140 displays various information according to the control of the control unit 150. When a touch panel is adopted in the information processing device 100, the input unit 130 and the output unit 140 are integrated. In the following description, the output unit 140 may be referred to as a screen.

[0022] (Control unit 150) The control unit 150 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 150 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0023] The control unit 150 has an acquisition unit 151, an estimation unit 152, and an output control unit 153 as functional units, and may realize or execute the information processing action described below. Note that the internal configuration of the control unit 150 is not limited to the configuration shown in FIG. 2, and may be other configurations as long as they perform the information processing described below. Also, each functional unit indicates a function of the control unit 150, and does not necessarily have to be physically distinct.

[0024] (Acquisition part 151) The acquisition unit 151 acquires a plurality of captured images captured from a plurality of different directions from a plurality of imaging devices that capture images of a fish tank in which a plurality of fish are present from a plurality of different directions. For example, the acquisition unit 151 acquires the plurality of captured images from each of the plurality of imaging devices via the communication unit 110.

[0025] FIG. 3 is a diagram showing an example of the arrangement of a plurality of imaging devices that capture images of a fish cage from a plurality of different directions. In FIG. 3, the fish cage 10 has a box-like shape with the top side removed, and is placed in seawater, for example. In FIG. 3, the length of one side of the fish cage is 3 m. Also, imaging devices are placed on each side of the fish cage 10 according to the depth from the water surface. For example, imaging devices 11, 21, 31, and 41 are placed on each side at a depth of 1 m from the water surface. Also, imaging devices 12, 22, 32, and 42 are placed on each side at a depth of 2 m from the water surface. Also, imaging devices 13, 23, 33, and 43 are placed on each side at a depth of 3 m from the water surface. In FIG. 3, each of the imaging devices 11 to 13, 21 to 23, 31 to 33, and 41 to 43 is a stereo camera composed of two cameras.

[0026] As shown in FIG. 3, the position of each imaging device relative to the fish cage 10 is known in advance. The angle of view of each imaging device is also known in advance. The imaging range of each imaging device is also known in advance based on the position and angle of view of each imaging device. The acquisition unit 151 identifies information on the three-dimensional position coordinates of the space of the fish cage included in the imaging range of each imaging device based on the information on the imaging range of each imaging device. The storage unit 120 stores the information on the three-dimensional position coordinates of the space of the fish cage included in the imaging range of each imaging device in association with the camera identification information.

[0027] (Estimation part 152) The estimation unit 152 estimates the distance from the imaging device corresponding to each captured image acquired by the acquisition unit 151 to each fish captured in each captured image (hereinafter, may be referred to as the camera-fish distance). Specifically, the estimation unit 152 refers to the storage unit 120 and acquires information indicating the inter-camera distance, focal length, and parallax of the imaging device that is a stereo camera from the camera information associated with the camera identification information acquired by the acquisition unit 151. Next, the estimation unit 152 estimates the camera-fish distance based on the acquired information indicating the inter-camera distance, focal length, and parallax and each captured image.

[0028] Next, the estimation unit 152 refers to the storage unit 120 and acquires information on the three-dimensional position coordinates of the space of the fish tank included in the imaging range of each imaging device from the camera information associated with the camera identification information acquired by the acquisition unit 151. Next, the estimation unit 152 estimates first fish position information indicating the position of each fish captured in each captured image in the fish tank based on the estimated camera-fish distance and the information on the three-dimensional position coordinates of the space of the fish tank included in the imaging range of each imaging device for each captured image. Next, the estimation unit 152 places virtual objects corresponding to each fish at each position corresponding to the first fish position information in the virtual space corresponding to the fish tank (hereinafter also referred to as the fish tank space). The estimation unit 152 places virtual objects at each position in the fish tank space corresponding to the first fish position information estimated for each captured image.

[0029] Furthermore, after performing object detection for each captured image, the estimation unit 152 tracks each detected fish to determine the direction of each fish (the direction of the head of each fish).

[0030] Fig. 4 is a diagram showing an example of images of a fish pen in which a plurality of fish are present, taken from a plurality of different directions, and arrangement of virtual objects corresponding to each of the taken images. The upper part of Fig. 4 shows taken images 12A, 22A, 32A, and 42A taken by the imaging devices 12, 22, 32, and 42, respectively. The lower part of Fig. 4 shows virtual images 12B, 22B, 32B, and 42B showing a state in which the estimation unit 152 has placed virtual objects at positions in the fish pen space corresponding to the first fish position information estimated for each of the taken images 12A, 22A, 32A, and 42A. That is, the lower part of Figure 4 shows virtual images 12B, 22B, 32B, and 42B obtained by placing virtual objects at each position in the fish cage space when viewed from the directions corresponding to the imaging devices 12, 22, 32, and 42, and capturing images of the fish cage space using virtual cameras 12', 22', 32', and 42' corresponding to the imaging devices 12, 22, 32, and 42.

[0031] Fig. 5 is a diagram showing a state where virtual objects of fish seen from each direction are arranged overlapping on the same virtual space. Fig. 5 shows a virtual image 50 seen from a certain direction, in which virtual objects corresponding to virtual images 12B, 22B, 32B, and 42B shown in the lower part of Fig. 4 are arranged overlapping on the same virtual space. Fig. 5 shows a state where each virtual object seen from each direction is arranged overlapping on the same virtual space.

[0032] Furthermore, the estimation unit 152 aggregates a plurality of pieces of first fish position information corresponding to positions within a predetermined range from a predetermined position in the fish pen into second fish position information indicating the position of one fish. Specifically, when there is another virtual object overlapping the position of the predetermined virtual object arranged in the virtual space, the estimation unit 152 aggregates the first fish position information of the predetermined virtual object and the first fish position information of the other virtual object into second fish position information indicating the position of one fish. More specifically, when a plurality of virtual objects overlap at the same position in the virtual space, the estimation unit 152 aggregates the position information of the virtual objects by selecting one virtual object from among the plurality of virtual objects and deleting virtual objects other than the selected virtual object. For example, the estimation unit 152 compares the areas of fish regions in which each fish is captured in each captured image. Next, the estimation unit 152 aggregates the plurality of pieces of first fish position information into the second fish position information by selecting the first fish position information corresponding to the captured image in which the area of ​​the fish region is the largest from among the plurality of pieces of first fish position information as the second fish position information.

[0033] Fig. 6 is a diagram showing the state after the position information of the virtual fish objects seen from each direction is aggregated. In Fig. 6, the number of virtual objects has been reduced by the amount of aggregation compared to Fig. 5, and the virtual image is similar to a captured image of a school of fish in an actual fish pen.

[0034] In the above embodiment, the estimation unit 152 aggregates the position information of virtual objects by selecting one virtual object from among the virtual objects when the virtual objects overlap at the same position in the virtual space and deleting virtual objects other than the selected virtual object. However, the aggregation method is not limited to this. For example, when the virtual objects overlap at the same position in the virtual space, the estimation unit 152 calculates a representative value of the first fish position information of the overlapping virtual objects. For example, the estimation unit 152 calculates the average value, median value, or mode value of the first fish position information of the overlapping virtual objects as an example of the representative value. Then, the estimation unit 152 may aggregate the multiple first fish position information into the second fish position information by adopting the calculated representative value of the first fish position information as the second fish position information.

[0035] The estimation unit 152 also generates a plurality of virtual images obtained by capturing an image of a virtual space in which virtual objects corresponding to each fish are arranged at positions corresponding to the second fish position information in the virtual space corresponding to the fish pen, using a plurality of virtual cameras corresponding to each of the imaging devices. For example, the estimation unit 152 generates a plurality of virtual images obtained by capturing an image of the fish pen space in which the aggregated virtual objects shown in Fig. 6 are arranged, using a plurality of virtual cameras corresponding to each of the imaging devices. That is, the estimation unit 152 generates a virtual image corresponding to each imaging device.

[0036] Next, the estimation unit 152 changes the parameter value for each virtual object so that the error between each virtual image and each captured image corresponding to each virtual image is reduced. Specifically, the estimation unit 152 compares each virtual image with each captured image corresponding to each virtual image, and changes the size and orientation of each virtual object so that the error between each virtual image and each captured image is reduced. More specifically, the estimation unit 152 may use a genetic algorithm to determine the parameter value for each virtual object that minimizes the error between each virtual image and each captured image corresponding to each virtual image. For example, the parameter value for each virtual object is a value indicating the position of each virtual object, a value indicating the orientation of each virtual object, or a value indicating the size of each virtual object.

[0037] Note that the estimation unit 152 may determine the parameter values ​​using a machine learning model instead of using a genetic algorithm. For example, the estimation unit 152 may determine the parameter values ​​for each virtual object using a machine learning model that has been trained to output, as output information, parameter values ​​that minimize an error between each virtual image and each captured image corresponding to each virtual image when each virtual image and each captured image corresponding to each virtual image are input as input information.

[0038] Next, the estimation unit 152 estimates information about the multiple fishes based on the parameter values ​​of each virtual object after the change. For example, the information about the multiple fishes is fish position information indicating the position of each fish in the fish tank, fish orientation information indicating the orientation of each fish, fish size information indicating the size of each fish, or tail number information indicating the number of multiple fishes. In this way, the estimation unit 152 estimates information about the multiple fishes based on the second fish position information.

[0039] (Output control unit 153) The output control unit 153 outputs information about the multiple fish estimated by the estimation unit 152 to the output unit 140. For example, the output control unit 153 causes the output unit 140 to display fish size information and number of fish in the fish pen estimated by the estimation unit 152.

[0040] 3. Modifications The above-described information processing device 100 may be implemented in various different forms other than the above-described embodiment. Therefore, modifications of the embodiment will be described below.

[0041] 3-1. First Modified Example In the above-described embodiment, a case has been described in which the estimation unit 152 estimates information about a plurality of fishes based on position information of a virtual object. In the first modified example, the estimation unit 152 estimates posture information about the posture of a virtual object, and estimates information about a plurality of fishes based on the posture information. This allows the information processing device 100 to estimate, for example, the size of each fish with higher accuracy. Therefore, the information processing device 100 can improve the estimation accuracy of information about a plurality of fishes.

[0042] Specifically, the estimation unit 152 performs object detection on the aggregated virtual images corresponding to each captured image. Then, the estimation unit 152 detects each virtual object in each virtual image, and estimates posture information related to the posture of each detected virtual object based on the size of a bounding box surrounding each detected virtual object. Then, the estimation unit 152 estimates information related to the multiple fishes based on the posture information.

[0043] Fig. 7 is a diagram showing how virtual objects of fish viewed from each direction according to the first modified example are detected by object detection. Fig. 7 shows virtual images 12C, 22C, 32C, and 42C obtained by capturing images of the fish pen space after aggregation using virtual cameras 12', 22', 32', and 42' corresponding to the imaging devices 12, 22, 32, and 42. Also, in Fig. 7, the virtual images 12C, 22C, 32C, and 42C respectively include the same fish 62, 63, 64, and 65 detected by object detection, and bounding boxes 52, 53, 54, and 55 surrounding the fish 62, 63, 64, and 65.

[0044] The estimation unit 152 also estimates the posture of the virtual object that enters each bounding box. Specifically, the estimation unit 152 estimates how the skeletal model of the virtual object that enters each bounding box bends. More specifically, the estimation unit 152 estimates the angle between a pair of a joint and a bone that constitutes the skeletal model. A pair of a spherical joint and a conical bone is called a rig. That is, the estimation unit 152 estimates what angle each rig should maintain in order to enter each bounding box. In this way, the estimation unit 152 estimates the posture of the virtual object that enters each bounding box for the same virtual object detected in each virtual image. For example, the estimation unit 152 may use a genetic algorithm to estimate the posture of the virtual object that minimizes the error between posture information estimated to enter each bounding box.

[0045] Note that the estimation unit 152 may estimate the posture of a virtual object using a machine learning model instead of using a genetic algorithm. For example, the estimation unit 152 may estimate the posture of a virtual object using a machine learning model that has been trained to output, as output information, posture information of a virtual object that minimizes an error between posture information estimated to be included in each bounding box when each bounding box for the same virtual object detected in each virtual image and an image of the virtual object included in each bounding box are input as input information.

[0046] Fig. 8 is a diagram showing the relationship between a skeletal model of a virtual fish object and the tail length of the fish. Fig. 8 shows a virtual fish object 70, a skeletal model 71 of the virtual object 70, and a spherical joint and a conical bone constituting the skeletal model 71. The upper part of Fig. 8 shows a virtual object contained in a bounding box as shown in Fig. 7, with its body bent into a C shape. When estimating the posture of the virtual object, the estimation unit 152 can accurately estimate the tail length of the virtual object by straightening the bent rig of the skeletal model.

[0047] Although the above description has been given of a case in which the estimation unit 152 estimates the posture of a virtual object by using an aggregated virtual image corresponding to each captured image, the estimation unit 152 may estimate the posture of a virtual object by using an aggregated virtual image corresponding to at least one of the captured images. Specifically, the estimation unit 152 generates a second estimation model that estimates the posture of a fish virtual object by performing transfer learning on a first estimation model that estimates the posture of a person. Next, the estimation unit 152 estimates the posture of a virtual object included in the aggregated virtual image corresponding to at least one captured image by inputting the aggregated virtual image corresponding to at least one captured image as input information of the second estimation model.

[0048] [3-2. Second modified example] In the above-described embodiment, a case where the information processing device 100 can estimate the camera-to-fish distance has been described. In the second modified example, a case where the information processing device 100 cannot estimate the camera-to-fish distance will be described. Here, a case where the information processing device 100 cannot estimate the camera-to-fish distance corresponds to, for example, a case where each imaging device capturing an image of the fish pen is a monocular camera rather than a stereo camera.

[0049] FIG. 9 is a diagram showing an example of a method for arranging virtual objects of fish according to the second modified example. The left side of FIG. 9 is a diagram showing the fish cage space viewed from above. In FIG. 7, a plurality of virtual objects are arranged in a line at the same position from each of the virtual cameras in four directions. Also, in FIG. 7, the size of each virtual object is the same. In FIG. 7, the estimation unit 152 gradually moves the position of each virtual object in the fish cage space away from the position of the virtual camera, and moves each virtual object to a position where it appears to be about the same size as each fish captured in the captured image captured by the imaging device corresponding to the virtual camera. The estimation unit 152 performs the same process from four directions and arranges each virtual object at the position where it best matches when viewed from the four directions. When the numerical value indicating the degree of matching when viewed from the four directions is equal to or less than a predetermined threshold, the estimation unit 152 arranges each virtual object at the position when viewed from each direction.

[0050] [4. Effects] As described above, the information processing device 100 according to the embodiment includes an acquisition unit 151 and an estimation unit 152. The acquisition unit 151 acquires a plurality of captured images captured from a plurality of different directions from a plurality of imaging devices that capture images of a fish tank in which a plurality of fish are present, from a plurality of different directions. The estimation unit 152 estimates, for each captured image, first fish position information indicating a position in the fish tank of each fish captured in each captured image based on a distance from an imaging device corresponding to each captured image acquired by the acquisition unit 151 to each fish captured in each captured image, aggregates a plurality of pieces of first fish position information corresponding to positions within a predetermined range from a predetermined position in the fish tank into second fish position information indicating the position of one fish, and estimates information regarding the plurality of fish based on the second fish position information.

[0051] This allows the information processing device 100 to accurately estimate the position, size, orientation, and number of each of the multiple fish in the fish tank from the captured image of the fish tank in which the multiple fish are present. Therefore, the information processing device 100 can accurately estimate information about the multiple fish in the fish tank from the image.

[0052] In addition, the estimation unit 152 aggregates the multiple first fish position information into the second fish position information by selecting, from the multiple first fish position information, the first fish position information corresponding to the captured image with the largest fish area based on the area of ​​the fish area in each captured image, thereby consolidating the multiple first fish position information into the second fish position information.

[0053] Here, it is considered that the first fish position information corresponding to a captured image with a large fish area has higher accuracy than the first fish position information corresponding to a captured image with a small fish area, and thus the information processing device 100 can select the first fish position information with higher accuracy from among the multiple pieces of first fish position information as the second fish position information.

[0054] In addition, the estimation unit 152 generates a plurality of virtual images captured by a plurality of virtual cameras corresponding to a plurality of imaging devices of a virtual space corresponding to the fish pen, in which virtual objects corresponding to each fish are arranged at positions corresponding to the second fish position information, changes the values ​​of parameters related to each virtual object so as to reduce an error between each virtual image and each captured image corresponding to each virtual image, and estimates information related to the plurality of fish based on the parameter values ​​related to each virtual object after the changes.

[0055] As a result, the information processing device 100 can accurately estimate the position, size, orientation, and number of each of the multiple fish in the fish tank from an image of the fish tank in which the multiple fish are present, based on a comparison between each virtual image and each captured image corresponding to each virtual image.

[0056] Furthermore, the parameter value for each virtual object is a value indicating the position of each virtual object, a value indicating the orientation of each virtual object, or a value indicating the size of each virtual object.

[0057] This allows the information processing device 100 to accurately estimate the position, size, orientation, and number of each of the multiple fish in the fish pen from a captured image of the fish pen in which the multiple fish are present.

[0058] In addition, the estimation unit 152 performs object detection on a virtual image corresponding to at least one of the captured images, detects each virtual object in the virtual image, estimates posture information regarding the posture of each detected virtual object based on the size of a bounding box surrounding each detected virtual object, and estimates information regarding the multiple fishes based on the posture information.

[0059] This allows the information processing device 100 to estimate, for example, the size of each fish with higher accuracy. Therefore, the information processing device 100 can improve the estimation accuracy of information related to a plurality of fish.

[0060] In addition, the information regarding the multiple fish is fish position information indicating the position of each fish in the fish tank, fish orientation information indicating the direction of each fish, fish size information indicating the size of each fish, or tail number information indicating the number of multiple fish.

[0061] This enables the information processing device 100 to provide the fish farmer with information on how many fish of shipping size there are.

[0062] [5. Hardware Configuration] Moreover, the information processing device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 10, for example. Fig. 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0063] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0064] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0065] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0066] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0067] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 150. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from other devices via a predetermined communication network.

[0068] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.

[0069] [6. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0070] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0071] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0072] 100 Information processing device 110 Communications Department 120 Storage section 130 Input section 140 Output section 150 Control section 151 Acquisition Department 152 Estimation Department 153 Output control section

Claims

1. an acquisition step of acquiring a plurality of captured images from a plurality of imaging devices that capture images of a fish pen in which a plurality of fish are present from a plurality of different directions; an estimation step of estimating, for each captured image acquired by the acquisition step, first fish position information indicating a position in the fish pen of each fish captured in each captured image based on a distance from the imaging device corresponding to each captured image acquired by the acquisition step to each fish captured in each captured image, aggregating a plurality of first fish position information corresponding to positions within a predetermined range from a predetermined position in the fish pen into second fish position information indicating a position of one fish, and estimating information about the plurality of fish based on the second fish position information; The estimation procedure comprises: based on an area of ​​a fish region in each captured image in which each fish is captured, selecting, as the second fish position information, from among the plurality of first fish position information, the first fish position information corresponding to the captured image in which the area of ​​the fish region is the largest, thereby consolidating the plurality of first fish position information into the second fish position information; Information processing program.

2. An acquisition step of acquiring a plurality of captured images from a plurality of imaging devices that capture images of a fish tank in which a plurality of fish are present from each of a plurality of different directions; an estimation step of estimating, for each captured image acquired by the acquisition step, first fish position information indicating a position in the fish pen of each fish captured in each captured image based on a distance from the imaging device corresponding to each captured image acquired by the acquisition step to each fish captured in each captured image, aggregating a plurality of first fish position information corresponding to positions within a predetermined range from a predetermined position in the fish pen into second fish position information indicating a position of one fish, and estimating information about the plurality of fish based on the second fish position information; The estimation procedure comprises: generating a plurality of virtual images by capturing images of the virtual space corresponding to the fish pen, in which virtual objects corresponding to each fish are arranged at positions corresponding to the second fish position information, using a plurality of virtual cameras corresponding to each of the plurality of imaging devices; changing values ​​of parameters related to each virtual object so as to reduce errors between each virtual image and each captured image corresponding to each virtual image; and estimating information related to the plurality of fish based on the parameter values ​​related to each virtual object after the changes; Information processing program.

3. The parameter value for each virtual object is a value indicating the position of each virtual object, a value indicating the orientation of each virtual object, or a value indicating the size of each virtual object. The information processing program according to claim 2.

4. The estimation procedure comprises: performing object detection on the virtual image corresponding to at least one of the captured images, detecting each virtual object in the virtual image, estimating posture information regarding a posture of each detected virtual object based on a size of a bounding box surrounding each detected virtual object, and estimating information regarding the plurality of fish based on the posture information; The information processing program according to claim 2.

5. The information about the plurality of fish is fish position information indicating the position of each fish in the fish cage, fish orientation information indicating the orientation of each fish, fish size information indicating the size of each fish, or fish number information indicating the number of the plurality of fish.

3. The information processing program according to claim 1 or 2.

6. an acquisition unit that acquires a plurality of captured images from a plurality of imaging devices that capture images of a fish pen in which a plurality of fish are present from a plurality of different directions; an estimation unit that estimates first fish position information indicating a position in the fish pen of each fish captured in each captured image based on a distance from the imaging device corresponding to each captured image acquired by the acquisition unit to each fish captured in each captured image, aggregates a plurality of first fish position information corresponding to positions within a predetermined range from a predetermined position in the fish pen into second fish position information indicating a position of one fish, and estimates information regarding the plurality of fish based on the second fish position information, The estimation unit is based on an area of ​​a fish region in each captured image in which each fish is captured, selecting, as the second fish position information, from among the plurality of first fish position information, the first fish position information corresponding to the captured image in which the area of ​​the fish region is the largest, thereby consolidating the plurality of first fish position information into the second fish position information; Information processing device.

7. An acquisition unit that acquires a plurality of captured images from a plurality of imaging devices that capture images of a fish tank in which a plurality of fish are present from a plurality of different directions, the captured images being captured from the plurality of different directions; an estimation unit that estimates first fish position information indicating a position in the fish pen of each fish captured in each captured image based on a distance from the imaging device corresponding to each captured image acquired by the acquisition unit to each fish captured in each captured image, aggregates a plurality of first fish position information corresponding to positions within a predetermined range from a predetermined position in the fish pen into second fish position information indicating a position of one fish, and estimates information regarding the plurality of fish based on the second fish position information, The estimation unit is generating a plurality of virtual images by capturing images of the virtual space corresponding to the fish pen, in which virtual objects corresponding to each fish are arranged at positions corresponding to the second fish position information, using a plurality of virtual cameras corresponding to each of the plurality of imaging devices; changing values ​​of parameters related to each virtual object so as to reduce errors between each virtual image and each captured image corresponding to each virtual image; and estimating information related to the plurality of fish based on the parameter values ​​related to each virtual object after the changes; Information processing device.

8. An information processing method implemented by a program executed by an information processing device, comprising: an acquiring step of acquiring a plurality of captured images from a plurality of different directions from a plurality of imaging devices that capture images of a fish pen in which a plurality of fish are present, the captured images being captured from the plurality of different directions; an estimation step of estimating, for each captured image acquired by the acquisition step, first fish position information indicating a position in the fish pen of each fish captured in each captured image based on a distance from the imaging device corresponding to each captured image acquired by the acquisition step to each fish captured in each captured image, aggregating a plurality of first fish position information corresponding to positions within a predetermined range from a predetermined position in the fish pen into second fish position information indicating a position of one fish, and estimating information regarding the plurality of fish based on the second fish position information, The estimation step includes: based on an area of ​​a fish region in each captured image in which each fish is captured, selecting, as the second fish position information, from among the plurality of first fish position information, the first fish position information corresponding to the captured image in which the area of ​​the fish region is the largest, thereby consolidating the plurality of first fish position information into the second fish position information; Information processing methods.

9. An information processing method implemented by a program executed by an information processing device, comprising: an acquiring step of acquiring a plurality of captured images from a plurality of different directions from a plurality of imaging devices that capture images of a fish pen in which a plurality of fish are present, the captured images being captured from the plurality of different directions; an estimation step of estimating, for each captured image acquired by the acquisition step, first fish position information indicating a position in the fish pen of each fish captured in each captured image based on a distance from the imaging device corresponding to each captured image acquired by the acquisition step to each fish captured in each captured image, aggregating a plurality of first fish position information corresponding to positions within a predetermined range from a predetermined position in the fish pen into second fish position information indicating a position of one fish, and estimating information regarding the plurality of fish based on the second fish position information, The estimation step includes: generating a plurality of virtual images by capturing images of the virtual space corresponding to the fish pen, in which virtual objects corresponding to each fish are arranged at positions corresponding to the second fish position information, using a plurality of virtual cameras corresponding to each of the plurality of imaging devices; changing values ​​of parameters related to each virtual object so as to reduce errors between each virtual image and each captured image corresponding to each virtual image; and estimating information related to the plurality of fish based on the parameter values ​​related to each virtual object after the changes; Information processing methods.

Images