Information processing system, program, and information processing method

Abstract

Provided is an information processing system comprising: a GT data generation unit that uses simulation data, in which fish are simulated, to generate size GT data that includes an image of a fish and image size information indicating the three-dimensional size of the fish in an image space; a training execution unit that generates, by executing machine learning using a plurality of the size GT data items, a learning model that takes as input an image of a fish and that outputs image size information indicating the three-dimensional size of the fish in the image space; and a size estimation unit that acquires image size information of the fish by inputting a fish captured image, in which a fish has been captured, into the learning model generated by the training execution unit.

Description

Information processing system, program, and information processing method

[0001] The present invention relates to an information processing system, a program, and an information processing method.

[0002] Patent Document 1 describes a fish size calculation device that calculates the actual size of a fish, i.e., its length and height, from an image of the fish while it is migrating. [Prior art documents] [Patent documents] [Patent document 1] Japanese Patent No. 6694039 General disclosure

[0003] According to one embodiment of the present invention, there is provided an information processing system. The information processing system may include a GT data generation unit. The GT data generation unit may use simulation data of a fish to generate size GT data including an image of the fish and image size information indicating the three-dimensional size of the fish in image space. The information processing system may include a learning execution unit. The learning execution unit may perform machine learning using a plurality of the size GT data to generate a learning model that inputs an image of the fish and outputs image size information indicating the three-dimensional size of the fish in image space. The information processing system may include a size estimation unit. The size estimation unit may input a fish image captured by a fish into the learning model generated by the learning execution unit to obtain the image size information of the fish.

[0004] In the information processing system, the GT data generation unit may use simulation data that simulates a fish to generate posture GT data that includes an image of a fish and posture information that indicates the posture of the fish, and the learning execution unit may execute machine learning using the plurality of size GT data and the plurality of posture GT data to generate the learning model that includes a posture learning model that receives an image of a fish as an input and outputs posture information that indicates the posture of the fish, and a size learning model that receives an image of the fish and the posture information as an input and outputs image size information that indicates the three-dimensional size of the fish in image space, and the size estimation unit may input the captured image of the fish to the posture learning model, and input the posture information output from the posture learning model and the captured image of the fish to the size learning model, to acquire the image size information of the fish.

[0005] Any of the information processing systems may include a real size information generation unit that generates real size information indicating the three-dimensional size in real space for each of the plurality of fish using image size information of the plurality of fish acquired by the size estimation unit when inputting a fish image including the plurality of fish into the learning model, and a weight estimation unit that estimates a weight for each of the plurality of fish using the real size information. The real size information generation unit may determine a distance between a camera that captured the fish image and one of the plurality of fish, determine a relative distance between the plurality of fish, and generate the real size information for each of the plurality of fish using the distance between the camera and the one fish, the relative distances between the plurality of fish, and the image size information of the plurality of fish. The information processing system may include a memory unit that stores a relative distance learning model that receives an image including multiple fish as an input and outputs relative distances between the multiple fish. The actual size information generation unit may determine the distance between a camera that captured the fish image and one of the multiple fish, input the fish image to the relative distance learning model to determine the relative distances between the multiple fish, and generate the actual size information for each of the multiple fish using the distance between the camera and the one fish, the relative distances between the multiple fish, and image size information of the multiple fish 30. The actual size information generation unit may determine the distance between the camera and the one fish using stereoscopic vision. The actual size information generation unit may identify, from the multiple fish included in the fish image, a tagged fish whose actual size in real space is known, using the tag, and determine the distance between the camera and the one fish based on the actual size of the one fish. The actual size information generation unit may input the fish image containing an object whose actual size in real space and distance from the camera are known into the relative distance learning model to determine the relative distance between each of the multiple fish and the objects, and generate the actual size information for each of the multiple fish using the actual size of the object, the distance from the camera, and the relative distance.

[0006] In any of the information processing systems, the GT data generation unit may use simulation data that simulates a fish to generate distance GT data including an image of the fish captured by a virtual camera, the distance between the fish and the virtual camera, and parameters of the camera, and the learning execution unit may perform machine learning using a plurality of the distance GT data to generate a distance learning model that receives as input fish images and parameters of the camera that captured the images and outputs the distance between the fish and the camera. The information processing system may include: a real size information generation unit that generates real size information indicating the three-dimensional size of each of the plurality of fish in real space based on image size information of the plurality of fish obtained by the size estimation unit inputting fish image images containing the plurality of fish into the learning model and the distances from the camera of each of the plurality of fish output from the distance learning model by inputting the fish image images and parameters of the camera that captured the fish image images into the distance learning model; and a weight estimation unit that estimates weight for each of the plurality of fish using the real size information.

[0007] Any of the information processing systems may include a distance information acquisition unit that acquires distance information indicating the distance between the distance measuring sensor and each of the multiple fishes, measured by the distance measuring sensor shining light onto the multiple fishes; a real size information generation unit that generates real size information indicating the three-dimensional size in real space for each of the multiple fishes, using image size information of the multiple fishes acquired by the size estimation unit by inputting fish image images including the multiple fishes into the learning model and the distances to each of the multiple fishes measured by the distance measurement unit; and a weight estimation unit that estimates the weight of each of the multiple fishes using the real size information.

[0008] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the information processing system.

[0009] According to one embodiment of the present invention, there is provided an information processing method executed by a computer. The information processing method may include a GT data generation step of generating size GT data including a fish image and image size information indicating the three-dimensional size of the fish in image space using simulation data of a fish. The information processing method may include a learning execution step of performing machine learning using a plurality of the size GT data to generate a learning model that takes a fish image as input and outputs image size information indicating the three-dimensional size of the fish in image space. The information processing method may include a size estimation step of inputting a fish image captured by a fish into the learning model generated in the learning execution step to acquire the image size information of the fish.

[0010] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions.

[0011] 1 schematically illustrates an example of an information processing system 100. 1 schematically illustrates an example of a functional configuration of the information processing system 100. 1 schematically illustrates an example of an information processing system 100. 1 schematically illustrates an example of an information processing system 100. 1 schematically illustrates an example of an information processing system 100. 1 schematically illustrates an example of an information processing system 100. 1 schematically illustrates an example of a hardware configuration of a computer 1200 that functions as the information processing system 100, a learning device 102, or an estimation device 104.

[0012] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0013] Conventionally, fish size and weight have been estimated. However, conventional fish size estimations only use length or fork length and height, resulting in low accuracy. This is because, depending on the fish species, some species grow longer as they grow, while others develop body width (thickness) once they reach a certain length. Body width is extremely important for weight estimation. However, three-dimensional size estimation is extremely difficult. Depth sensors such as LiDAR (Light Detection and Ranging) are not effective, especially underwater. The information processing system 100 according to this embodiment performs underwater size estimation based on simulation, uses actual body side data for ground truth of weight estimation, and performs highly accurate weight estimation using camera vision.

[0014] 1 schematically illustrates an example of an information processing system 100. The information processing system 100 may be configured with a learning device 102 and an estimation device 104. Note that the information processing system 100 configured with the learning device 102 and the estimation device 104 is merely an example, and is not limited to this.

[0015] The information processing system 100 may be configured by two devices as illustrated in Fig. 1, or may be configured by one device. The information processing system 100 may also be configured by three or more devices.

[0016] For example, the information processing system 100 may include only the learning device 102 out of the learning device 102 and the estimation device 104, and the estimation device 104 may be an external device to the information processing system 100. The information processing system 100 may include only the estimation device 104 out of the learning device 102 and the estimation device 104, and the learning device 102 may be an external device to the information processing system 100.

[0017] The information processing system 100 may communicate with the camera 200 via a network 20. The network 20 may include the Internet. The network 20 may include a local area network (LAN). The network 20 may include a mobile communication network. The mobile communication network may conform to any of the following communication methods: the long term evolution (LTE) communication method, the fifth generation (5G) communication method, the third generation (3G) communication method, and the sixth generation (6G) communication method or later.

[0018] The information processing system 100 may be connected to the network 20 by wire. The information processing system 100 may be connected to the network 20 by wireless. The information processing system 100 may be connected to the network 20 via a wireless base station. The information processing system 100 may be connected to the network 20 via a Wi-Fi (registered trademark) access point.

[0019] The training device 102 may be connected to the network 20 by wire. The training device 102 may be connected to the network 20 by wireless. The training device 102 may be connected to the network 20 via a wireless base station. The training device 102 may be connected to the network 20 via a Wi-Fi access point.

[0020] The estimation device 104 may be connected to the network 20 by wire. The estimation device 104 may be connected to the network 20 by wireless. The estimation device 104 may be connected to the network 20 via a wireless base station. The estimation device 104 may be connected to the network 20 via a Wi-Fi access point.

[0021] The camera 200 is installed at any location where it can capture an image of the fish 30. The camera 200 captures images of, for example, a plurality of fish 30 swimming in a fish tank or the like, the fish being the subject of size and weight estimation. The camera 200 may be connected to the network 20 by wire. The camera 200 may be connected to the network 20 wirelessly. The camera 200 may be connected to the network 20 via a wireless base station. The camera 200 may be connected to the network 20 via a Wi-Fi access point. The estimation device 104 and the camera 200 may be directly connected. The estimation device 104 may have a built-in camera 200. When the estimation device 104 has a built-in camera 200, the estimation device 104 may be installed at any location where it can capture an image of the fish 30.

[0022] The information processing system 100 estimates the three-dimensional size of the fish 30 in image space from a captured image of the fish 30. The three-dimensional size of the fish 30 may be the length, height, and body width of the fish 30. The information processing system 100 estimates the three-dimensional size of the fish 30 in real space from the estimated three-dimensional size of the fish 30 in image space. The information processing system 100 estimates the weight of the fish 30 using the estimated three-dimensional size of the fish 30 in real space.

[0023] The learning device 102 may perform learning to estimate the three-dimensional size in image space of the fish 30 from a captured image of the fish 30. To perform such learning, a large amount of GT data including images of the fish 30 and the size of the fish 30 is required, but it is not easy to prepare such a large amount of GT data.

[0024] Therefore, the learning device 102 according to this embodiment may use simulation data obtained by simulating the fish 30 to automatically generate a large amount of GT data (sometimes referred to as size GT data) including an image of the fish 30 and size information (sometimes referred to as image size information) indicating the three-dimensional size of the fish 30 in image space. The learning device 102 may perform machine learning using the large amount of generated size GT data to generate a learning model that takes an image of the fish 30 as input and outputs image size information indicating the three-dimensional size of the fish 30 in image space. When an image including one fish 30 is input, the learning model can output image size information for the single fish 30, and when an image including multiple fish 30 is input, the learning model can output image size information for each of the multiple fish 30. For example, the learning device 102 performs deep learning using the large amount of size GT data to generate a neural network (sometimes referred to as a size neural network) that takes an image of the fish 30 as input and outputs image size information indicating the three-dimensional size of the fish 30 in image space. An existing network for 3D object detection may be used as the deep network used for learning.

[0025] The learning device 102 may perform learning to estimate the posture of the fish 30 (sometimes referred to as pose estimation) from an image of the fish 30. For example, the learning device 102 may use simulation data of the fish 30 to automatically generate a large amount of GT data (sometimes referred to as posture GT data) including images of the fish 30 and posture information indicating the posture of the fish 30. The learning device 102 may perform machine learning using the large amount of posture GT data generated to generate a learning model (sometimes referred to as a posture learning model) that receives images of the fish 30 as input and outputs posture information indicating the posture of the fish 30. For example, the learning device 102 generates a neural network (sometimes referred to as a posture neural network) that receives images of the fish 30 as input and outputs the posture information of the fish 30. The learning device 102 may generate the posture neural network for fish by performing transfer learning on an existing neural network for human posture estimation.

[0026] The learning device 102 may then generate a learning model (sometimes referred to as a size learning model) that receives an image of the fish 30 and posture information of the fish 30 as input and outputs image size information indicating the three-dimensional size of the fish 30 in image space. In this way, the learning device 102 may generate a learning model (sometimes referred to as a blended learning model) that includes a posture learning model and a size learning model by performing machine learning using multiple pieces of size GT data and multiple pieces of posture GT data. The learning device 102 may generate a fused network of a posture neural network and a size neural network.

[0027] The estimation device 104 may receive the learning model generated by the learning device 102 from the learning device 102. The estimation device 104 may receive the pose learning model and the size learning model from the learning device 102. The estimation device 104 may receive the blended learning model from the learning device 102.

[0028] The estimation device 104 may estimate the three-dimensional size of the fish 30 in image space from the captured image of the fish 30 using the learning model generated by the learning device 102. For example, the estimation device 104 may receive captured fish images of a plurality of fish 30 from the camera 200, input the captured fish images to a learning model that takes the images of the fish 30 as input and outputs image size information of the fish 30, and obtain the image size information of the fish 30 output from the learning model.

[0029] The estimation device 104 may input the captured image of the fish to the posture learning model, and input the posture information of the fish 30 output from the posture learning model and the captured image of the fish to the size learning model to obtain the image size information of the fish 30 output from the size learning model. The estimation device 104 may input the captured image of the fish to the blended learning model to obtain the image size information of the fish 30 output from the blended learning model.

[0030] The estimation device 104 may use the image size information of the fish 30 to generate actual size information indicating the three-dimensional size in real space of the fish 30. The estimation device 104 may use the actual size information of the fish 30 to estimate the weight of the fish 30.

[0031] 2 schematically illustrates an example of the functional configuration of the information processing system 100. The information processing system 100 includes a storage unit 110, a data acquisition unit 112, a simulation data generation unit 114, a GT data generation unit 116, a learning execution unit 118, a size estimation unit 120, an actual size information generation unit 122, a distance information acquisition unit 124, a weight estimation unit 126, and an output control unit 128. It is not essential that the information processing system 100 include all of these units.

[0032] The data acquisition unit 112 acquires various data and stores the acquired data in the storage unit 110.

[0033] The data acquisition unit 112 acquires, for example, data to be used for learning. The data acquisition unit 112 acquires, for example, simulation data obtained by simulating the fish 30. The data acquisition unit 112 may acquire simulation data obtained by simulating a swimming fish 30. The data acquisition unit 112 may acquire simulation data obtained by simulating a stationary fish 30. The data acquisition unit 112 may externally receive simulation data generated by an external device.

[0034] The data acquisition unit 112 acquires, for example, data on the target fish 30 whose size and weight are to be estimated. The data acquisition unit 112 receives, from the camera 200, a fish image of the target fish 30 captured by the camera 200. The data acquisition unit 112 may receive, from the camera 200, a fish image of the target swimming fish 30 captured by the camera 200. The data acquisition unit 112 may also receive, from the camera 200, a fish image of the target stationary fish 30 captured by the camera 200.

[0035] The simulation data generating unit 114 generates simulation data that simulates the fish 30. The simulation data generating unit 114 may generate simulation data that simulates a swimming fish 30. The simulation data generating unit 114 may generate simulation data that simulates a stationary fish 30.

[0036] The simulation data may be data that simulates various types of fish 30 of various sizes.

[0037] The GT data generating unit 116 generates GT data. The GT data generating unit 116 stores the generated GT data in the storage unit 110.

[0038] For example, the GT data generation unit 116 uses simulation data to generate size GT data including an image of the fish 30 and image size information indicating the three-dimensional size of the fish 30 in the image space. The GT data generation unit 116 may generate size GT data including an image of a swimming fish 30 and image size information indicating the three-dimensional size of the fish 30 in the image space. The GT data generation unit 116 may generate size GT data including an image of a stationary fish 30 and image size information indicating the three-dimensional size of the fish 30 in the image space. The GT data generation unit 116 generates a plurality of size GT data targeting various types of fish 30 of various sizes.

[0039] For example, the GT data generation unit 116 uses the simulation data to generate posture GT data including an image of the fish 30 and posture information indicating the posture of the fish 30. The GT data generation unit 116 generates a plurality of posture GT data targeting various postures of various types and sizes of fish 30.

[0040] The learning execution unit 118 performs learning to estimate the three-dimensional size of the fish 30 in image space from a captured image of the fish 30. The learning execution unit 118 may perform machine learning using the multiple size GT data generated by the GT data generation unit 116 to generate a learning model that takes an image of the fish 30 as input and outputs image size information indicating the three-dimensional size of the fish 30 in image space. When an image including one fish 30 is input, the learning model can output image size information for the single fish 30, and when an image including multiple fish 30 is input, the learning model can output image size information for each of the multiple fish 30. The learning execution unit 118 may perform deep learning using the multiple size GT data to generate a size neural network that takes an image of the fish 30 as input and outputs image size information indicating the three-dimensional size of the fish 30 in image space. The learning execution unit 118 may perform deep learning using, for example, an existing network for 3D object detection.

[0041] The learning execution unit 118 may perform learning to estimate the posture of the fish 30 from an image of the fish 30. The learning execution unit 118 may perform machine learning using the multiple posture GT data generated by the GT data generation unit 116 to generate a posture learning model that receives an image of the fish 30 as an input and outputs posture information indicating the posture of the fish 30. When an image including one fish 30 is input, the posture learning model can output posture information for the single fish 30, and when an image including multiple fish 30 is input, the posture learning model can output posture information for each of the multiple fish 30. The learning execution unit 118 may perform deep learning using the multiple posture GT data to generate a posture neural network that receives an image of the fish 30 as an input and outputs posture information for the fish 30. The learning execution unit 118 may generate the posture neural network for fish by performing transfer learning on an existing neural network for human posture estimation.

[0042] The learning execution unit 118 may generate a size learning model that receives an image of the fish 30 and posture information of the fish 30 as input and outputs image size information indicating the three-dimensional size of the fish 30 in image space. In this way, the learning execution unit 118 may generate a blended learning model including the posture learning model and the size learning model by performing machine learning using multiple pieces of size GT data and multiple pieces of posture GT data. For example, the learning execution unit 118 generates a Fused network that blends a posture neural network and a size neural network.

[0043] The size estimation unit 120 uses the fish image of the fish 30 acquired by the data acquisition unit 112 to estimate the three-dimensional size of the fish 30 in the image space contained in the fish image. The size estimation unit 120 may input the fish image to a learning model generated by the learning execution unit 118 and acquire image size information output from the learning model. The size estimation unit 120 stores the acquired image size information in the storage unit 110. The GT data generation unit 116 can automatically generate a large amount of size GT data, and the learning model generated using the large amount of size GT data can estimate the size of the fish 30 with high accuracy. Therefore, the size estimation unit 120 can estimate the three-dimensional size of the fish 30 in the image space with high accuracy.

[0044] The size estimation unit 120 may input the captured image of the fish into the posture learning model generated by the learning execution unit 118, and input the posture information output from the posture learning model and the captured image of the fish into the size learning model generated by the learning execution unit 118 to acquire image size information of the fish 30. Alternatively, the size estimation unit 120 may input the captured image of the fish into a blended learning model generated by the learning execution unit 118 and acquire image size information of the fish 30 output from the blended learning model. For example, in an image of a swimming fish 30, the fish 30 may assume various postures relative to the camera 200. For example, if the side of the fish 30 is facing the camera 200, it is relatively easy to estimate the size of the fish 30. However, if the front or back of the fish 30 is facing the camera 200 or if the fish 30 is captured in an image with its body bent, it is relatively difficult to estimate the size of the fish 30. On the other hand, by estimating the posture of the fish 30, the learning model can be provided with the knowledge to convert the side of the fish 30 to a state facing the camera 200, thereby improving the accuracy of size estimation.

[0045] The real size information generating unit 122 generates real size information indicating the three-dimensional size in the real world for each of the plurality of fish 30, using the image size information of the plurality of fish 30 acquired by the size estimation unit 120 when the fish captured images including the plurality of fish 30 are input into the learning model. The real size information generating unit 122 stores the generated real size information in the storage unit 110.

[0046] For example, the real size information generation unit 122 identifies the distance between one of the multiple fish 30 and the camera 200 that captured the fish image. The real size information generation unit 122 inputs an image including the multiple fish 30 into a relative distance learning model that outputs the relative distances between the multiple fish 30, and identifies the relative distances between the multiple fish 30. The real size information generation unit 122 generates real size information for each of the multiple fish 30 using the identified distance between the camera 200 and the single fish 30, the identified relative distances between the multiple fish 30, and image size information of the multiple fish 30. As a specific example, the real size information generation unit 122 calculates the distance between the camera 200 and each of the multiple fish 30 using the distance between the camera 200 and the single fish 30 and the relative distances between the multiple fish 30. The actual size information generating unit 122 generates actual size information of the multiple fishes 30 using the distance between the camera 200 and each of the multiple fishes 30 and the image size information of the multiple fishes 30 .

[0047] The relative distance learning model may be acquired in advance from an external source by the data acquisition unit 112 and stored in the storage unit 110. The relative distance learning model can be generated by performing machine learning using GT data including an image containing multiple fish 30 and distances from a camera that captured the multiple fish 30. Because the brightness of the fish 30 included in the image varies depending on the distance from the camera, it is possible to generate a learning model that estimates the relative distances between multiple fish 30 from the image using machine learning. The relative distance learning model may be generated by the learning execution unit 118. For example, the GT data generation unit 116 uses simulation data to generate multiple pieces of GT data including images containing multiple fish 30 and distances from a virtual camera. The learning execution unit 118 generates the relative distance learning model by performing machine learning using the multiple GT data generated by the GT data generation unit 116.

[0048] The real size information generation unit 122 may, for example, use stereoscopic vision to determine the distance between the camera 200 and one fish 30. The real size information generation unit 122 may identify one of the multiple fish 30 included in the fish captured image, which is tagged and whose real size in real space is known, by using the tag, and determine the distance between the camera 200 and the single fish 30 based on the real size of the single fish 30. The real size information generation unit 122 may input the fish captured image including an object whose real size in real space and distance from the camera 200 are known into a relative distance learning model to determine the relative distances between each of the multiple fishes and the objects, and generate real size information for each of the multiple fish 30 using the object's real size, its distance from the camera 200, and the relative distance.

[0049] The information processing system 100 may generate actual size information of the multiple fish 30 from fish-captured images containing the multiple fish 30 by performing learning including camera parameters. Examples of camera parameters used by the information processing system 100 include, but are not limited to, lens focal length, lens distortion, brightness, and projection. For example, the GT data generation unit 116 uses the simulation data generated by the simulation data generation unit 114 to generate GT data (sometimes referred to as distance GT data) including images of the fish 30 captured by a virtual camera, the distance between the fish 30 and the virtual camera, and camera parameters. The learning execution unit 118 performs machine learning using the multiple distance GT data to generate a learning model (sometimes referred to as distance learning model) that receives images of the fish 30 and parameters of the camera that captured the images as inputs and outputs the distance between the fish 30 and the camera. Then, the actual size information generation unit 122 generates actual size information for each of the multiple fish 30 based on the image size information of the multiple fish 30 obtained by the size estimation unit 120 by inputting the fish image containing the multiple fish 30 into the learning model, and the distance from the camera to each of the multiple fish 30 output from the distance learning model by inputting the fish image and parameters of the camera that captured the fish image into the distance learning model.

[0050] The distance information acquisition unit 124 acquires distance information indicating the distance between the distance measurement sensor and each of the multiple fishes 30, measured by the distance measurement sensor irradiating light onto the multiple fishes 30. The actual size information generation unit 122 may generate actual size information for each of the multiple fishes 30 using the image size information of the multiple fishes 30 acquired by the size estimation unit 120 by inputting fish images including the multiple fishes 30 into a learning model, and the distance information acquired by the distance information acquisition unit 124.

[0051] The weight estimation unit 126 estimates the weight of each of the plurality of fish 30 using the actual size information generated by the actual size information generation unit 122. The weight estimation unit 126 stores the estimated weight in the memory unit 110.

[0052] For example, the weight estimation unit 126 uses actual size-weight correspondence data that associates the actual size and weight of the fish 30 that have been actually measured. The weight estimation unit 126 refers to the actual size-weight correspondence data based on the actual size indicated by the actual size information generated by the actual size information generation unit 122, and sets the weight corresponding to the actual size as the estimated weight. The actual size-weight correspondence data may be acquired in advance from an external source by the data acquisition unit 112 and stored in the storage unit 110.

[0053] For example, the weight estimation unit 126 uses a weight learning model that receives actual size information as input and outputs weight. The weight estimation unit 126 inputs the actual size information generated by the actual size information generation unit 122 into the weight learning model, and uses the weight output from the weight learning model as the weight of the estimated result. The weight learning model may be acquired in advance from an external source by the data acquisition unit 112 and stored in the storage unit 110. The weight learning model may be generated by the learning execution unit 118. For example, the data acquisition unit 112 acquires GT data (sometimes referred to as size-weight GT data) including the actual size and weight of the fish 30, and the learning execution unit 118 performs machine learning using multiple pieces of size-weight GT data to generate the weight learning model.

[0054] The output control unit 128 controls the output of data stored in the storage unit 110. For example, the output control unit 128 displays the data on a display provided in the information processing system 100. For example, the output control unit 128 transmits and outputs the data to the outside.

[0055] The output control unit 128 may perform control to output the image size information estimated by the size estimation unit 120. The output control unit 128 may perform control to output the actual size information generated by the actual size information generation unit 122. The output control unit 128 may perform control to output the weight estimated by the weight estimation unit 126.

[0056] When the information processing system 100 is configured by a single device, the information processing system 100 may include a memory unit 110, a data acquisition unit 112, a simulation data generation unit 114, a GT data generation unit 116, a learning execution unit 118, a size estimation unit 120, an actual size information generation unit 122, a distance information acquisition unit 124, a weight estimation unit 126, and an output control unit 128. When the information processing system 100 is configured by a learning device 102 and an estimation device 104, the learning device 102 may include the memory unit 110, the data acquisition unit 112, the simulation data generation unit 114, the GT data generation unit 116, and the learning execution unit 118, and the estimation device 104 may include the memory unit 110, the size estimation unit 120, the actual size information generation unit 122, the distance information acquisition unit 124, the weight estimation unit 126, and the output control unit 128. The memory unit 110 may be shared by the learning device 102 and the estimation device 104. The learning device 102 and the estimation device 104 may each have the memory unit 110. When the information processing system 100 is configured by three or more devices, the memory unit 110, the data acquisition unit 112, the simulation data generation unit 114, the GT data generation unit 116, the learning execution unit 118, the size estimation unit 120, the actual size information generation unit 122, the distance information acquisition unit 124, the weight estimation unit 126, and the output control unit 128 may be appropriately distributed among the three or more devices.

[0057] 3 is a schematic diagram of an example of the information processing system 100. In the example shown in FIG. 3, the information processing system 100 is capable of communicating with a stereo camera 210. The stereo camera 210 may be an example of the camera 200.

[0058] In the information processing system 100 illustrated in Fig. 3, the actual size information generation unit 122 may determine the distance between the stereo camera 210 and one fish 30 by using stereo vision with the stereo camera 210. Note that in the information processing system 100 illustrated in Fig. 3, the actual size information generation unit 122 may determine the distance between the stereo camera 210 and each of the target fish 30 by using stereo vision with the stereo camera 210. In this case, the actual size information generation unit 122 may generate actual size information for each of the multiple fish 30 from image size information of the multiple fish 30 acquired by the size estimation unit 120 inputting fish images of the multiple fish 30 captured by the stereo camera 210 into a learning model, and the determined distance between the stereo camera 210 and each of the multiple fish 30.

[0059] Fig. 4 schematically illustrates an example of the information processing system 100. In the example illustrated in Fig. 4, a tag 32 is attached to at least one of a plurality of fish 30. The actual size in real space of the fish 30 to which the tag 32 is attached is known.

[0060] In the information processing system 100 illustrated in Figure 4, the actual size information generation unit 122 may identify one of the multiple fish 30 included in the fish image, which has a tag 32 attached and whose actual size is known, using the tag 32, and may determine the distance between the camera 200 and the one fish 30 based on the actual size of the one fish 30.

[0061] Fig. 5 schematically illustrates an example of the information processing system 100. In the example illustrated in Fig. 5, an object 34 is placed in a location where a fish 30 is swimming. The object 34 has a known actual size and is placed at a position whose distance from the camera 200 is known. For example, if the fish 30 is swimming in a fish pen, the object 34 is placed in the fish pen. In Fig. 5, the shape of the object 34 is a cube, but the shape of the object 34 is not limited to this and may be another shape.

[0062] 5 , the actual size information generation unit 122 may input a fish image including a plurality of fish 30 and an object 34 into a relative distance learning model to determine the relative distances between the plurality of fish 30 and the object 34. The actual size information generation unit 122 may generate actual size information for each of the plurality of fish 30 using the actual size of the object 34, the distance from the camera 200, and the determined relative distances.

[0063] Fig. 6 schematically illustrates an example of an information processing system 100. In the example illustrated in Fig. 6, the information processing system 100 is capable of communicating with a distance measurement sensor 300. The distance measurement sensor 300 is capable of measuring the distance between the distance measurement sensor 300 and the fish 30 by irradiating light onto the fish 30 and receiving light reflected from the fish 30. The distance measurement sensor 300 is disposed at a position corresponding to the camera 200. The memory unit 110 stores information regarding the positional relationship between the distance measurement sensor 300 and the camera 200.

[0064] 6 , the distance information acquisition unit 124 may acquire distance information measured by the distance measurement sensor 300, indicating the distance between the distance measurement sensor 300 and each of the multiple fishes 30. The actual size information generation unit 122 may calculate the distance between the camera 200 and each of the multiple fishes 30 based on the distance information acquired by the distance information acquisition unit 124 and the positional relationship between the camera 200 and the distance measurement sensor 300. The actual size information generation unit 122 may generate actual size information for each of the multiple fishes 30 using image size information of the multiple fishes 30 acquired by the size estimation unit 120 by inputting fish images including the multiple fishes 30 into a learning model, and the distance between the camera 200 and each of the multiple fishes 30.

[0065] 7 schematically illustrates an example of the hardware configuration of a computer 1200 functioning as the information processing system 100, the learning device 102, or the estimation device 104. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or to perform operations associated with the apparatus according to the present embodiment or one or more "parts," and / or to perform a process according to the present embodiment or steps of the process. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0066] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and a legacy input / output unit such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0067] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller 1216 itself, and causes the image data to be displayed on the display device 1218.

[0068] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0069] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0070] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or a method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0071] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0072] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0073] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0074] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0075] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0076] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-ray discs, memory sticks, integrated circuit cards, and the like.

[0077] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0078] The computer-readable instructions may be provided to a general-purpose computer, a special-purpose computer, or another programmable data processing device processor or programmable circuit, either locally or via a local area network (LAN), a wide area network (WAN) such as the Internet, so that the processor or programmable circuit of the programmable data processing device, such as a computer, executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Here, the computer may be a personal computer (PC), a tablet computer, a smartphone, a workstation, a server computer, a general-purpose computer, a special-purpose computer, or the like, or may be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system, and is a broad definition of computer. In a distributed computing system, multiple computers collectively execute a program by each executing a portion of the program and passing data between the computers as needed during program execution.

[0079] Examples of processors include computer processors, central processing units, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc. A computer may have one processor or multiple processors. In a multiprocessor system with multiple processors, each processor executes a portion of a program and passes data between processors as needed during program execution, allowing the multiple processors to collectively execute the program. For example, in multitasking, each of the multiple processors may execute a portion of each task in small chunks by switching tasks at each time slice. In this case, which portion of a program each processor executes changes dynamically. Which portion of a program each of the multiple processors executes may also be statically determined by multiprocessor-aware programming.

[0080] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0081] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order.

[0082] 20 Network, 30 Fish, 32 Tag, 100 Information processing system, 102 Learning device, 104 Estimation device, 110 Memory unit, 112 Data acquisition unit, 114 Simulation data generation unit, 116 GT data generation unit, 118 Learning execution unit, 120 Size estimation unit, 122 Actual size information generation unit, 124 Distance information acquisition unit, 126 Weight estimation unit, 128 Output control unit, 200 Camera, 210 Stereo camera, 300 Distance measurement sensor, 1200 Computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 Input / output controller, 1222 Communication interface, 1224 Storage device, 1230 ROM, 1240 Input / output chip

Claims

1. An information processing system comprising: a GT data generation unit that uses simulation data that simulates a fish to generate size GT (Ground Truth) data including an image of a fish and image size information that indicates the three-dimensional size of the fish in image space; a learning execution unit that executes machine learning using a plurality of the size GT data to generate a learning model that takes an image of a fish as input and outputs image size information that indicates the three-dimensional size of the fish in image space; and a size estimation unit that inputs a fish image that captures a fish into the learning model generated by the learning execution unit to obtain the image size information of the fish.

2. The information processing system of claim 1, wherein the GT data generation unit uses simulation data that simulates a fish to generate posture GT data including an image of a fish and posture information indicating the posture of the fish; the learning execution unit executes machine learning using the multiple size GT data and multiple of the posture GT data to generate the learning models including a posture learning model that receives an image of a fish as an input and outputs posture information indicating the posture of the fish, and a size learning model that receives an image of the fish and the posture information as an input and outputs image size information indicating the three-dimensional size of the fish in image space; and the size estimation unit inputs the fish image to the posture learning model and inputs the posture information output from the posture learning model and the fish image to the size learning model to obtain the image size information of the fish.

3. The information processing system of claim 1 or 2, further comprising: an actual size information generation unit that generates actual size information indicating the three-dimensional size in real space for each of the plurality of fish using image size information of the plurality of fish obtained by the size estimation unit by inputting a fish image containing the plurality of fish into the learning model; and a weight estimation unit that estimates a weight for each of the plurality of fish using the actual size information.

4. An information processing system as described in claim 3, comprising a memory unit that stores a relative distance learning model that receives an image including multiple fish as input and outputs the relative distances between the multiple fish, wherein the actual size information generation unit determines the distance between a camera that captured the fish image and one of the multiple fish, inputs the fish image into the relative distance learning model to determine the relative distances between the multiple fish, and generates the actual size information for each of the multiple fish using the distance between the camera and the one fish, the relative distances between the multiple fish, and image size information of the multiple fish.

5. The information processing system according to claim 4, wherein the actual size information generating unit determines the distance between the camera and the one fish by stereoscopic vision.

6. The information processing system of claim 4, wherein the actual size information generation unit identifies a single fish among a plurality of fish included in the fish image, which is tagged and whose actual size in real space is known, by using the tag, and determines the distance between the camera and the single fish based on the actual size of the single fish.

7. The information processing system of claim 4, wherein the actual size information generation unit inputs the fish image including an object whose actual size in real space and distance from the camera are known into the relative distance learning model to determine the relative distance between each of the multiple fish and the objects, and generates the actual size information for each of the multiple fish using the actual size of the object, the distance from the camera, and the relative distance.

8. The GT data generation unit uses simulation data that simulates a fish to generate distance GT data including an image of the fish captured by a virtual camera, a distance between the fish and the virtual camera, and parameters of the camera; the learning execution unit executes machine learning using a plurality of the distance GT data to generate a distance learning model that takes as input an image of the fish and parameters of the camera that captured the image and outputs the distance between the fish and the camera; and the information processing system comprises: a real size information generation unit that generates real size information indicating a three-dimensional size in real space for each of the plurality of fish based on image size information of the plurality of fish obtained by the size estimation unit inputting a fish image including the plurality of fish into the learning model and the distance from the camera of each of the plurality of fish output from the distance learning model by inputting the fish image and parameters of the camera that captured the fish image into the distance learning model; and a weight estimation unit that estimates a weight for each of the plurality of fish using the real size information.

9. An information processing system as described in claim 1 or 2, comprising: a distance information acquisition unit that acquires distance information indicating the distance between the distance measuring sensor and each of the multiple fishes, measured by the distance sensor shining light onto the multiple fishes; an actual size information generation unit that generates actual size information indicating the three-dimensional size in real space for each of the multiple fishes, using image size information of the multiple fishes acquired by the size estimation unit by inputting fish image images including the multiple fishes into the learning model and the distance information acquired by the distance information acquisition unit; and a weight estimation unit that estimates weight for each of the multiple fishes using the actual size information.

10. A program for causing a computer to function as the information processing system according to any one of claims 1 to 9.

11. An information processing method executed by a computer, comprising: a GT data generation step of generating size GT (Ground Truth) data including an image of a fish and image size information indicating a three-dimensional size of the fish in an image space, using simulation data obtained by simulating a fish; a learning execution step of executing machine learning using a plurality of the size GT data to generate a learning model that takes an image of a fish as an input and outputs image size information indicating the three-dimensional size of the fish in an image space; and a size estimation step of inputting a fish image obtained by capturing an image of a fish into the learning model generated in the learning execution step, and acquiring the image size information of the fish.

Images