Information processing system, program, and information processing method

Abstract

SOLUTION: Provided is an information processing system comprising: a GT data generation part that generates size GT data including an image of each fish and image size information indicating a three-dimensional size of each fish in an image space, by using simulation data simulating each fish; a learning execution part that executes machine learning using multiple pieces of the size GT data to generate a learning model that takes the image of each fish as input and outputs image size information indicating the three-dimensional size of each fish in the image space; and a size estimation part that inputs fish imaging images obtained by imaging each fish into the learning model generated by the learning execution part to obtain the image size information of each fish.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, a program, and an information processing method. [Background technology]

[0002] Patent Document 1 describes a fish size calculation device that calculates the actual size of a fish, that is, the length and height of the fish, from an image captured of the migrating fish. [Prior art document] [Patent documents] [Patent Document 1] Patent No. 6694039 Summary of the Invention [Means for solving the problem]

[0003] According to an embodiment of the present invention, an information processing system is provided. The information processing system may include a GT data generation unit. The GT data generation unit may generate size GT data including an image of a fish and image size information indicating a three-dimensional size of the fish in an image space by using simulation data obtained by simulating a fish. The information processing system may include a learning execution unit. The learning execution unit may execute machine learning using a plurality of the size GT data to generate a learning model in which an image of a fish is input and image size information indicating a three-dimensional size of the fish in an image space is output. The information processing system may include a size estimation unit. The size estimation unit may input a fish image obtained by capturing an image of a fish into the learning model generated by the learning execution unit to obtain image size information of the fish.

[0004] In the information processing system, the GT data generation unit may use simulation data that simulates a fish to generate posture GT data including an image of a fish and posture information indicating the posture of the fish, and the learning execution unit may execute machine learning using the multiple size GT data and the multiple posture GT data to generate the learning model including a posture learning model that takes an image of the fish as an input and outputs posture information indicating the posture of the fish, and a size learning model that takes an image of the fish and the posture information as an input and outputs image size information indicating the three-dimensional size of the fish in an image space, and the size estimation unit may input the fish image to the posture learning model and input the posture information output from the posture learning model and the fish image to the size learning model to acquire the image size information of the fish.

[0005] Any of the information processing systems may include a real size information generating unit that generates real size information indicating a three-dimensional size in real space for each of the multiple fishes using image size information of the multiple fishes acquired by inputting a fish image including the multiple fishes into the learning model by the size estimation unit, and a weight estimating unit that estimates a weight for each of the multiple fishes using the real size information. The information processing system may include a storage unit that stores a relative distance learning model that receives an image including the multiple fishes as an input and outputs a relative distance between the multiple fishes, and the real size information generating unit may specify a distance between a camera that captures the fish image and one of the multiple fishes, input the fish image into the relative distance learning model to specify the relative distance between the multiple fishes, and generate the real size information for each of the multiple fishes using the distance between the camera and the one fish, the relative distance between the multiple fishes, and image size information of the multiple fishes 30. The real size information generating unit may specify the distance between the camera and the one fish by stereoscopic vision. The real size information generating unit may identify a fish, which is tagged and has a known real size in real space, from among the multiple fish included in the fish image, by using the tag, and may identify a distance between the camera and the single fish based on the real size of the single fish. The real size information generating unit may input the fish image including an object, the real size in real space of which and the distance from the camera of which are known, to the relative distance learning model to identify relative distances between the multiple fish and the object, and may generate the real size information of each of the multiple fish using the real size of the object, the distance from the camera to the object, and the relative distance.

[0006] In any of the information processing systems, the GT data generation unit may use simulation data of a fish to generate distance GT data including an image of the fish captured by a virtual camera, a distance between the fish and the virtual camera, and parameters of the camera, and the learning execution unit may execute machine learning using a plurality of the distance GT data to generate a distance learning model that inputs an image of the fish and parameters of the camera that captured the image and outputs the distance between the fish and the camera. The information processing system may include a real size information generation unit that generates real size information indicating a three-dimensional size in real space for each of the plurality of fishes based on image size information of the plurality of fishes acquired by inputting a fish image including the plurality of fishes into the learning model and the distances from the camera of each of the plurality of fishes output from the distance learning model by inputting the fish image and parameters of the camera that captured the fish image into the distance learning model, and a weight estimation unit that estimates weight for each of the plurality of fishes using the real size information.

[0007] Any of the information processing systems may include a distance information acquisition unit that acquires distance information indicating the distance between the distance measuring sensor and each of the multiple fishes, measured by the distance measuring sensor shining light onto the multiple fishes, a real size information generation unit that generates real size information indicating the three-dimensional size in real space for each of the multiple fishes using image size information of the multiple fishes acquired by the size estimation unit by inputting fish image images including the multiple fishes into the learning model and the distances to each of the multiple fishes measured by the distance measurement unit, and a weight estimation unit that estimates a weight for each of the multiple fishes using the real size information.

[0008] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the information processing system.

[0009] According to one embodiment of the present invention, there is provided an information processing method executed by a computer. The information processing method may include a GT data generation step of generating size GT data including an image of a fish and image size information indicating a three-dimensional size of the fish in an image space by using simulation data obtained by simulating a fish. The information processing method may include a learning execution step of generating a learning model in which an image of a fish is input and image size information indicating the three-dimensional size of the fish in an image space is output by executing machine learning using a plurality of the size GT data. The information processing method may include a size estimation step of inputting a fish image obtained by capturing an image of a fish into the learning model generated in the learning execution step to acquire image size information of the fish.

[0010] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0011] [Figure 1] 1 illustrates an example of an information processing system 100. [Diagram 2] 2 illustrates an example of a functional configuration of an information processing system 100. [Diagram 3] 1 illustrates an example of an information processing system 100. [Figure 4] 1 illustrates an example of an information processing system 100. [Diagram 5] 1 illustrates an example of an information processing system 100. [Figure 6] 1 illustrates an example of an information processing system 100. [Figure 7] An example of a hardware configuration of a computer 1200 functioning as the information processing system 100, the learning device 102, or the estimation device 104 is shown in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0013] Conventionally, fish size is estimated and weight is estimated. However, conventional fish size estimation is based only on length or fork length and height, and therefore the accuracy is low. The reason is that some fish species grow longer as they grow, while others grow wider (thicker) when they reach a certain length. Body width is very important for weight estimation. However, 3D size estimation is very difficult. Depth sensors such as LiDAR (Light Detection And Ranging) are not effective especially underwater. In the information processing system 100 according to the present embodiment, size estimation in water is performed on a simulation basis, actual body side data is used for ground truth of weight estimation, and weight estimation is performed with high accuracy from camera vision.

[0014] 1 illustrates an example of an information processing system 100. The information processing system 100 may be configured with a learning device 102 and an estimation device 104. Note that the information processing system 100 configured with the learning device 102 and the estimation device 104 is merely an example, and is not limited to this.

[0015] The information processing system 100 may be configured by two devices as illustrated in Fig. 1, or may be configured by one device. The information processing system 100 may also be configured by three or more devices.

[0016] For example, the information processing system 100 may include only the learning device 102 out of the learning device 102 and the estimation device 104, and the estimation device 104 may be an external device to the information processing system 100. The information processing system 100 may include only the estimation device 104 out of the learning device 102 and the estimation device 104, and the learning device 102 may be an external device to the information processing system 100.

[0017] The information processing system 100 may communicate with the camera 200 via a network 20. The network 20 may include the Internet. The network 20 may include a LAN (Local Area Network). The network 20 may include a mobile communication network. The mobile communication network may conform to any of the following communication methods: the Long Term Evolution (LTE) communication method, the 5th Generation (5G) communication method, the 3rd Generation (3G) communication method, and the 6th Generation (6G) communication method or later.

[0018] The information processing system 100 may be connected to the network 20 by wire. The information processing system 100 may be connected to the network 20 by wireless. The information processing system 100 may be connected to the network 20 via a wireless base station. The information processing system 100 may be connected to the network 20 via a Wi-Fi (registered trademark) access point.

[0019] The training device 102 may be connected to the network 20 by wire. The training device 102 may be connected to the network 20 by wireless. The training device 102 may be connected to the network 20 via a wireless base station. The training device 102 may be connected to the network 20 via a Wi-Fi access point.

[0020] The estimation device 104 may be connected to the network 20 by wire. The estimation device 104 may be connected to the network 20 by wireless. The estimation device 104 may be connected to the network 20 via a wireless base station. The estimation device 104 may be connected to the network 20 via a Wi-Fi access point.

[0021] The camera 200 is installed at any location where the fish 30 can be imaged. The camera 200 images, for example, a plurality of fish 30 swimming in a fish tank or the like, which are the subjects of size and weight estimation. The camera 200 may be connected to the network 20 by wire. The camera 200 may be connected to the network 20 wirelessly. The camera 200 may be connected to the network 20 via a wireless base station. The camera 200 may be connected to the network 20 via a Wi-Fi access point. The estimation device 104 and the camera 200 may be directly connected. The estimation device 104 may have the camera 200 built in. When the estimation device 104 has the camera 200 built in, the estimation device 104 may be installed at any location where the fish 30 can be imaged.

[0022] The information processing system 100 estimates the three-dimensional size of the fish 30 in the image space from a captured image of the fish 30. The three-dimensional size of the fish 30 may be the length, height, and body width of the fish 30. The information processing system 100 estimates the three-dimensional size of the fish 30 in the real space from the estimated three-dimensional size of the fish 30 in the image space. The information processing system 100 estimates the weight of the fish 30 using the estimated three-dimensional size of the fish 30 in the real space.

[0023] The learning device 102 may perform learning to estimate the three-dimensional size in the image space of the fish 30 from a captured image of the fish 30. To perform such learning, a large amount of GT data including the image of the fish 30 and the size of the fish 30 is required, but it is not easy to prepare such a large amount of GT data.

[0024] Therefore, the learning device 102 according to the present embodiment may automatically generate a large amount of GT data (may be referred to as size GT data) including an image of the fish 30 and size information (may be referred to as image size information) indicating the three-dimensional size of the fish 30 in the image space by using simulation data obtained by simulating the fish 30. The learning device 102 may generate a learning model in which an image of the fish 30 is input and image size information indicating the three-dimensional size of the fish 30 in the image space is output by executing machine learning using the large amount of generated size GT data. When an image including one fish 30 is input, the learning model can output image size information of the one fish 30, and when an image including multiple fishes 30 is input, the learning model can output image size information of each of the multiple fishes 30. For example, the learning device 102 generates a neural network (may be referred to as size neural network) in which an image of the fish 30 is input and image size information indicating the three-dimensional size of the fish 30 in the image space is output by executing deep learning using the large amount of size GT data. An existing network for 3D object detection may be used as the deep network used for learning.

[0025] The learning device 102 may perform learning to estimate the posture of the fish 30 (sometimes referred to as pose estimation) from an image of the fish 30. For example, the learning device 102 may automatically generate a large amount of GT data (sometimes referred to as posture GT data) including an image of the fish 30 and posture information indicating the posture of the fish 30, by using simulation data obtained by simulating the fish 30. The learning device 102 may perform machine learning using the large amount of posture GT data generated to generate a learning model (sometimes referred to as posture learning model) in which an image of the fish 30 is input and posture information indicating the posture of the fish 30 is output. For example, the learning device 102 generates a neural network (sometimes referred to as posture neural network) in which an image of the fish 30 is input and posture information of the fish 30 is output. The learning device 102 may generate a posture neural network for fish by performing transfer learning on an existing neural network for human posture estimation.

[0026] The learning device 102 may generate a learning model (which may be referred to as a size learning model) that receives an image of the fish 30 and posture information of the fish 30 as input and outputs image size information indicating the three-dimensional size of the fish 30 in image space. In this way, the learning device 102 may generate a learning model (which may be referred to as a mixed learning model) that includes a posture learning model and a size learning model by executing machine learning using a plurality of size GT data and a plurality of posture GT data. The learning device 102 may generate a fused network of a posture neural network and a size neural network.

[0027] The estimation device 104 may receive the learning model generated by the learning device 102 from the learning device 102. The estimation device 104 may receive the posture learning model and the size learning model from the learning device 102. The estimation device 104 may receive the blended learning model from the learning device 102.

[0028] The estimation device 104 may estimate the three-dimensional size of the fish 30 in the image space from the captured image of the fish 30, using the learning model generated by the learning device 102. For example, the estimation device 104 may receive fish captured images of a plurality of fish 30 from the camera 200, input the fish captured images to a learning model that takes the images of the fish 30 as input and outputs image size information of the fish 30, and obtain the image size information of the fish 30 output from the learning model.

[0029] The estimation device 104 may input the fish image to the posture learning model, and input the posture information of the fish 30 output from the posture learning model and the fish image to the size learning model to obtain the image size information of the fish 30 output from the size learning model. The estimation device 104 may input the fish image to the mixture learning model to obtain the image size information of the fish 30 output from the mixture learning model.

[0030] The estimation device 104 may use the image size information of the fish 30 to generate real size information indicating the three-dimensional size in real space of the fish 30. The estimation device 104 may use the real size information of the fish 30 to estimate the weight of the fish 30.

[0031] 2 shows an example of a schematic functional configuration of the information processing system 100. The information processing system 100 includes a storage unit 110, a data acquisition unit 112, a simulation data generation unit 114, a GT data generation unit 116, a learning execution unit 118, a size estimation unit 120, an actual size information generation unit 122, a distance information acquisition unit 124, a weight estimation unit 126, and an output control unit 128. It is not essential that the information processing system 100 includes all of these units.

[0032] The data acquisition unit 112 acquires various data. The data acquisition unit 112 stores the acquired data in the storage unit 110.

[0033] The data acquiring unit 112 acquires, for example, data used for learning. The data acquiring unit 112 acquires, for example, simulation data obtained by simulating the fish 30. The data acquiring unit 112 may acquire simulation data obtained by simulating a swimming fish 30. The data acquiring unit 112 may acquire simulation data obtained by simulating a stationary fish 30. The data acquiring unit 112 may receive simulation data generated by an external device from the outside.

[0034] The data acquisition unit 112 acquires, for example, data on the target fish 30 whose size and weight are to be estimated. The data acquisition unit 112 receives a fish image of the target fish 30 captured by the camera 200 from the camera 200. The data acquisition unit 112 may receive a fish image of the target swimming fish 30 captured by the camera 200 from the camera 200. The data acquisition unit 112 may receive a fish image of the target stationary fish 30 captured by the camera 200 from the camera 200.

[0035] The simulation data generating unit 114 generates simulation data that simulates the fish 30. The simulation data generating unit 114 may generate simulation data that simulates a swimming fish 30. The simulation data generating unit 114 may generate simulation data that simulates a stationary fish 30.

[0036] The simulation data may be data that simulates fish 30 of various types and sizes.

[0037] The GT data generating unit 116 generates GT data. The GT data generating unit 116 stores the generated GT data in the storage unit 110.

[0038] For example, the GT data generating unit 116 uses the simulation data to generate size GT data including an image of the fish 30 and image size information indicating the three-dimensional size of the fish 30 in the image space. The GT data generating unit 116 may generate size GT data including an image of the swimming fish 30 and image size information indicating the three-dimensional size of the fish 30 in the image space. The GT data generating unit 116 may generate size GT data including an image of the stationary fish 30 and image size information indicating the three-dimensional size of the fish 30 in the image space. The GT data generating unit 116 generates multiple size GT data targeting various types of fish 30 of various sizes.

[0039] For example, the GT data generation unit 116 uses the simulation data to generate posture GT data including an image of the fish 30 and posture information indicating the posture of the fish 30. The GT data generation unit 116 generates a plurality of posture GT data targeting various postures of various types and sizes of fish 30.

[0040] The learning execution unit 118 executes learning to estimate the three-dimensional size of the fish 30 in the image space from the captured image of the fish 30. The learning execution unit 118 may execute machine learning using the multiple size GT data generated by the GT data generation unit 116 to generate a learning model that receives an image of the fish 30 as an input and outputs image size information indicating the three-dimensional size of the fish 30 in the image space. When an image including one fish 30 is input, the learning model can output image size information of the single fish 30, and when an image including multiple fishes 30 is input, the learning model can output image size information of each of the multiple fishes 30. The learning execution unit 118 may execute deep learning using the multiple size GT data to generate a size neural network that receives an image of the fish 30 as an input and outputs image size information indicating the three-dimensional size of the fish 30 in the image space. The learning execution unit 118 may execute deep learning using an existing network for 3D object detection, for example.

[0041] The learning execution unit 118 may perform learning for estimating the posture of the fish 30 from an image of the fish 30. The learning execution unit 118 may perform machine learning using the multiple posture GT data generated by the GT data generation unit 116 to generate a posture learning model that receives an image of the fish 30 as an input and outputs posture information indicating the posture of the fish 30. When an image including one fish 30 is input, the posture learning model can output posture information of the single fish 30, and when an image including multiple fish 30 is input, the posture learning model can output posture information of each of the multiple fish 30. The learning execution unit 118 may perform deep learning using the multiple posture GT data to generate a posture neural network that receives an image of the fish 30 as an input and outputs posture information of the fish 30. The learning execution unit 118 may perform transfer learning on an existing neural network for estimating human posture to generate a posture neural network for fish.

[0042] The learning execution unit 118 may generate a size learning model that receives an image of the fish 30 and posture information of the fish 30 as input and outputs image size information indicating the three-dimensional size of the fish 30 in the image space. In this manner, the learning execution unit 118 may generate a mixed learning model including a posture learning model and a size learning model by executing machine learning using a plurality of size GT data and a plurality of posture GT data. For example, the learning execution unit 118 generates a fused network that mixes a posture neural network and a size neural network.

[0043] The size estimation unit 120 estimates the three-dimensional size of the fish 30 in the image space contained in the fish image, using the fish image of the fish 30 acquired by the data acquisition unit 112. The size estimation unit 120 may input the fish image to a learning model generated by the learning execution unit 118 and acquire image size information output from the learning model. The size estimation unit 120 stores the acquired image size information in the storage unit 110. The GT data generation unit 116 can automatically generate a large amount of size GT data, and the learning model generated using the large amount of size GT data can estimate the size of the fish 30 with high accuracy. Therefore, the size estimation unit 120 can estimate the three-dimensional size of the fish 30 in the image space with high accuracy.

[0044] The size estimation unit 120 may input the fish image to the posture learning model generated by the learning execution unit 118, and input posture information output from the posture learning model and the fish image to the size learning model generated by the learning execution unit 118 to obtain image size information of the fish 30. The size estimation unit 120 may also input the fish image to the mixed learning model generated by the learning execution unit 118, and obtain image size information of the fish 30 output from the mixed learning model. For example, the fish 30 in an image of a swimming fish 30 may take various postures with respect to the camera 200. For example, when the side of the fish 30 faces the camera 200, it is relatively easy to estimate the size of the fish 30. However, when the front or back of the fish 30 faces the camera 200 or when the fish 30 is imaged in a bent state, it is relatively difficult to estimate the size of the fish 30. In contrast, by estimating the posture of the fish 30, the learning model can be provided with knowledge to convert the side of the fish 30 to a state facing the camera 200, thereby improving the accuracy of size estimation.

[0045] The real size information generating unit 122 generates real size information indicating the three-dimensional size in the real world for each of the multiple fishes 30, using image size information of the multiple fishes 30 acquired by inputting the fish captured images including the multiple fishes 30 into the learning model by the size estimation unit 120. The real size information generating unit 122 stores the generated real size information in the storage unit 110.

[0046] For example, the real size information generating unit 122 specifies the distance between one of the multiple fishes 30 and the camera 200 that captured the fish image. The real size information generating unit 122 inputs an image including the multiple fishes 30 into a relative distance learning model that outputs the relative distances between the multiple fishes 30, and specifies the relative distances between the multiple fishes 30. The real size information generating unit 122 generates real size information for each of the multiple fishes 30 using the specified distance between the camera 200 and the single fish 30, the specified relative distances between the multiple fishes 30, and image size information of the multiple fishes 30. As a specific example, the real size information generating unit 122 calculates the distance between the camera 200 and each of the multiple fishes 30 using the distance between the camera 200 and the single fish 30 and the relative distances between the multiple fishes 30. The real size information generating unit 122 generates real size information of the multiple fishes 30 using the distance between the camera 200 and each of the multiple fishes 30 and the image size information of the multiple fishes 30.

[0047] The relative distance learning model may be acquired in advance by the data acquisition unit 112 from the outside and stored in the storage unit 110. The relative distance learning model can be generated by executing machine learning using GT data including an image including multiple fishes 30 and a distance from a camera that captures the multiple fishes 30. Since the brightness of the fishes 30 included in the image varies depending on the distance from the camera, a learning model that estimates the relative distance between the multiple fishes 30 from the image can be generated by machine learning. The relative distance learning model may be generated by the learning execution unit 118. For example, the GT data generation unit 116 generates multiple GT data including an image including multiple fishes 30 and a virtual distance from a camera using simulation data. The learning execution unit 118 generates the relative distance learning model by executing machine learning using the multiple GT data generated by the GT data generation unit 116.

[0048] The real size information generating unit 122 may, for example, use stereoscopic vision to determine the distance between the camera 200 and the single fish 30. The real size information generating unit 122 may identify a single fish 30 that is tagged and has a known real size in real space among the multiple fish 30 included in the fish captured image, by using the tag, and determine the distance between the camera 200 and the single fish 30 based on the real size of the single fish 30. The real size information generating unit 122 may input a fish captured image including an object whose real size in real space and distance from the camera 200 are known to a relative distance learning model, determine the relative distances between the multiple fishes and the objects, and generate real size information for each of the multiple fishes 30 using the real size of the object, its distance from the camera 200, and the relative distance.

[0049] The information processing system 100 may generate real size information of the multiple fishes 30 from a fish image including the multiple fishes 30 by performing learning including the camera parameters. Examples of the camera parameters used by the information processing system 100 include, but are not limited to, the focal length of the lens, the lens distortion, the brightness, and the projection. For example, the GT data generating unit 116 generates GT data (which may be referred to as distance GT data) including an image of the fish 30 captured by a virtual camera, the distance between the fish 30 and the virtual camera, and the parameters of the camera, using the simulation data generated by the simulation data generating unit 114. The learning executing unit 118 performs machine learning using the multiple distance GT data to generate a learning model (which may be referred to as distance learning model) that receives the image of the fish 30 and the parameters of the camera that captured the image as input and outputs the distance between the fish 30 and the camera. Then, the actual size information generation unit 122 generates actual size information for each of the multiple fish 30 based on the image size information of the multiple fish 30 obtained by the size estimation unit 120 by inputting a fish image including the multiple fish 30 into a learning model, and the distance from the camera to each of the multiple fish 30 output from the distance learning model by inputting the fish image and parameters of the camera that captured the fish image into the distance learning model.

[0050] The distance information acquisition unit 124 acquires distance information indicating the distance between the distance measurement sensor and each of the multiple fishes 30, measured by the distance measurement sensor irradiating light onto the multiple fishes 30. The actual size information generation unit 122 may generate actual size information for each of the multiple fishes 30 using the image size information of the multiple fishes 30 acquired by the size estimation unit 120 by inputting a fish image including the multiple fishes 30 into a learning model, and the distance information acquired by the distance information acquisition unit 124.

[0051] The weight estimation unit 126 estimates the weight of each of the multiple fish 30, using the actual size information generated by the actual size information generation unit 122. The weight estimation unit 126 stores the estimated weight in the memory unit 110.

[0052] For example, the weight estimation unit 126 uses actual size-weight correspondence data that associates the actual size and weight of the fish 30 that have been actually measured. The weight estimation unit 126 refers to the actual size-weight correspondence data based on the actual size indicated by the actual size information generated by the actual size information generation unit 122, and sets the weight corresponding to the actual size as the weight of the estimated result. The actual size-weight correspondence data may be previously acquired from an external source by the data acquisition unit 112 and stored in the storage unit 110.

[0053] For example, the weight estimation unit 126 uses a weight learning model that takes real size information as input and weight as output. The weight estimation unit 126 inputs the real size information generated by the real size information generation unit 122 to the weight learning model, and sets the weight output from the weight learning model as the weight of the estimation result. The weight learning model may be acquired in advance by the data acquisition unit 112 from the outside and stored in the storage unit 110. The weight learning model may be generated by the learning execution unit 118. For example, the data acquisition unit 112 acquires GT data (which may be referred to as size and weight GT data) including the real size and weight of the fish 30, and the learning execution unit 118 executes machine learning using multiple size and weight GT data to generate the weight learning model.

[0054] The output control unit 128 controls the output of data stored in the storage unit 110. For example, the output control unit 128 displays the data on a display included in the information processing system 100. For example, the output control unit 128 transmits and outputs the data to the outside.

[0055] The output control unit 128 may perform control to output the image size information estimated by the size estimation unit 120. The output control unit 128 may perform control to output the actual size information generated by the actual size information generation unit 122. The output control unit 128 may perform control to output the weight estimated by the weight estimation unit 126.

[0056] When the information processing system 100 is configured by one device, the information processing system 100 may include a storage unit 110, a data acquisition unit 112, a simulation data generation unit 114, a GT data generation unit 116, a learning execution unit 118, a size estimation unit 120, an actual size information generation unit 122, a distance information acquisition unit 124, a weight estimation unit 126, and an output control unit 128. When the information processing system 100 is configured by a learning device 102 and an estimation device 104, the learning device 102 may include a storage unit 110, a data acquisition unit 112, a simulation data generation unit 114, a GT data generation unit 116, and a learning execution unit 118, and the estimation device 104 may include a storage unit 110, a size estimation unit 120, an actual size information generation unit 122, a distance information acquisition unit 124, a weight estimation unit 126, and an output control unit 128. The memory unit 110 may be shared by the learning device 102 and the estimation device 104. Each of the learning device 102 and the estimation device 104 may have the memory unit 110. When the information processing system 100 is configured by three or more devices, the memory unit 110, the data acquisition unit 112, the simulation data generation unit 114, the GT data generation unit 116, the learning execution unit 118, the size estimation unit 120, the actual size information generation unit 122, the distance information acquisition unit 124, the weight estimation unit 126, and the output control unit 128 may be appropriately distributed among the three or more devices.

[0057] Fig. 3 illustrates an example of the information processing system 100. In the example illustrated in Fig. 3, the information processing system 100 is capable of communicating with a stereo camera 210. The stereo camera 210 may be an example of the camera 200.

[0058] In the information processing system 100 illustrated in Fig. 3, the real size information generating unit 122 may determine the distance between the stereo camera 210 and one fish 30 by stereo vision using the stereo camera 210. In the information processing system 100 illustrated in Fig. 3, the real size information generating unit 122 may determine the distance between the stereo camera 210 and each of the target fishes 30 by stereo vision using the stereo camera 210. In this case, the real size information generating unit 122 may generate real size information for each of the multiple fishes 30 from image size information of the multiple fishes 30 acquired by inputting fish images of the multiple fishes 30 captured by the stereo camera 210 into a learning model by the size estimation unit 120, and the determined distance between the stereo camera 210 and each of the multiple fishes 30.

[0059] Fig. 4 is a schematic diagram of an example of the information processing system 100. In the example shown in Fig. 4, a tag 32 is attached to at least one of a plurality of fish 30. The real size in real space of the fish 30 to which the tag 32 is attached is known.

[0060] In the information processing system 100 illustrated in Figure 4, the actual size information generation unit 122 may identify a single fish 30 among multiple fish 30 contained in a fish image, which has a tag 32 attached thereto and whose actual size is known, by using the tag 32, and may determine the distance between the camera 200 and the single fish 30 based on the actual size of the single fish 30.

[0061] Fig. 5 shows an example of the information processing system 100. In the example shown in Fig. 5, an object 34 is placed in a place where fish 30 are swimming. The object 34 has a known actual size and is placed at a position where the distance from the camera 200 is known. For example, when the fish 30 are swimming in a fish pen, the object 34 is placed in the fish pen. In Fig. 5, the shape of the object 34 is a cube, but the shape of the object 34 is not limited to this and may be another shape.

[0062] 5, the real size information generating unit 122 may input a fish image including a plurality of fish 30 and an object 34 to a relative distance learning model to determine the relative distances between the plurality of fish 30 and the object 34. The real size information generating unit 122 may generate real size information for each of the plurality of fish 30 using the real size of the object 34 and its distance from the camera 200, and the determined relative distance.

[0063] Fig. 6 shows a schematic diagram of an example of the information processing system 100. In the example shown in Fig. 6, the information processing system 100 is capable of communicating with a distance measurement sensor 300. The distance measurement sensor 300 is capable of measuring the distance between the fish 30 and the distance measurement sensor 300 by irradiating light onto the fish 30 and receiving reflected light from the fish 30. The distance measurement sensor 300 is disposed at a position corresponding to the camera 200. The memory unit 110 stores information on the positional relationship between the distance measurement sensor 300 and the camera 200.

[0064] 6, the distance information acquisition unit 124 may acquire distance information measured by the distance measurement sensor 300, indicating the distance between the distance measurement sensor 300 and each of the multiple fishes 30. The real size information generation unit 122 may calculate the distance between the camera 200 and each of the multiple fishes 30 from the distance information acquired by the distance information acquisition unit 124 and the positional relationship between the camera 200 and the distance measurement sensor 300. The real size information generation unit 122 may generate real size information for each of the multiple fishes 30, using image size information of the multiple fishes 30 acquired by the size estimation unit 120 by inputting a fish capture image including the multiple fishes 30 into a learning model, and the distance between the camera 200 and each of the multiple fishes 30.

[0065] 7 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the information processing system 100, the learning device 102, or the estimation device 104. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the device according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0066] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive, a solid state drive, etc. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0067] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0068] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0069] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0070] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0071] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0072] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0073] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0074] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0075] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0076] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0077] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0078] The computer-readable instructions may be provided to a general-purpose computer, a special-purpose computer, or a processor of another programmable data processing device, or a programmable circuit, locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc., so that the processor of the programmable data processing device, such as a computer, or the programmable circuit executes the computer-readable instructions to generate means for performing the operations specified in the flowchart or block diagram. Here, the computer may be a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, a general-purpose computer, or a special-purpose computer, or may be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system, and is a computer in the broad sense. In a distributed computing system, multiple computers collectively execute a program by each executing a part of the program and transferring data during program execution between the computers as necessary.

[0079] Examples of processors include computer processors, central processing units, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc. A computer may have one processor or multiple processors. In a multiprocessor system with multiple processors, each processor executes a part of a program, and the multiple processors collectively execute a program by passing data during program execution between the processors as necessary. For example, in executing multitasks, each of the multiple processors may execute a part of each task in small chunks by switching tasks for each time slice. In this case, which part of a program each processor executes changes dynamically. Which part of a program each of the multiple processors executes may be statically determined by programming that takes the multiprocessor into consideration.

[0080] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0081] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]

[0082] 20 network, 30 fish, 32 tag, 100 information processing system, 102 learning device, 104 estimation device, 110 memory unit, 112 data acquisition unit, 114 simulation data generation unit, 116 GT data generation unit, 118 learning execution unit, 120 size estimation unit, 122 real size information generation unit, 124 distance information acquisition unit, 126 weight estimation unit, 128 output control unit, 200 camera, 210 stereo camera, 300 distance measurement sensor, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1230 ROM, 1240 input / output chip

Claims

1. a GT data generating unit that generates size GT (Ground Truth) data including an image of a fish and image size information indicating a three-dimensional size of the fish in an image space, using simulation data obtained by simulating a fish; A learning execution unit that executes machine learning using a plurality of the size GT data to generate a learning model in which an image of a fish is input and image size information indicating a three-dimensional size of the fish in an image space is output; a size estimation unit that inputs a fish image obtained by capturing an image of a fish into the learning model generated by the learning execution unit and acquires image size information of the fish; An information processing system comprising:

2. The GT data generation unit generates posture GT data including an image of a fish and posture information indicating a posture of the fish using simulation data obtained by simulating a fish, The learning execution unit executes machine learning using the plurality of size GT data and the plurality of posture GT data to generate the learning models including a posture learning model that receives an image of a fish as an input and outputs posture information indicating the posture of the fish, and a size learning model that receives an image of the fish and the posture information as an input and outputs image size information indicating a three-dimensional size of the fish in an image space, The information processing system according to claim 1 , wherein the size estimation unit inputs the fish image into the posture learning model, and inputs the posture information output from the posture learning model and the fish image into the size learning model to obtain the image size information of the fish.

3. a real size information generating unit that generates real size information indicating a three-dimensional size in real space for each of the plurality of fishes using image size information of the plurality of fishes acquired by the size estimation unit by inputting a fish image including the plurality of fishes into the learning model; a weight estimation unit that estimates a weight of each of the plurality of fish using the actual size information; The information processing system according to claim 1 .

4. A storage unit for storing a relative distance learning model that receives an image including a plurality of fish as an input and outputs the relative distances between the plurality of fish. Equipped with The information processing system of claim 3, wherein the actual size information generation unit determines the distance between the camera that captured the fish image and one of the multiple fish, inputs the fish image into the relative distance learning model to determine the relative distance between the multiple fish, and generates the actual size information for each of the multiple fish using the distance between the camera and the one fish, the relative distance between the multiple fish, and image size information of the multiple fish.

5. The information processing system according to claim 4 , wherein the actual size information generating unit determines a distance between the camera and the one fish by stereoscopic vision.

6. The information processing system of claim 4, wherein the actual size information generation unit identifies a single fish among a plurality of fish included in the fish image, which is tagged and whose actual size in real space is known, by using the tag, and determines the distance between the camera and the single fish based on the actual size of the single fish.

7. The information processing system of claim 4, wherein the actual size information generation unit inputs the fish image including an object whose actual size in real space and distance from the camera are known into the relative distance learning model to determine the relative distance between each of the multiple fish and the objects, and generates the actual size information for each of the multiple fish using the actual size of the object, the distance from the camera, and the relative distance.

8. The GT data generation unit uses simulation data obtained by simulating a fish to generate distance GT data including an image of the fish captured by a virtual camera, a distance between the fish and the virtual camera, and parameters of the camera; The learning execution unit executes machine learning using a plurality of the distance GT data to generate a distance learning model that receives an image of a fish and parameters of a camera that captured the image as input and outputs a distance between the fish and the camera, The information processing system includes: a real size information generating unit that generates real size information indicating a three-dimensional size in real space for each of the plurality of fishes based on image size information of the plurality of fishes acquired by the size estimation unit by inputting a fish image including the plurality of fishes into the learning model, and based on the distances from the camera of each of the plurality of fishes output from the distance learning model by inputting the fish image and parameters of the camera that captured the fish image into the distance learning model; a weight estimation unit that estimates a weight of each of the plurality of fish using the actual size information; The information processing system according to claim 1 , further comprising:

9. a distance information acquisition unit that acquires distance information indicating a distance between the distance measuring sensor and each of the plurality of fishes, the distance information being measured by the distance measuring sensor irradiating light onto the plurality of fishes; a real size information generating unit that generates real size information indicating a three-dimensional size in real space for each of the plurality of fishes, using image size information of the plurality of fishes acquired by the size estimation unit by inputting a fish image including the plurality of fishes into the learning model and the distance information acquired by the distance information acquiring unit; a weight estimation unit that estimates a weight of each of the plurality of fish using the actual size information; The information processing system according to claim 1 , further comprising:

10. A program for causing a computer to function as the information processing system according to any one of claims 1 to 7.

11. 1. A computer-implemented information processing method, comprising: a GT data generation step of generating size GT (Ground Truth) data including an image of a fish and image size information indicating a three-dimensional size of the fish in an image space, using simulation data obtained by simulating a fish; A learning execution stage in which machine learning is performed using a plurality of the size GT data to generate a learning model in which an image of a fish is input and image size information indicating the three-dimensional size of the fish in the image space is output; a size estimation step of inputting a fish image obtained by capturing an image of the fish into the learning model generated in the learning execution step to acquire image size information of the fish; An information processing method comprising:

Images