Information processing device and information processing method

Abstract

This makes it possible to automatically generate swarm behavior of living organisms. [Solution] The information processing device of the present application comprises an acquisition unit that acquires individual information indicating behavioral characteristics of individuals belonging to a flock of organisms to be processed that are placed in a virtual space, relative information indicating relative behavioral characteristics of other individuals located around the individual belonging to the flock of organisms to be processed relative to the individual, and environmental information indicating characteristics of the environment surrounding the individual belonging to the flock of organisms to be processed, and a determination unit that determines the behavior of the individuals belonging to the flock of organisms to be processed based on the individual information, relative information, and environmental information acquired by the acquisition unit.

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Deep learning has produced impressive results in the fields of computer vision and speech recognition. For example, counting the number of fish in a fish pen in a fish farm is a challenge, but it is expected that deep learning techniques will be used to improve this problem. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 045091 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, the accuracy of analysis using deep learning depends on the training data. In other words, to obtain highly accurate analysis results, it is necessary to train a machine learning model with a large amount of training data covering a wide variety of patterns. For example, to accurately estimate information about a school of fish (e.g., the number of fish) from images of a school of fish in a fish pen using computer vision, it is necessary to train a machine learning model with a large amount of training data covering a wide variety of combinations of conditions such as the environment around the fish, the fish species, and the number of fish.

[0005] However, it is not easy to create a large amount of training data that covers a wide variety of patterns. For example, it is extremely difficult to create a large amount of training data by manually adding correct answer data to a huge number of images that cover a wide variety of combinations of conditions.

[0006] In this way, if it is not possible to obtain a large amount of training data that covers a wide variety of patterns, it may be difficult to accurately estimate information about schools of fish from actual photographs, as the machine learning model cannot perform high-quality learning.

[0007] In order to solve the above problems, a technique is known in which a simulated image of a school of fish located in a three-dimensional simulation space captured by a virtual camera is used as training data, instead of a photographed image of a school of fish actually existing in a three-dimensional space (for example, in a fish tank). However, there is room for improvement in the technique for generating the school behavior of fish contained in the simulated image.

[0008] Therefore, an object of the present disclosure is to provide an information processing device and an information processing method that are capable of automatically generating group behavior of living organisms. [Means for solving the problem]

[0009] The present invention aims to at least partially overcome the problems of the prior art.

[0010] According to one aspect of an embodiment, the information processing device of the present application includes an acquisition unit that acquires individual information indicating behavioral characteristics of individuals belonging to a flock of organisms to be processed arranged in a virtual space, relative information indicating behavioral characteristics of other individuals located around the individual belonging to the flock of organisms to be processed relative to the individual, and environmental information indicating characteristics of the environment around the individual belonging to the flock of organisms to be processed, and a determination unit that determines the behavior of the individuals belonging to the flock of organisms to be processed based on the individual information, the relative information, and the environmental information acquired by the acquisition unit.

[0011] The above and other objects, features, advantages and technical and industrial significance of the present invention will become better understood by reference to the following detailed description of preferred embodiments of the present invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an overview of nine parameters used to generate a simulation image according to the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining the relationship between water temperature and fish behavior. [Figure 4] FIG. 4 is a diagram for explaining the relationship between illuminance and fish behavior. [Diagram 5] FIG. 5 is a diagram for explaining the relationship between the net of the fish cage and the behavior of the fish. [Figure 6] FIG. 6 is a diagram for explaining the range within which fish exhibit schooling behavior toward other fish. [Figure 7] FIG. 7 is a diagram for explaining the relationship between the body length of a fish and the speed of the fish. [Figure 8] FIG. 8 is a diagram illustrating the relationship between the reaction rate of fish Mauthner cells and the length of time and time interval required for decision-making in fish. [Figure 9] FIG. 9 is a diagram showing a 3D CG image of a fish and a 3D bounding box. [Figure 10] FIG. 10 shows a three-dimensional CG image of a fish pen. [Figure 11] FIG. 11 illustrates a finite state machine that determines the state of the fish. [Figure 12] FIG. 12 is a diagram showing an example of a simulation image in which a parameter indicating a chlorophyll concentration according to the first embodiment is changed. [Figure 13] FIG. 13 is a diagram showing an example of a simulation image in which a parameter indicating the concentration of suspended matter according to the first embodiment is changed. [Figure 14] FIG. 14 is a diagram showing an example of a simulation image obtained by changing parameters indicating the intensity of sunlight and the depth of the fish pen from the water surface according to the first embodiment. [Figure 15] FIG. 15 is a diagram showing an example of a simulation image in which a parameter indicating the distance from the water surface to the depth of the virtual camera position according to the first embodiment is changed. [Figure 16] FIG. 16 is a diagram showing an example of training data including labels indicating the degree of overlap of each fish included in the simulation image according to the first embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of a machine learning model according to the first embodiment. [Figure 18] FIG. 18 is a diagram showing an example of a screen on which the estimation result of the captured image is output. [Figure 19] FIG. 19 is a diagram showing an example of a screen on which the estimation result of the captured image is output. [Figure 20] FIG. 20 is a diagram showing an example of a screen on which the estimation result of the captured image is output. [Figure 21] FIG. 21 is a diagram illustrating an example of the configuration of an information processing device according to the second embodiment. [Figure 22] FIG. 22 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, a detailed description will be given of an information processing device and an information processing method according to the present application (hereinafter, referred to as an "embodiment") with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to the embodiment. In addition, the same components in the following embodiments are given the same reference numerals, and duplicated descriptions are omitted.

[0014] 1. First embodiment

[0015] 1-1. Introduction

[0016] In recent years, fish farming has been attracting attention as a means to solve the global food problem. In order to supply high-quality fish through fish farming, it is important to accurately know the number of fish, which is closely related to feeding (the feeding of fish).

[0017] However, in the special underwater environment of aquaculture farms, it can be difficult to utilize land-based IT technology. For this reason, traditionally, people would scoop up some of the fish with a net, weigh them, and then visually count the number of fish. This method places a heavy burden on the fish and is inaccurate.

[0018] In recent years, a method of automatically counting the number of tails in a school of fish from images of the school of fish in a fish pen using computer vision has been attracting attention. A specific example is a method of training a machine learning model for image recognition to estimate the number of tails in a school of fish from a photographed image. However, in order to accurately estimate the number of tails from a photographed image using a machine learning model, it is necessary to train the machine learning model with a large amount of training data that covers a wide range of combinations of conditions such as the environment around the fish (e.g., season, time, weather, water temperature, illuminance, water pollution, etc.) and the biological characteristics of the fish (e.g., fish species, fish size, fish speed, number of tails, etc.).

[0019] However, it is not easy to create a large amount of training data that covers a wide variety of combinations of conditions as described above. For example, it is very difficult to create a large amount of training data by manually adding correct answer data to each of a huge number of images that cover a wide variety of combinations of conditions. In addition, the quality of the training data is also important to accurately estimate the number of tails from captured images using a machine learning model. However, when correct answer data is added manually, the quality of the training data may not necessarily be high.

[0020] For example, a photographed image of a school of fish swimming in a three-dimensional fish tank may include an area where a fish located closer to the camera (hereinafter also referred to as a fish located in the foreground) overlaps with a fish located further back than the fish (hereinafter also referred to as a fish located in the background). In this way, in a photographed image including an area where a fish located in the foreground and a fish located in the background overlap, a part of the body of the fish located in the background is hidden, so it may be difficult to identify the position of the fish included in the photographed image with the human eye. Therefore, when an annotation work for a bounding box surrounding the position of a fish is manually performed on a photographed image including an area where a fish located in the foreground and a fish located in the background overlap, for example, it may be difficult to attach an annotation indicating the exact position of the fish. In addition, since it is difficult to identify the position of a fish included in a photographed image with the human eye, there is also a problem that the annotation results differ depending on the worker.

[0021] In contrast, the information processing device according to the first embodiment generates a simulation image that visually displays information indicating the degree of overlap of multiple fish in a two-dimensional simulation image captured by a virtual camera of multiple fish located in a three-dimensional simulation space (see FIG. 16 described later).

[0022] In this way, the information processing device according to the first embodiment uses a simulation image of a school of fish located in a three-dimensional simulation space captured by a virtual camera, instead of a captured image of a school of fish actually existing in a three-dimensional space (for example, in a fish tank). This allows the information processing device to automatically calculate information indicating the degree of overlap of a plurality of fish in the simulation image. Furthermore, the information processing device can add information indicating the degree of overlap of a plurality of fish to the simulation image as an annotation, so that accurate and high-quality training data can be easily generated. This allows the information processing device to, for example, allow a machine learning model that estimates information about a school of fish (for example, the number of tails) from a captured image to learn a large amount of high-quality training data. That is, the information processing device can improve the estimation accuracy of a machine learning model that estimates information about a school of fish (for example, the number of tails) from a captured image, for example, by learning a large amount of high-quality training data. Therefore, the information processing device can accurately estimate information about a school of fish.

[0023] [1-2. Configuration of information processing device]

[0024] 1 is a diagram showing an example of the configuration of an information processing device 100 according to the first embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, an input unit 130, an output unit 140, and a control unit 150.

[0025] (Communication unit 110)

[0026] The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to a network by wire or wirelessly, and transmits and receives information to and from, for example, a terminal device used by a manager who manages fish.

[0027] (Storage unit 120)

[0028] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various programs (an example of an information processing program) used in the simulation. The storage unit 120 also stores values ​​of various parameters used in the simulation.

[0029] FIG. 2 is a diagram for explaining an outline of nine parameters used to generate a simulation image according to the first embodiment. The information processing device according to the first embodiment generates a simulation image including the behavior of each fish belonging to a school of fish based on the values ​​of the nine parameters shown in FIG. 2. The upper part of FIG. 2 shows school parameters related to the characteristics of three types of school behavior (alignment, joining, and separation). The middle part of FIG. 2 shows external parameters related to water temperature, illuminance, nets of the fish pen, etc. The lower part of FIG. 2 shows internal parameters such as the range in which fish behave in a school, the relationship between the size and movement speed of the fish, and the time required for the fish to make a decision. The nine parameters will be described in detail later.

[0030] In this way, the information processing device 100 generates a simulation image including the behavior of a school of fish based on information on the biological characteristics of the fish, information on the environment surrounding the fish, and information on the characteristics of the school behavior of the fish, and therefore it is possible to accurately reproduce the behavior of a school of fish in an actual fish pen. Furthermore, the information processing device 100 can easily generate a large number of simulation images covering a variety of patterns by changing the values ​​of multiple parameters used to generate the simulation image. Furthermore, the information processing device 100 can utilize the values ​​of the parameters used to generate the simulation image as correct answer data, and therefore it is possible to easily generate a large amount of high-quality training data compared to the case where correct answer data is manually added.

[0031] This allows the information processing device 100 to, for example, allow a machine learning model that estimates information about a school of fish (for example, the number of tails) from a captured image to learn a large amount of high-quality training data. That is, the information processing device 100 can improve the estimation accuracy of the machine learning model that estimates information about a school of fish (for example, the number of tails) from a captured image by learning a large amount of high-quality training data. Therefore, the information processing device 100 can accurately estimate information about a school of fish.

[0032] (Input unit 130)

[0033] Various operations are input by the user to the input unit 130. For example, the input unit 130 may receive various operations from the user via a display surface (e.g., the output unit 140) using a touch panel function. The input unit 130 may also receive various operations from buttons provided on the information processing device 100, or a keyboard or mouse connected to the information processing device 100.

[0034] (Output unit 140)

[0035] The output unit 140 is a display screen realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. The output unit 140 displays various information according to the control of the control unit 150. When a touch panel is adopted in the information processing device 100, the input unit 130 and the output unit 140 are integrated. In the following description, the output unit 140 may be referred to as a screen.

[0036] (Control unit 150)

[0037] The control unit 150 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 150 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0038] The control unit 150 has an acquisition unit 151, a generation unit 152, an estimation unit 153, and an output control unit 154 as functional units, and may realize or execute the information processing action described below. Note that the internal configuration of the control unit 150 is not limited to the configuration shown in FIG. 1, and may be other configurations as long as they perform the information processing described below. Also, each functional unit indicates a function of the control unit 150, and does not necessarily have to be physically distinct.

[0039] (Acquisition part 151)

[0040] The acquisition unit 151 acquires values ​​of intrinsic parameters related to biological characteristics of the fish, values ​​of extrinsic parameters related to characteristics of the environment around the fish, and values ​​of school parameters related to characteristics of schooling behavior, which is behavior that fish exhibit toward other fish. Specifically, the acquisition unit 151 acquires values ​​of intrinsic parameters, extrinsic parameters, and school parameters input by a user via the input unit 130.

[0041] More specifically, the acquisition unit 151 acquires values ​​of intrinsic parameters set for each fish belonging to a school of fish included in the simulation image. For example, when the number of tails in the school of fish included in the simulation image is N (N is a natural number), the acquisition unit 151 acquires values ​​of N intrinsic parameters set for each of the N individual fish.

[0042] The acquisition unit 151 also acquires the values ​​of school parameters that are set for the schools of fish included in the simulation image. The school parameter values ​​are set to values ​​that are common to each fish in each school of fish. For example, when the number of tails in a school of fish included in the simulation image is N (N is a natural number), the acquisition unit 151 acquires the values ​​of school parameters that are common to all N individual fish.

[0043] The acquisition unit 151 also acquires values ​​of external parameters that are set for each environment surrounding the school of fish included in the simulation image. For example, the acquisition unit 151 acquires values ​​of external parameters that are set for each fish pen in which the school of fish included in the simulation image is located. For example, the acquisition unit 151 acquires values ​​of external parameters that are set to different values ​​depending on the water depth of the fish pen in which the school of fish included in the simulation image is located.

[0044] (Generation unit 152)

[0045] The generation unit 152 generates a simulation image by controlling the behavior of each fish belonging to the school of fish based on the intrinsic parameter value, the extrinsic parameter value, and the school parameter value acquired by the acquisition unit 151. Below, a case where the generation unit 152 controls the behavior of each fish belonging to the school of fish based on the intrinsic parameter value, the extrinsic parameter value, and the school parameter value will be described in detail with reference to Figs. 3 to 8.

[0046] FIG. 3 is a diagram for explaining the relationship between water temperature and fish behavior. It is known that, as a biological characteristic of fish, each individual fish has a preferred water temperature, and each individual fish moves toward the area of ​​the preferred water temperature. In the example shown in the upper part of FIG. 3, the generation unit 152 controls the behavior of the fish so that the fish moves toward the area of ​​the preferred water temperature because the water temperature around the fish is higher than the water temperature preferred by the fish. In the example shown in the lower part of FIG. 3, the generation unit 152 controls the behavior of the fish so that the fish moves toward the area of ​​the preferred water temperature because the water temperature around the fish is lower than the water temperature preferred by the fish. For example, the generation unit 152 controls the movement direction of the fish so that the movement direction of the fish moves toward the area of ​​the preferred water temperature. In this way, the generation unit 152 controls the behavior of each fish so that each fish moves toward the area of ​​the preferred water temperature based on the water temperature preferred by each fish belonging to the school of fish as the value of the internal parameter.

[0047] It is also known that the distribution of water temperature in a cage varies depending on the season (or seasonal tides) and water depth. In the example shown in the upper part of FIG. 3, it is summer, and the Kuroshio Current flows into the cage in which the fish are located, causing the water temperature in the cage to be high. It is also known that the water temperature in a cage in summer is higher the closer to the water surface of the cage and lower the closer to the bottom of the cage. Therefore, the generation unit 152 changes the distribution of water temperature in the cage depending on the water depth, so that the water temperature is higher the closer to the water surface of the cage and lower the closer to the bottom of the cage, as the value of the external parameter. In the example shown in the lower part of FIG. 3, it is winter, and the Oyashio Current flows into the cage in which the fish are located, causing the water temperature in the cage to be low. It is also known that the water temperature in a cage in winter is lower the closer to the water surface of the cage and higher the closer to the bottom of the cage. Therefore, the generation unit 152 changes the distribution of water temperature in the cage according to the water depth so that the water temperature is lower the closer to the water surface of the cage and higher the closer to the bottom of the cage, as the value of the external parameter. In this way, the generation unit 152 controls the behavior of each fish so that each fish moves toward its preferred water temperature area based on the distribution of water temperature in the cage in which each fish belonging to the school of fish is located, as the value of the external parameter.

[0048] It is also known that, as a biological characteristic of fish, the higher the water temperature around a fish, the faster the fish moves, and the lower the water temperature around a fish, the slower the fish moves. Therefore, the generation unit 152 controls the behavior of each fish so that the higher the water temperature around each fish belonging to the school of fish, the faster the movement speed of each fish, based on the distribution of water temperatures in the fish pen in which each fish belonging to the school of fish is located, as the value of the internal parameter.

[0049] FIG. 4 is a diagram for explaining the relationship between illuminance and fish behavior. It is known that, as a biological characteristic of fish, each individual fish has a preferred illuminance, and each individual fish moves toward the area of ​​the preferred illuminance. In the example shown in the upper part of FIG. 4, the generation unit 152 controls the behavior of the fish so that the fish moves toward the area of ​​the preferred illuminance because the illuminance around the fish is lower than the illuminance preferred by the fish. In the example shown in the lower part of FIG. 4, the generation unit 152 controls the behavior of the fish so that the fish moves toward the area of ​​the preferred illuminance because the illuminance around the fish is higher than the illuminance preferred by the fish. For example, the generation unit 152 controls the movement direction of the fish so that the movement direction of the fish moves toward the area of ​​the preferred illuminance. In this way, the generation unit 152 controls the behavior of each fish so that each fish moves toward the area of ​​the preferred illuminance based on the illuminance preferred by each fish belonging to the school of fish as the value of the internal parameter.

[0050] It is also known that the distribution of illuminance in the cage varies depending on the season, time, weather, and water depth. In the example shown in the upper part of FIG. 4, the season is winter (or the time is morning or night, etc.), and the illuminance in the cage is low. In the example shown in the lower part of FIG. 4, the season is summer (or the time is daytime, etc.), and the illuminance in the cage is high. It is also known that the illuminance in the cage is generally higher the closer to the water surface of the cage and lower the closer to the bottom of the cage. Therefore, the generating unit 152 changes the distribution of illuminance in the cage depending on the water depth, so that the illuminance is higher the closer to the water surface of the cage and lower the closer to the bottom of the cage, as the value of the external parameter. In this way, the generation unit 152 controls the behavior of each fish so that each fish moves toward an area of ​​preferred illuminance based on the distribution of illuminance in the fish pen in which each fish belonging to the school of fish is located, as the value of the external parameter.

[0051] FIG. 5 is a diagram for explaining the relationship between the net of the fish pen and the behavior of the fish. It is known that, as a biological characteristic of fish, when a fish approaches an obstacle such as the net of the fish pen or the water surface of the fish pen, the fish will take evasive action to avoid the obstacle. The inventors of the present invention have obtained experimental results that show that when the distance between the fish and the obstacle is four times or less than the body length of the fish, the fish detects the obstacle and takes evasive action to avoid the obstacle. Therefore, the generation unit 152 controls the behavior of each fish so that each fish takes evasive action to avoid the obstacle when the distance between each fish belonging to the school of fish and the obstacle is equal to or less than a predetermined threshold value (for example, a predetermined multiple of the body length of each fish) as the value of the internal parameter.

[0052] As shown in FIG. 5, it is known that, as a biological characteristic of fish, the closer (farther) the distance between the fish and the net of the fish pen, the faster (slower) the fish will be able to avoid collision. Therefore, the generation unit 152 controls the behavior of each fish based on a repulsive force that increases as the distance between each fish belonging to the school of fish and an obstacle decreases. Here, the repulsive force refers to a force that causes a fish to move away from an obstacle to avoid colliding with the obstacle. For example, when the distance between the fish and the obstacle is equal to or less than a predetermined threshold, the generation unit 152 controls the behavior of the fish based on a repulsive force that is proportional to the distance between the fish and the obstacle. For example, the generation unit 152 controls the behavior of the fish so that the fish avoids the obstacle (the net of the fish pen in FIG. 5) by applying a repulsive force to the fish in a direction opposite to the moving direction of the fish (the direction in which the fish moves toward the net of the fish pen in FIG. 5). In this way, the generation unit 152 controls the behavior of each fish based on the repulsive force acting on each fish when the distance between each fish belonging to the school of fish and an obstacle is equal to or less than a predetermined threshold value as the value of the internal parameter.

[0053] FIG. 6 is a diagram for explaining the range in which fish exhibit schooling behavior toward other fish. It is known that, as a biological characteristic of fish, fish exhibit schooling behavior toward other fish located around them. The inventors of the present invention obtained experimental results that fish detect other fish located within a distance range of three times their own body length and exhibit schooling behavior toward other fish located within a distance range of three times their own body length. Therefore, the generation unit 152 controls the behavior of each fish so that each fish exhibits schooling behavior when other fish are located within a predetermined range (for example, within a distance range of a predetermined multiple of the body length of each fish) from the position of each fish belonging to the school of fish as the value of the school parameter.

[0054] Here, the school behavior of fish toward other fish refers to behavior that follows three rules: cohesion, separation, and alignment. "Cohesion" refers to the force that makes individuals approach each other without scattering (hereinafter also referred to as cohesion force), "separation" refers to the force that makes individuals move away from each other to avoid collision (hereinafter also referred to as separation force), and "alignment" refers to the force that makes individuals move in the same direction as a school (hereinafter also referred to as alignment force). Fish behavior is determined by the sum of these three forces. Hereinafter, the three forces of cohesion, separation, and alignment are collectively referred to as "school parameters." Each of the three forces is determined by the values ​​of parameters that indicate the magnitude, direction, and range of the force. For example, the school parameters are determined by the size of the space in which the school is located (e.g., the fish pen), the size of the fish, and the density of the school of fish. In this way, the generation unit 152 controls the behavior of each fish based on the values ​​of the school parameters, which are the bonding force that causes each fish in the school to approach each other so as not to become separated, the separation force that causes each fish in the school to move away from each other to avoid collisions, and the alignment force that causes each fish in the school to align their movement direction as a school.

[0055] FIG. 7 is a diagram for explaining the relationship between the body length of a fish and the speed of the fish. It is known that, as a biological characteristic of fish, the larger (smaller) the body length of a fish is, the faster (slower) the moving speed of the fish is. Therefore, the generation unit 152 controls the behavior of each fish, based on the body length of each fish belonging to the school of fish as the value of the internal parameter, so that the moving speed of each fish is faster as the body length is larger. It is also known that, as a biological characteristic of fish, the larger (smaller) the amplitude of the tail fin of a fish is, the faster (slower) the moving speed of the fish is. Therefore, the generation unit 152 may control the behavior of each fish, based on the magnitude of the amplitude of the tail fin of each fish belonging to the school of fish as the value of the internal parameter, so that the moving speed of each fish is faster as the magnitude of the amplitude is larger.

[0056] FIG. 8 is a diagram for explaining the relationship between the reaction speed of fish Mauthner cells and the length of time and time interval required for a fish to make a decision. It is known that the time required for a fish to make a decision to take each action is determined by the reaction speed of the fish Mauthner cells (a type of nerve cell). Here, the decision to take each action refers to the instantaneous decision of the fish to "where to swim" and "how to swim". In the example shown in FIG. 8, the time required for a fish to make a decision to take each action is about 200 milliseconds, and the time interval between decisions is several hundred milliseconds. Therefore, the generation unit 152 controls the action of each fish based on the length of time and time interval required for each fish belonging to the school of fish to make a decision to take each action as the value of the internal parameter. Specifically, the generation unit 152 sets the update time of the simulation image to the time required for a fish to make a decision to take each action (for example, several hundred milliseconds). For example, the generation unit 152 controls the length of time that the fish stays in a predetermined state (for example, the "Slow Swim" state shown in FIG. 11 described later) to the time required for the fish to make a decision to take each action (for example, several hundred milliseconds). Next, the generation unit 152 controls the time interval for the fish to transition from a predetermined state (for example, the "Slow Swim" state shown in FIG. 11 described later) to another state (for example, the "C-start Left" state shown in FIG. 11 described later) to the time interval between decisions (for example, several hundred milliseconds). Next, the generation unit 152 controls the length of time that the fish stays in another state (for example, the "C-start Left" state shown in FIG. 11 described later) to the time required for the fish to make a decision to take each action (for example, several hundred milliseconds).

[0057] It is also known that, as a biological characteristic of fish, the lower (higher) the water temperature around the fish (or the body temperature of the fish), the slower (faster) the reaction speed of the fish's Mauthner cells. Therefore, the generation unit 152 may control the behavior of each fish based on the length of time and time interval required for decision-making, which are longer as the water temperature around the fish is lower, as the value of the internal parameter.

[0058] FIG. 9 is a diagram showing a 3D CG image and a 3D bounding box of a fish. The fish shown in FIG. 9 is a 3D CG image of a fish generated based on the body side data (hereinafter, also referred to as the actual data of the fish tank) measured for each fish species. The generation unit 152 generates a simulation image using a realistic 3D CG image of a fish generated based on the actual data of the fish tank. The generation unit 152 may generate a simulation image using a 3D CG image of a fish not based on the actual data of the fish tank. For example, the generation unit 152 may generate a simulation image using a 3D CG image of a fish having body side data different from the actual data of the fish tank based on peripheral values ​​of the actual data of the fish tank (for example, values ​​within a predetermined range from a representative value of the actual measurement data). The generation unit 152 may also make the body length, moving speed, initial position, initial state, weight, etc. of each fish belonging to the school of fish included in the simulation image different from each other.

[0059] Moreover, the frame surrounding the fish shown in FIG. 9 is a three-dimensional bounding box for identifying the position information and size of the fish. The generation unit 152 generates the position information of each fish. For example, the generation unit 152 generates position information indicating the center position of the fish OB and the range in which the fish OB exists. The fish position information is generated using, for example, three-dimensional coordinate information. In the example of FIG. 9, the generation unit 152 generates the coordinates of the center of the fish OB (b x , b y , b z ) (not shown), and the depth b of the minimum size rectangular parallelepiped (three-dimensional bounding box) that contains the fish OB d , width b w and height b h Generate information about.

[0060] FIG. 10 is a diagram showing a 3D CG image of a fish cage. FIG. 10 shows a 3D CG image of a cage generated based on the latitude and longitude, water temperature, illuminance, underwater image data, etc. of an actual cage (hereinafter, also referred to as actual data of the cage). The generating unit 152 generates a simulation image using a realistic 3D CG image of the cage generated based on the actual data of the cage. The generating unit 152 may generate a simulation image using a 3D CG image of the cage that is not based on the actual data of the cage. For example, the generating unit 152 may generate a simulation image using a 3D CG image of the cage having data different from the actual data of the cage (for example, the area, depth, water temperature, illuminance, etc. of the cage are values ​​within a predetermined range from the representative value of the actual measurement data) based on peripheral values ​​of the actual data of the cage (for example, values ​​within a predetermined range from the representative value of the actual measurement data). In addition, the generation unit 152 may generate a simulation image using a 3D CG image of the fish pen having data different from the actual data of the fish pen, based on values ​​that do not exist in the actual data (for example, data for a time period when no actual measurement data exists).

[0061] FIG. 11 is a diagram showing a finite state machine that determines the state of a fish. It is known that, as a biological characteristic of fish, the way fish swim (also called swimming style or swimming) can be classified into specific patterns for each fish species. Note that FIG. 11 describes a case where the fish species is salmon, but the swimming pattern may differ for each fish species. The generation unit 152 generates 12 types of animations that show the swimming states of 12 types of fish as shown in FIG. 11. Furthermore, the generation unit 152 transitions the swimming state of each fish by determining the swimming state of each fish from the 12 types of animations using a finite state machine (FSM).

[0062] For example, the generation unit 152 transitions the swimming state of the fish from a "slow swim" state to a "C-start left" state. Then, the generation unit 152 transitions the swimming state of the fish from the "C-start left" state to a "fast swim" state. Then, the generation unit 152 transitions the swimming state of the fish from a "fast swim" state to a "semi fast swim" state. Then, the generation unit 152 transitions the swimming state of the fish from a "semi fast swim" state to a "slow swim" state. Then, the generation unit 152 transitions the swimming state of the fish from a "slow swim" state to a "rest swim" state. Furthermore, the generation unit 152 can realize a smooth animation by complementing the state before the transition and the state after the transition at the time of the state transition.

[0063] Furthermore, the generation unit 152 realizes a 3D simulation that closely resembles a school of fish swimming in a fish tank by controlling the behavior of the 3D CG of realistic fish (see FIG. 9) so that the 3D CG of realistic fish swims in a 3D simulation space (see FIG. 10) that reproduces a realistic fish tank based on the intrinsic parameter values, extrinsic parameter values, and school parameter values ​​acquired by the acquisition unit 151. Next, the generation unit 152 generates a simulation image by capturing an image of the 3D simulation that closely resembles a school of fish swimming in the fish tank with a virtual camera installed at a position at a predetermined depth from the water surface in the 3D simulation space that reproduces the fish tank.

[0064] Furthermore, when generating the simulation image, the generating unit 152 may generate training data to which a two-dimensional bounding box indicating position information of each fish included in the generated simulation image is added. Note that the generating unit 152 can easily generate training data to which a two-dimensional bounding box is added by using the position information of each fish used when generating the simulation image.

[0065] Furthermore, when generating the simulation image, the generating unit 152 may generate training data to which a three-dimensional bounding box indicating position information of each fish included in the generated simulation image is added. Note that the generating unit 152 can easily generate training data to which a three-dimensional bounding box is added by using the position information of each fish used when generating the simulation image.

[0066] Furthermore, when generating a simulation image, the generating unit 152 may generate training data in which each fish included in the generated simulation image is replaced with a silhouette. For example, the generating unit 152 generates training data in which each fish replaced with a silhouette is displayed in a different color. Note that color coding corresponds to adding a color element. For example, when there are multiple fish, the generating unit 152 may color each fish with a different color, or may color each fish according to the classification information of the fish.

[0067] In the above example, the generation unit 152 generates a simulation image including the behavior of each fish belonging to a school of fish based on the values ​​of intrinsic parameters related to the biological characteristics of the fish, the values ​​of extrinsic parameters related to the characteristics of the environment around the fish, and the values ​​of school parameters related to the characteristics of school behavior, which is the behavior of the fish toward other fish. However, the types of parameters are not limited to these. In Figs. 12 to 15, a case will be described in which the generation unit 152 generates a simulation image based on the values ​​of color parameters related to the characteristics of the color of water in the water where the fish are present, in addition to the above-mentioned intrinsic parameters, extrinsic parameters, and school parameters. Specifically, the acquisition unit 151 acquires the values ​​of the intrinsic parameters, the extrinsic parameters, the school parameters, and the color parameters. Specifically, the acquisition unit 151 acquires the values ​​of the intrinsic parameters, the extrinsic parameters, the school parameters, and the color parameters input by the user via the input unit 130. The generation unit 152 generates a simulation image including the behavior of each fish belonging to the school of fish, based on the values ​​of the intrinsic parameters, the extrinsic parameters, the school parameters, and the color parameters acquired by the acquisition unit 151.

[0068] FIG. 12 is a diagram showing an example of a simulation image in which a parameter indicating the concentration of chlorophyll according to the first embodiment is changed. The six images shown in FIG. 12 are simulation images in which the concentration of chlorophyll decreases toward the left and increases toward the right. Chlorophyll is a green pigment contained in phytoplankton and algae present in water. It is generally known that chlorophyll in water is likely to absorb light of a wavelength corresponding to blue (hereinafter also referred to as blue light). Therefore, it is known that the higher the concentration of chlorophyll in water, the more easily blue light is attenuated, so that water with a high concentration of chlorophyll appears reddish. In addition, chlorophyll causes water to become cloudy. Therefore, it is known that water with a high concentration of chlorophyll appears reddish and cloudy, and as a result, it appears reddish brownish. Therefore, the generation unit 152 generates a simulation image based on the concentration of chlorophyll in the water of a fish pen in which each fish belonging to a school of fish is located, as a value of a color parameter. Specifically, the generation unit 152 generates a simulation image by controlling the RGB values ​​of each pixel of the simulation image so that the higher the concentration of chlorophyll in the fish pond water, the smaller the luminance value of B (blue) among the RGB values ​​of each pixel of the simulation image.

[0069] FIG. 13 is a diagram showing an example of a simulation image in which a parameter indicating the concentration of floating matter according to the first embodiment is changed. The six images shown in FIG. 13 are simulation images in which the concentration of floating matter decreases toward the left and increases toward the right. Floating matter is rolled up seabed soil, shellfish or jellyfish eggs, or garbage. Floating matter causes water to become cloudy. It is known that water with a high concentration of floating matter appears cloudy. Therefore, the generating unit 152 generates a simulation image based on the concentration of floating matter in the water of a fish pen in which each fish belonging to a school of fish is located as a value of a color parameter. Specifically, the generating unit 152 generates a simulation image by controlling the luminance value of each pixel of the simulation image so that the luminance value of each pixel of the simulation image becomes smaller as the concentration of floating matter in the water of the fish pen increases.

[0070] FIG. 14 is a diagram showing an example of a simulation image in which the parameters indicating the intensity of sunlight and the depth of the fish pen from the water surface according to the first embodiment are changed. The six images shown in FIG. 14 are simulation images in which the intensity of sunlight is changed between noon and 6 p.m. In FIG. 14, the simulation image on the left side is taken at noon, so the sunlight intensity is strong and the water in the fish pen looks bright. On the other hand, the simulation image on the right side is taken after sunset, so the sunlight intensity is low and the water in the fish pen looks dark. It is generally known that the absorbance of light incident on a dilute solution such as seawater is proportional to the concentration of the dilute solution and the length of the light path (Lambert-Beer law). Here, when the intensity of light before it enters the dilute solution is I0 and the intensity of light after it passes through the dilute solution is I, I / I0=T is called the transmittance. In addition, the negative value of the common logarithm of the transmittance T, "-logT=E", is called the absorbance. In addition, the intensity of sunlight illuminating the water of the cage and the depth of the cage from the water surface change according to the latitude and longitude indicating the location of the cage and the date and time. That is, the color of the seawater in the cage changes according to the intensity of sunlight illuminating the water of the cage, the depth of the cage from the water surface, and the depth of the cage from the water surface. Therefore, the generation unit 152 generates a simulation image based on the intensity of sunlight illuminating the water of the cage in which each fish belonging to the school of fish is located and the depth of the cage from the water surface as the value of the color parameter. Specifically, the generation unit 152 generates a simulation image by controlling the luminance value of each pixel of the simulation image so as to follow the Beer-Lambert law.

[0071] FIG. 15 is a diagram showing an example of a simulation image in which a parameter indicating the distance from the water surface to the depth of the virtual camera position according to the first embodiment is changed. The six images shown in FIG. 15 are simulation images in which the virtual camera position is shallower as it moves to the left, and deeper as it moves to the right. It is generally known that the deeper the water is from the water surface, the darker the color of the water becomes because more light is absorbed. Therefore, the generation unit 152 generates a simulation image based on the distance from the water surface of a fish pen in which each fish belonging to a school of fish is located to the depth of the virtual camera position as the value of the color parameter. Specifically, the generation unit 152 generates a simulation image by controlling the luminance value of each pixel of the simulation image so that the luminance value of each pixel of the simulation image becomes smaller as the distance from the water surface of the fish pen to the depth of the virtual camera position increases.

[0072] Fig. 16 is a diagram showing an example of training data including labels indicating the degree of overlap of each fish included in the simulation image according to the first embodiment. In Fig. 16, each fish included in the simulation image is replaced with a silhouette. In Fig. 16, the acquisition unit 151 acquires a two-dimensional simulation image obtained by capturing a three-dimensional CG image of a plurality of fish swimming in a three-dimensional simulation space simulating a fish pen using a virtual camera.

[0073] The generating unit 152 generates a simulation image that visually displays information indicating the degree of overlap of a plurality of fishes in a two-dimensional simulation image obtained by capturing a plurality of fishes located in a three-dimensional simulation space by a virtual camera. Specifically, the generating unit 152 determines whether or not there are any fishes that appear to overlap in the image among the plurality of fishes included in the simulation image based on the simulation image acquired by the acquiring unit 151. For example, the generating unit 152 determines whether or not there are any fishes that appear to overlap in the image when viewed from the position of the virtual camera based on the position coordinates of each of the plurality of fishes located in the three-dimensional simulation space and the position coordinates of the virtual camera. In FIG. 16, the generating unit 152 determines that three fishes OB1 to OB3 included in the simulation image appear to overlap in the image.

[0074] Next, the generation unit 152 calculates information indicating the degree of overlap of the multiple fish in the simulation image based on the position coordinates of each of the multiple fish located in the three-dimensional simulation space and the position coordinates of the virtual camera. For example, the generation unit 152 calculates the ratio of the area of ​​the visible region of each of the multiple fish as the information indicating the degree of overlap. In FIG. 16, the generation unit 152 calculates that the ratio of the area of ​​the visible region of the fish OB1 located at the foreground is 100% because the entire body of the fish is visible. In addition, the generation unit 152 calculates that the ratio of the area of ​​the region of the fish OB2 located behind the fish OB1 that is not visible because part of its body is hidden by the fish OB1 located in the foreground is 20%, and the ratio of the area of ​​the visible region is 80%. In addition, the generation unit 152 calculates that the area of ​​the fish OB3 located at the back (fish OB3 located behind fish OB2) that is not visible because part of its body is hidden by the fish OB2 located in front is 40%, and the area of ​​the visible region is 60%.

[0075] Next, when the generating unit 152 calculates the information indicating the degree of overlap, the generating unit 152 visualizes the information indicating the degree of overlap and adds it to the simulation image. For example, the generating unit 152 displays labels indicating the proportion of the area of ​​the visible region of each of the multiple fishes at a position within a predetermined range from each fish. In FIG. 16, the generating unit 152 displays a label L1 including the characters "100%" indicating the proportion of the area of ​​the visible region of the fish OB1 at a position within a predetermined range from the fish OB1. In addition, the generating unit 152 displays a label L2 including the characters "80%" indicating the proportion of the area of ​​the visible region of the fish OB2 at a position within a predetermined range from the fish OB2. In addition, the generating unit 152 displays a label L3 including the characters "60%" indicating the proportion of the area of ​​the visible region of the fish OB3 at a position within a predetermined range from the fish OB3.

[0076] FIG. 17 is a diagram illustrating an example of a machine learning model according to the first embodiment. The machine learning model illustrated in FIG. 17 includes a YOLO (You Only Look Once) neural network. When a simulation image is input to the machine learning model, the generation unit 152 trains the machine learning model so as to output individual identification information of each fish included in the simulation image and information corresponding to the individual identification information of each fish (e.g., the number of tails). Note that the generation unit 152 may apply any known neural network used for object detection to the machine learning model, not limited to YOLO. For example, the generation unit 152 can apply Faster R-CNN developed for object detection or SSD (Single Shot MultiBox Detector) to the machine learning model in addition to YOLO.

[0077] For example, when the generation unit 152 inputs the generated simulation image into the machine learning model M1, the generation unit 152 trains the machine learning model M1 so as to output a simulation image in which a three-dimensional bounding box indicating the position information of each individually identified fish is superimposed, and the number of tails of the individually identified fish (the number of tails in a school of fish).

[0078] In addition, the generation unit 152 trains the machine learning model M2 so that when the generated simulation image is input to the machine learning model M2, it outputs a simulation image in which a two-dimensional bounding box indicating the size of each individually identified fish is superimposed, and the number of tails of the individually identified fish (the number of tails in a school of fish).

[0079] In addition, the generation unit 152 trains the machine learning model M3 so that when the generated simulation image is input to the machine learning model M3, it outputs a simulation image in which each individually identified fish is replaced with a silhouette and the number of tails of the individually identified fish (the number of tails in a school of fish).

[0080] Furthermore, the generation unit 152 trains each of the machine learning models M1 to M3 so that, when the generated simulation image is input to each of the machine learning models M1 to M3, the model outputs a simulation image in which visualized information indicating the degree of overlap of a plurality of fishes in the simulation image is superimposed. For example, the generation unit 152 trains each of the machine learning models M1 to M3 so that, when the generated simulation image is input to each of the machine learning models M1 to M3, the model outputs a simulation image in which a label indicating the proportion of the area of ​​the visible region of each of the plurality of fishes in the simulation image is superimposed.

[0081] In this way, the generation unit 152 trains the machine learning model so that when the generated simulation image is input to the machine learning model, it outputs correct answer data or information corresponding to the correct answer data generated based on the parameter information used to generate the simulation image.

[0082] (Estimation part 153)

[0083] The estimation unit 153 estimates information about a plurality of objects included in a captured image from a captured image in which the plurality of objects are captured, using the trained machine learning model generated by the generation unit 152. For example, the estimation unit 153 estimates information about a school of fish included in a captured image from a captured image in which a school of fish is captured, using the trained machine learning model generated by the generation unit 152.

[0084] Furthermore, the estimation unit 153 estimates the number of multiple objects included in the captured image as information regarding the multiple objects. For example, the estimation unit 153 inputs the captured image to the trained machine learning model M2 generated by the generation unit 152. Next, the estimation unit 153 outputs from the machine learning model M2 the captured image in which a two-dimensional bounding box and a label indicating the proportion of the area of ​​the visible region of each of the multiple fish in the captured image are superimposed, and the number of individually identified fish tails (the number of tails in a school of fish). The estimation unit 153 regards the information output from the machine learning model M2 as an estimation result.

[0085] (Output control unit 154)

[0086] The output control unit 154 controls so as to display on the screen the estimation result estimated by the estimation unit 153. Figures 18 to 20 are diagrams showing examples of the screen on which the estimation result of the captured image is output.

[0087] 18, the output control unit 154 causes the captured image to be displayed on the screen with a two-dimensional bounding box surrounding each fish detected from the captured image and a label indicating the type of each detected fish superimposed thereon, as the estimation result estimated by the estimation unit 153. Note that the output control unit 154 may also cause the captured image to be displayed on the screen with a label indicating the proportion of the area of ​​the visible region of each detected fish superimposed thereon, as the estimation result estimated by the estimation unit 153.

[0088] 19, the output control unit 154 causes the captured image to be displayed on the screen with a 3D bounding box surrounding each fish detected from the captured image and a label indicating the type of each detected fish superimposed thereon, as the estimation result estimated by the estimation unit 153. Note that the output control unit 154 may also cause the captured image to be displayed on the screen with a label indicating the proportion of the area of ​​the visible region of each detected fish superimposed thereon, as the estimation result estimated by the estimation unit 153.

[0089] In the example shown in Figure 20, the output control unit 154 displays on the screen the captured image with a two-dimensional bounding box added and the number of individually identified fish tails (the number of tails in a school of fish), "146," as the estimation result estimated by the estimation unit 153.

[0090] In the above-described first embodiment, the object included in the simulation image is a fish, and the multiple objects are multiple fish included in a school of fish. However, the object in the simulation image is not limited to fish. For example, the object in the simulation image may be a person, an animal, a vehicle, etc., other than fish. Furthermore, the information processing device 100 may generate a simulation image that visually displays information indicating the degree of overlap of multiple objects (e.g., multiple people, multiple animals, multiple vehicles, etc.) located in a three-dimensional simulation space in a two-dimensional simulation image captured by a virtual camera.

[0091] 2. Second embodiment

[0092] 2-1. Introduction

[0093] In the above-described first embodiment, a case has been described in which the information processing device 100 generates a simulation image in which a school of fish located in a three-dimensional simulation space is captured by a virtual camera, instead of a photographed image of a school of fish actually existing in a three-dimensional space (for example, in a fish tank). However, there is room for improvement in the technology for generating the school behavior of fish contained in the simulation image. Therefore, in the second embodiment, the information processing device 100A performs reinforcement learning or deep reinforcement learning on a machine learning model that determines the school behavior of fish arranged in a three-dimensional simulation space (hereinafter also referred to as a virtual space). Then, a case will be described in which the information processing device 100A determines the school behavior of fish using the machine learning model that has been subjected to reinforcement learning or deep reinforcement learning.

[0094] [2-2. Configuration of information processing device]

[0095] 21 is a diagram showing an example of the configuration of an information processing device 100A according to the second embodiment. The information processing device 100A includes a communication unit 110, a storage unit 120, an input unit 130, an output unit 140, and a control unit 150A. Note that in the information processing device 100A, explanations of the same points as those in the information processing device 100 will be omitted as appropriate.

[0096] (Control unit 150A)

[0097] The control unit 150A is a controller, and is realized, for example, by a CPU, an MPU, a GPU, etc., executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100A using a RAM as a working area. The control unit 150A is also a controller, and is realized, for example, by an integrated circuit such as an ASIC or an FPGA.

[0098] 21, the control unit 150A has an acquisition unit 151A, a generation unit 152A, a determination unit 153A, an estimation unit 154A, and an output control unit 155A, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 150A is not limited to the configuration shown in FIG. 21, and may be other configurations as long as they perform the information processing described below.

[0099] (Acquisition part 151A)

[0100] The acquisition unit 151A acquires individual information indicating behavioral characteristics of individuals belonging to a school of fish to be processed arranged in a virtual space. Specifically, the acquisition unit 151A acquires individual information input by a user via the input unit 130. For example, the acquisition unit 151A acquires, as the individual information, information indicating the position of the individual in the virtual space, the movement speed, the movement direction, and the body length of the individual. The acquisition unit 151A also acquires, as the individual information, information indicating the water temperature and illuminance preferred by the individual. The acquisition unit 151A also acquires the individual information of each individual belonging to the school of fish to be processed.

[0101] The acquisition unit 151A also acquires relative information indicating characteristics of the behavior of other individuals located around the individual belonging to the school of fish to be processed relative to the individual. Specifically, the acquisition unit 151A acquires relative information input by the user via the input unit 130. For example, the acquisition unit 151A acquires, as the relative information, information on other individuals located within a distance from the position of the individual that is a predetermined multiple of the individual's body length (for example, a distance three times or twice the body length of the individual). The acquisition unit 151A also acquires, as the relative information, information indicating the relative positions and relative speeds of the other individuals with respect to the individual. The acquisition unit 151A acquires the relative information of each individual belonging to the school of fish to be processed.

[0102] The acquisition unit 151A also acquires environmental information indicating characteristics of the environment surrounding an individual belonging to the school of fish to be processed. Specifically, the acquisition unit 151A acquires environmental information input by a user via the input unit 130. For example, the acquisition unit 151A acquires, as the environmental information, information on a space within a distance from the position of the individual that is a predetermined multiple of the individual's body length (for example, a distance three times or twice the individual's body length). The acquisition unit 151A also acquires, as the environmental information, information indicating the temperature and illuminance in the space surrounding the individual. The acquisition unit 151A acquires the environmental information of each individual belonging to the school of fish to be processed.

[0103] (Generation section 152A)

[0104] The generating unit 152A generates a reinforcement learning model, which is a machine learning model that has been subjected to reinforcement learning or deep reinforcement learning, so as to output the behavior of an individual belonging to a school of fish to be learned when individual information indicating the characteristics of the behavior of the individual belonging to the school of fish to be learned, relative information indicating the characteristics of the behavior of other individuals located around the individual belonging to the school of fish to be learned, and environmental information indicating the characteristics of the environment around the individual belonging to the school of fish to be learned, based on a reward given to the behavior of the individual belonging to the school of fish to be learned arranged in a virtual space. Here, deep reinforcement learning is a learning method that combines deep learning such as so-called deep learning and reinforcement learning. For example, the reinforcement learning model may include a DNN (Deep Neural Network), which is a neural network having an input layer, multiple intermediate layers, and an output layer, and may determine the behavior of the individual belonging to the school of fish based on information output from the output layer when input information is input from the input layer. Specifically, the generating unit 152A trains the reinforcement learning model so as to maximize the reward r expressed by the following formula (1).

[0105]

number

[0106] The reward r expressed in the above formula (1) is the negative reward r given to an individual belonging to the learning target fish school when it collides with a boundary in the virtual space. BC Here, the virtual space is, for example, the three-dimensional simulation space in the first embodiment. Also, the boundary in the virtual space includes the boundary with the wall (net) of the fish pen placed in the virtual space and the boundary with the water surface of the fish pen.

[0107] The reward r is a negative reward r given to an individual belonging to the learning fish school when it collides with other individuals located around it. NC For example, negative reward r NC is added in proportion to the number of times an individual collides with other individuals located around it.

[0108] The reward r is given to an individual belonging to the learning target fish school when it avoids collisions with the boundary in the virtual space. BD For example, the reward r BD is given to the behavior of an individual belonging to a school of fish keeping a predetermined distance from the boundary of the sensing range where the individual can sense information about its surroundings. Here, the sensing range where an individual can sense information about its surroundings is, for example, within a distance from the individual's position that is a predetermined multiple of the individual's body length (for example, a distance three times or twice the individual's body length).

[0109] The reward r is given when an individual belonging to the learning fish school is close to other individuals around the individual and aligns its orientation with the orientation of the other individuals. ND For example, the reward r ND is given to individuals belonging to a school of fish when they move close to other individuals within a range where they can detect information about their surroundings. ND is given for the behavior of an individual fish belonging to a school of fish aligning its own orientation with that of other individuals located within a range in which it can detect information about its surroundings.

[0110] The reward r is the negative reward r given to an individual belonging to the learning fish school for the behavior of rotating. E Includes negative rewards r E is given to the individual when it rotates its body. Also, the negative reward r E is given to the behavior of an individual rotating its body and moving forward like a drill.

[0111] The reward r is a negative reward r given to individuals belonging to the learning fish school who move at a speed exceeding the first speed (speed threshold). M Includes negative rewards r M is given to the behavior of the individual moving at a speed exceeding the first speed (speed threshold). Also, a negative reward r M is also given to the behavior of the individual fish belonging to the learning fish school that changes its pitch angle to an angle beyond the first pitch angle. For example, a negative reward r M is given for actions in which an individual suddenly changes its orientation relative to the direction of travel.

[0112] The reward r is a reward given to an individual belonging to a school of fish to be trained when it chases other individuals located around the individual, or a negative reward given to an individual when it escapes from other individuals that are chasing the individual. C Includes.

[0113] In addition, the generation unit 152A trains the reinforcement learning model by reinforcement learning or deep reinforcement learning. For example, the generation unit 152A trains the reinforcement learning model by a proximal policy optimization (PPO) algorithm expressed by the following formula (2). CLIP (θ) is the surrogate objective function, π θ is the policy before the update, π θold is the updated policy, l t (θ) is the policy before the update π θ and the updated policy π θold The ratio of is shown.

[0114]

number

[0115] (Decision section 153A)

[0116] The determination unit 153A determines the behavior of individuals belonging to the school of fish to be processed, based on the individual information, relative information, and environmental information acquired by the acquisition unit 151 A. Specifically, the determination unit 153A determines the behavior of individuals belonging to the school of fish to be processed, using the reinforcement learning model generated by the generation unit 152A.

[0117] Furthermore, the generating unit 152A generates a simulation image including the behavior of individuals belonging to the school of fish to be processed determined by the determining unit 153A.

[0118] In addition, when the generation unit 152A receives a generated simulation image as input, it generates an estimation model, which is a machine learning model trained to output correct answer data or information corresponding to the correct answer data generated based on the parameter information used to generate the simulation image.

[0119] (Estimation part 154A)

[0120] The estimation unit 154A estimates information about a school of fish included in a captured image from a captured image of a school of fish located in real space. Specifically, the estimation unit 154A estimates information about the school of fish from the captured image using the estimation model generated by the generation unit 152A.

[0121] (Output control unit 155A)

[0122] The output control unit 155A is

[0123] In the above-described second embodiment, the information processing device 100A has been described as determining the behavior of individuals belonging to a school of fish, but the information processing device 100A may determine the behavior of individuals belonging to a school of organisms that behave in a school other than fish. For example, the information processing device 100A may determine the behavior of individuals belonging to a school of organisms that are birds, bats, elephants, deer, humans, or virtual organisms (such as 3D CG or characters of fish).

[0124] In the above-described second embodiment, the information processing device 100A generates a simulation image including the behavior of an individual belonging to a school of fish determined by the determination unit 153A. However, the application example of the behavior of an individual belonging to a school of fish determined by the determination unit 153A is not limited to an image. For example, the information processing device 100A may output the behavior of an individual belonging to a school of creatures determined by the determination unit 153A to a creature-type robot located in a real space. For example, the information processing device 100A may output the behavior of an individual belonging to a school of fish determined by the determination unit 153A to a fish-type robot located in an aquarium in the real space.

[0125] 3. Effects

[0126] As described above, the information processing device 100A according to the second embodiment includes an acquisition unit 151A and a determination unit 153A. The acquisition unit 151A acquires individual information indicating the characteristics of the behavior of an individual belonging to a group of organisms to be processed arranged in a virtual space, relative information indicating the characteristics of the behavior of other individuals located around the individual belonging to the group of organisms to be processed relative to the individual, and environmental information indicating the characteristics of the environment around the individual belonging to the group of organisms to be processed. The determination unit 153A determines the behavior of the individual belonging to the group of organisms to be processed based on the individual information, relative information, and environmental information acquired by the acquisition unit 151A.

[0127] Thereby, the information processing device 100A can automatically generate the behavior of each individual belonging to a group of organisms. Therefore, the information processing device 100A can automatically generate group behavior of organisms. Furthermore, since the information processing device 100A can automatically generate group behavior of organisms, it can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilience, innovate and innovate." Furthermore, since the information processing device 100A can automatically generate group behavior of organisms, it can automatically generate a simulation image including group behavior of organisms. Furthermore, the information processing device 100A can train a machine learning model that estimates information on group behavior of organisms included in an image from an image using the generated simulation image including group behavior of organisms. Thus, the information processing device 100A can improve the estimation accuracy of the machine learning model that estimates information on group behavior of organisms included in an image from an image. In addition, the information processing device 100A can improve the estimation accuracy of a machine learning model that estimates information about the school behavior of fish contained in an image from an image, for example, and can contribute to the achievement of Goal 14 of the Sustainable Development Goals (SDGs), "Conserve and sustainably use the oceans and marine life."

[0128] The information processing device 100A further includes a generating unit 152A. The generating unit 152A generates a reinforcement learning model, which is a machine learning model that has been reinforced to output the behavior of the individual belonging to the flock of organisms to be learned, when individual information indicating the characteristics of the behavior of the individual belonging to the flock of organisms to be learned, relative information indicating the characteristics of the behavior of other individuals located around the individual belonging to the flock of organisms to be learned, and environmental information indicating the characteristics of the environment around the individual belonging to the flock of organisms to be learned, based on a reward given to the behavior of the individual belonging to the flock of organisms to be learned arranged in a virtual space. The determining unit 153A uses the reinforcement learning model generated by the generating unit 152A to determine the behavior of the individual belonging to the flock of organisms to be processed.

[0129] As a result, the information processing device 100A can appropriately determine the behavior of individuals belonging to a group of living things by using a reinforcement learning model.

[0130] The rewards also include a negative reward that is given when an individual member of the flock of organisms being studied collides with a boundary in the virtual space.

[0131] This allows the information processing device 100A to train a reinforcement learning model so that individuals belonging to a flock of living things refrain from actions that would cause them to collide with boundaries in the virtual space.

[0132] The rewards also include a negative reward that is given to an individual belonging to the group of organisms being studied when the individual collides with other individuals located around the individual.

[0133] This allows the information processing device 100A to train a reinforcement learning model so that an individual belonging to a flock of living things refrains from a behavior of colliding with other individuals located around the individual.

[0134] The rewards also include rewards that are given to individuals belonging to the flock of organisms being studied when they avoid collision with a boundary in the virtual space.

[0135] This allows the information processing device 100A to train a reinforcement learning model so that individuals belonging to a flock of living things are more likely to take actions to avoid collisions with boundaries in the virtual space.

[0136] The rewards also include rewards given when an individual belonging to the group of organisms being studied positions itself close to other individuals positioned around it and aligns its orientation with that of the other individuals.

[0137] As a result, the information processing device 100A can train a reinforcement learning model so that an individual belonging to a group of organisms is located near other individuals located around the individual, and the individual behaves to align its orientation with the orientation of the other individuals.

[0138] The rewards also include a negative reward that is given to an individual belonging to the group of organisms being studied when the individual rotates on its axis.

[0139] This allows the information processing device 100A to train a reinforcement learning model so that individuals belonging to a flock of living things refrain from rotating on their own axes.

[0140] The rewards also include a negative reward that is given to an individual belonging to the group of organisms being learned when the individual moves at a speed that exceeds the first speed.

[0141] This enables the information processing device 100A to train the reinforcement learning model so that individuals belonging to a flock of living things refrain from a behavior of moving at a speed exceeding the first speed.

[0142] The reward also includes a negative reward that is given to an individual belonging to the group of organisms to be learned when the individual changes its pitch angle to an angle that exceeds the first pitch angle.

[0143] Thereby, the information processing device 100A can train the reinforcement learning model so that each individual belonging to the group of living things refrains from the behavior of changing the pitch angle of the individual to an angle exceeding the first pitch angle.

[0144] The rewards also include rewards given to individuals belonging to the group of organisms being studied for the behavior of chasing other individuals located around the individual, and negative rewards given to individuals for the behavior of running away from other individuals chasing the individual.

[0145] In this way, the information processing device 100A can train a reinforcement learning model so that an individual belonging to a group of living things is likely to take a behavior of chasing other individuals located around the individual. Also, the information processing device 100A can train a reinforcement learning model so that an individual refrains from a behavior of running away from other individuals chasing the individual.

[0146] The individual information also includes information indicating the individual's position in the virtual space, the individual's movement speed, movement direction, and body length.

[0147] This allows the information processing device 100A to appropriately determine the behavior of each individual belonging to a group of living things, based on the position, movement speed, movement direction, and body length of each individual.

[0148] The individual information also includes information indicating the water temperature and illuminance preferred by the individual.

[0149] This allows the information processing device 100A to appropriately determine the behavior of each individual member of the group of living things, based on the water temperature and illuminance preferred by each individual.

[0150] Furthermore, relative information is information about other individuals located within a distance range of a predetermined multiple of the individual's body length from the position of the individual.

[0151] This allows the information processing device 100A to appropriately determine the behavior of each individual belonging to a group of living things, based on the range in which the individual can detect information about its surroundings.

[0152] The relative information also includes information indicating the relative position and relative speed of other individuals with respect to the individual.

[0153] This allows the information processing device 100A to appropriately determine the behavior of an individual belonging to a flock of living things, based on the relative positions and relative speeds of other individuals with respect to the individual.

[0154] Moreover, the environmental information is information on a space within a distance range of a predetermined multiple of the individual's body length from the position of the individual.

[0155] This allows the information processing device 100A to appropriately determine the behavior of each individual belonging to a group of living things, based on the range in which the individual can detect information about its surroundings.

[0156] The environmental information also includes information indicating the temperature and illuminance in the space surrounding the individual.

[0157] This allows the information processing device 100A to appropriately determine the behavior of an individual belonging to a group of living things, based on the temperature and illuminance in the space around the individual.

[0158] Furthermore, the generating unit 152A generates a simulation image including the behavior of individuals belonging to the group of organisms to be processed determined by the determining unit 153A.

[0159] This enables the information processing device 100A to automatically generate a simulation image including the group behavior of living things.

[0160] The information processing device 100A further includes an estimation unit 154A. The estimation unit 154A estimates information about a group of living things included in a captured image from a captured image of a group of living things located in a real space. When a simulation image is input, the generation unit 152A generates an estimation model, which is a machine learning model trained to output correct answer data or information corresponding to the correct answer data generated based on parameter information used to generate the simulation image. The estimation unit 154A estimates information about the group of living things from the captured image using the estimation model.

[0161] Thereby, the information processing device 100A can train a machine learning model that estimates information on the group behavior of organisms included in an image from an image, using a simulation image including the group behavior of organisms generated by the information processing device 100A. As a result, the information processing device 100A can improve the estimation accuracy of the machine learning model that estimates information on the group behavior of organisms included in an image from an image.

[0162] The creature is also a fish.

[0163] This makes it possible for the information processing device 100A to automatically generate school behavior of fish.

[0164] [4. Hardware Configuration]

[0165] Moreover, the information processing device such as the information processing device 100 according to the first embodiment and the information processing device 100A according to the second embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 22, for example. Fig. 22 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device such as the information processing device 100 and the information processing device 100A. The information processing device 100A according to the second embodiment will be described below as an example. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0166] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0167] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0168] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0169] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0170] For example, when the computer 1000 functions as the information processing device 100A according to the second embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 150A by executing programs loaded onto the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from other devices via a predetermined communication network.

[0171] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the Disclosure of the Invention section.

[0172] [5. Other]

[0173] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0174] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0175] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0176] According to one aspect of the embodiment, it is possible to automatically generate swarm behavior of living organisms.

[0177] Although the present invention has been described with respect to specific embodiments for the purpose of a complete and clear disclosure, the appended claims should not be construed as being limited thereto, but rather as embodying all modifications and alternative configurations that may occur to those skilled in the art within the reasonable scope of the basic teachings of this specification.

Claims

1. A generation unit that generates a reinforcement learning model, which is a machine learning model that has been reinforced to output the behavior of an individual belonging to a flock of organisms to be learned when individual information indicating the behavior characteristics of the individual belonging to the flock of organisms to be learned arranged in a virtual space, relative information indicating the behavior characteristics of other individuals located around the individual belonging to the flock of organisms to be learned relative to the individual, and environmental information indicating the characteristics of the environment around the individual belonging to the flock of organisms to be learned, are input based on rewards given to the behavior of the individual belonging to the flock of organisms to be learned; An information processing device comprising:

2. An acquisition unit that acquires individual information indicating behavioral characteristics of individuals belonging to a flock of organisms to be processed arranged in a virtual space, relative information indicating behavioral characteristics of other individuals located around the individual belonging to the flock of organisms to be processed relative to the individual, and environmental information indicating characteristics of the environment around the individual belonging to the flock of organisms to be processed; A determination unit that determines a behavior of an individual belonging to the group of organisms to be processed based on the individual information, the relative information, and the environmental information acquired by the acquisition unit; Further equipped with The determination unit is determining a behavior of an individual belonging to the group of organisms to be processed using the reinforcement learning model generated by the generation unit; The information processing device according to claim 1 .

3. The reward includes a negative reward given to an individual belonging to the group of organisms to be learned when the individual collides with a boundary in the virtual space. The information processing device according to claim 1 .

4. The reward includes a negative reward given to an individual belonging to the group of organisms to be learned when the individual collides with another individual located around the individual. The information processing device according to claim 1 .

5. The reward includes a reward to be given to an individual belonging to the group of organisms to be learned for a behavior of avoiding a collision with a boundary in the virtual space. The information processing device according to claim 1 .

6. The reward includes a reward given when an individual belonging to the group of organisms to be learned is located near other individuals located around the individual, and the individual aligns its orientation with the orientation of the other individuals. The information processing device according to claim 1 .

7. The reward includes a negative reward given to an individual belonging to the group of organisms to be learned for a behavior of rotating on its own axis. The information processing device according to claim 1 .

8. The reward includes a negative reward given to an individual belonging to the group of organisms to be learned when the individual moves at a speed exceeding a first speed. The information processing device according to claim 1 .

9. The reward includes a negative reward given to an individual belonging to the group of organisms to be learned when the individual changes its pitch angle to an angle exceeding a first pitch angle. The information processing device according to claim 1 .

10. The reward includes a reward given to an individual belonging to the group of organisms to be learned when the individual chases another individual located around the individual, and a negative reward given to the individual when the individual runs away from the other individual chasing the individual. The information processing device according to claim 1 .

11. The individual information includes information indicating the position of the individual in the virtual space, the moving speed, the moving direction, and the body length of the individual.

3. The information processing device according to claim 1 or 2.

12. The individual information includes information indicating a water temperature and illuminance preferred by the individual.

3. The information processing device according to claim 1 or 2.

13. The relative information is information about the other individual located within a distance range of a predetermined multiple of the body length of the individual from the position of the individual.

3. The information processing device according to claim 1 or 2.

14. The relative information includes information indicating a relative position and a relative velocity of the other individual with respect to the individual.

3. The information processing device according to claim 1 or 2.

15. The environmental information is information on a space within a distance range of a predetermined multiple of a body length of the individual from the position of the individual.

3. The information processing device according to claim 1 or 2.

16. The environmental information includes information indicating a temperature and an illuminance in a space surrounding the individual.

3. The information processing device according to claim 1 or 2.

17. A generator for generating a simulation image including the behavior of an individual belonging to the group of organisms to be processed determined by the determiner, The information processing device according to claim 2 .

18. An estimation unit that estimates information about a group of living organisms contained in a captured image of the group of living organisms located in a real space from the captured image, The generation unit is generating an estimation model, which is a machine learning model trained to output, when the simulation image is input, correct answer data generated based on parameter information used to generate the simulation image or information corresponding to the correct answer data; The estimation unit is using the estimation model to estimate information about the group of organisms from the captured images; The information processing device according to claim 17.

19. The organism is a fish.

3. The information processing device according to claim 1 or 2.

20. An information processing method executed by an information processing device, comprising: A reinforcement learning model is generated, which is a machine learning model that has been reinforced to learn to output the behavior of an individual belonging to a flock of organisms to be learned when individual information indicating the behavior characteristics of the individual belonging to the flock of organisms to be learned, relative information indicating the behavior characteristics of other individuals located around the individual belonging to the flock of organisms to be learned relative to the individual, and environmental information indicating the characteristics of the environment around the individual belonging to the flock of organisms to be learned, based on a reward given to the behavior of the individual belonging to the flock of organisms to be learned that is arranged in a virtual space, is input. Information processing methods.

Images