Image generation method and image generation program

Abstract

This makes it possible to identify the cause of the algae decline. [Solution] The computer acquires first seaweed bed data showing the algae in the underwater area at a first time point, second seaweed bed data showing the algae in the underwater area at a second time point later than the first time point, and environmental data showing the environment of the underwater area at at least one of the first and second time points. Based on the first and second seaweed bed data, if the computer detects a decrease in the algae in the underwater area at the second time point compared to the first time point, it determines the cause of the decrease based on the environmental data. The computer generates image data showing the topography of the underwater area, which includes an algae object corresponding to the determined cause from among multiple types of algae objects, each representing an abnormal state of algae.

Description

Technical Field

[0001] The present invention relates to an image generation method and an image generation program.

Background Art

[0002] Algae such as seaweed and seagrass are expected as natural resources that absorb carbon dioxide, and are sometimes referred to as blue carbon. Accordingly, investigators such as administrative agencies and research institutions may periodically survey algae growing underwater. The survey results may be used to formulate environmental conservation activities for preserving blue carbon.

[0003] For example, there is a data processing system that uses an underwater camera to capture images of seaweed, analyzes the image data, and estimates the carbon dioxide absorption amount of the seaweed. This data processing system captures images of seaweed from a plurality of different imaging directions, and records the coordinates, angle, and water depth of the underwater camera together with the image data. The data processing system generates a three-dimensional model representing the three-dimensional shape of the seaweed from the recorded data, and estimates the carbon dioxide absorption amount using the three-dimensional model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0005] To support user understanding, an information processing apparatus is contemplated to visualize seaweed bed data representing underwater algae as image data. Algae may decrease for various reasons, such as predation by predatory organisms and poor growth due to water quality changes. Therefore, different image data corresponding to seaweed bed data at different points in time may indicate that algae has decreased in a certain underwater area.

[0006] However, in the process of measuring algae using a sensor device or analyzing the measurement data, the information processing device may only recognize algae in a normal state as "algae," and may not recognize algae in an abnormal state, such as those that have been preyed upon. Therefore, it can be difficult for users to identify the cause of the algae decline simply by looking at the image data. Thus, in one aspect, the present invention aims to enable the identification of the cause of the algae decline. [Means for solving the problem]

[0007] One aspect of the provided image generation method involves a computer performing a process to acquire first seaweed bed data showing algae in an underwater area at a first time point, second seaweed bed data showing algae in an underwater area at a second time point later than the first time point, and environmental data showing the environment of the underwater area at at least one of the first and second time points. If a decrease in the amount of algae in the underwater area at the second time point is detected based on the first and second seaweed bed data, the computer determines the cause of the decrease based on the environmental data, and generates image data showing the topography of the underwater area, which includes an algae object corresponding to the determined cause from among multiple types of algae objects, each representing an abnormal state of algae. [Effects of the Invention]

[0008] In one respect, it is possible to identify the cause of the decline in algae. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the information processing device of the first embodiment. [Figure 2] This figure shows an example of the hardware of the information processing device according to the second embodiment. [Figure 3] This figure shows an example of the reduction of algae objects within a digital twin. [Figure 4] This figure shows an example of seaweed bed data. [Figure 5]This figure shows an example of environmental data. [Figure 6] This figure shows an example of the data flow for generating a digital twin image. [Figure 7] This figure shows examples of algae object placements according to the cause of reduction. [Figure 8] This is a block diagram showing examples of functions of an information processing device. [Figure 9] This flowchart shows an example of a data analysis procedure. [Figure 10] This is a flowchart showing an example of the visualization procedure. [Modes for carrying out the invention]

[0010] This embodiment will be described below with reference to the drawings. (a) First embodiment Figure 1 is a diagram illustrating an information processing device according to a first embodiment. The information processing device 10 of the first embodiment visualizes the results of an aquatic algae survey as image data. The information processing device 10 may be a client device or a server device. The information processing device 10 may also be called a computer or an image generation device.

[0011] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile memory such as Random Access Memory (RAM). Alternatively, the storage unit 11 may be a non-volatile storage such as a Hard Disk Drive (HDD) or Solid State Drive (SSD).

[0012] The processing unit 12 is, for example, a processor such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or a Digital Signal Processor (DSP). However, the processing unit 12 may include an electronic circuit such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). The processor executes, for example, a program stored in a memory such as a RAM. The processor may be referred to as Processor Circuitry. Also, a set of processors may be referred to as a multiprocessor or simply a "processor". Different processes among a plurality of processes to be described later may be executed by different processors.

[0013] The storage unit 11 stores seaweed bed data 13, 14 and environmental data 15. The seaweed bed data 13 indicates algae in an underwater region at a first time point. The seaweed bed data 14 indicates algae in the same underwater region as the seaweed bed data 13 at a second time point that is later than the first time point. The second time point is, for example, half a year or one year after the first time point.

[0014] "Algae" in the first embodiment is algae in a broad sense that grows in water and is visible to humans, and may include Algae and Waterweed. Algae include Seaweed that grows in seawater areas and other algae that grow in freshwater areas. Waterweeds include Seagrass that grows in seawater areas and other waterweeds that grow in freshwater areas. As blue carbon, seaweed and seagrass are mainly assumed in some cases. A seaweed bed is a colony of algae. The seaweed bed may include a Seaweed Bed and a Seagrass Bed.

[0015] The seaweed bed data 13 and 14 each indicate the results of on-site surveys of seaweed. For the measurement of seaweed, a sensor device such as an underwater camera may be used, and an underwater drone equipped with a sensor device may be used. The information processing apparatus 10 or another information processing apparatus may generate the seaweed bed data 13 and 14 by analyzing the measurement data of the sensor device. Due to the nature of the sensor device or the nature of data analysis, the information processing apparatus 10 or another information processing apparatus may only recognize seaweed in a normal state and not recognize seaweed in an abnormal state. Therefore, the seaweed bed data 13 and 14 may only indicate seaweed in a normal state.

[0016] The underwater region includes sections of the seabed where seaweed can grow. For example, the underwater region is a subdivided unit region such as a 5-meter × 5-meter × 5-meter cubic space including the seabed portion. The seaweed bed data 13 and 14 may each indicate the type of seaweed growing in the underwater region. Further, the seaweed bed data 13 and 14 may each indicate the presence or absence of seaweed in the underwater region.

[0017] Further, the seaweed bed data 13 and 14 may each indicate the amount of seaweed, such as the density or volume of seaweed in the underwater region. The density is the proportion of the seabed covered by seaweed, and may be referred to as coverage. Further, the seaweed bed data 13 and 14 may each include an index value related to density or volume, such as carbon dioxide absorption. The seaweed bed data 13 or the seaweed bed data 14 may indicate that no seaweed is present in the underwater region.

[0018] The environmental data 15 indicates the environment of the same underwater region as the seaweed bed data 13 and 14 at at least one of the first time point and the second time point. The environmental data 15 may include first environmental data indicating the environment of the underwater region at the first time point, and may include second environmental data indicating the environment of the underwater region at the second time point.

[0019] Environmental data 15 may include indicator values ​​related to water quality, such as light intensity, water temperature, or nutrient concentration. Environmental data 15 may also include the number of predatory organisms. Predatory organisms are, for example, purple sea urchins or green sea turtles. The environment may be measured in the field in conjunction with algae. Sensor devices such as illuminance sensors, temperature sensors, conductivity sensors, or underwater cameras may be used to measure the environment, and underwater drones equipped with sensor devices may be used. Information processing device 10 or other information processing device may generate environmental data 15 by analyzing measurement data from sensor devices.

[0020] The processing unit 12 acquires seaweed bed data 13, 14 and environmental data 15. Based on the seaweed bed data 13, 14, the processing unit 12 detects a decrease in the amount of algae in the underwater area at the second time point compared to the first time point. The decrease may include disappearance in the strict sense or reduction in the strict sense. Disappearance in the strict sense is a situation in which algae were present in the underwater area at the first time point, but are absent at the second time point. Reduction in the strict sense is a situation in which algae are present in the underwater area at both the first and second time points, but the amount of algae has decreased.

[0021] For example, if seaweed bed data 13 indicates the presence of algae and seaweed bed data 14 indicates the absence of algae, the processing unit 12 detects the decrease. Also, for example, if the density of seaweed bed data 14 is lower than that of seaweed bed data 13, the processing unit 12 detects the decrease. Also, for example, if the volume of seaweed bed data 14 is lower than that of seaweed bed data 13, the processing unit 12 detects the decrease. Also, for example, if the amount of carbon dioxide absorbed by seaweed bed data 14 is lower than that of seaweed bed data 13, the processing unit 12 detects the decrease.

[0022] If a decrease is detected, the processing unit 12 determines the cause of the decrease based on the environmental data 15. For example, the processing unit 12 calculates the algal growth rate in that underwater area using the water quality index value at the second time point. If the growth rate is below a threshold (for example, less than 1), the processing unit 12 determines that the cause is poor growth due to changes in water quality. The decrease in seaweed beds due to changes in water quality is sometimes called "isoyake" (seaweed barrenness). Also, for example, if the number of predatory organisms at the second time point is above a threshold, and the number of organisms at the second time point is above the number of organisms at the first time point, the processing unit 12 determines that the cause is predation by predatory organisms.

[0023] The processing unit 12 generates image data 16 based on the determined result. Image data 16 is visualization data showing the topography of the underwater region. Image data 16 includes algae objects 17. Algae objects 17 are algae objects corresponding to the determined cause, from among several types of algae objects, each representing an abnormal state of algae. Algae objects 17 represent the abnormal state of algae using visual information such as shape, color, or pattern. For example, the processing unit 12 places the algae objects 17 on the topography of the underwater region.

[0024] Multiple types of algae objects may include algae objects that represent algae that have withered due to poor growth. If the determined cause is poor growth due to a change in water quality, the processing unit 12 may select such algae object. Also, multiple types of algae objects may include algae objects that represent algae that have been preyed upon. If the determined cause is predation by a predator, the processing unit 12 may select such algae object.

[0025] The processing unit 12 may generate image data 16 using a digital twin. A digital twin is a three-dimensional shape model that reproduces the topography of an underwater region in a three-dimensional virtual space. For example, the processing unit 12 places a three-dimensional shape model representing an algae object 17 into the digital twin. The processing unit 12 generates a two-dimensional image as image data 16, which is drawn when the digital twin after placement is viewed from a certain direction.

[0026] The processing unit 12 outputs the generated image data 16. The processing unit 12 may save the image data 16 to a non-volatile storage device. The processing unit 12 may also display the image data 16 on a display device connected to the information processing device 10. The processing unit 12 may also transmit the image data 16 to another information processing device.

[0027] As described above, the information processing device 10 of the first embodiment acquires seaweed bed data 13, 14 and environmental data 15. The seaweed bed data 13 shows the algae in the underwater area at a first time point. The seaweed bed data 14 shows the algae in the underwater area at a second time point, which is later than the first time point. The environmental data 15 shows the environment of the underwater area at at least one of the first and second time points. Based on the seaweed bed data 13, 14, if the information processing device 10 detects a decrease in the algae in the underwater area at the second time point compared to the first time point, it determines the cause of the decrease based on the environmental data 15. The information processing device 10 generates image data 16. The image data 16 shows the topography of the underwater area and includes an algae object 17 corresponding to the determined cause from among a plurality of types of algae objects, each representing an abnormal state of algae.

[0028] By viewing the image data 16, users can identify the cause of the algae decline in a particular underwater area. Therefore, users do not need to conduct additional investigations by experts for cause analysis. Consequently, users can quickly and inexpensively implement measures using the seaweed bed data 13,14, such as algae conservation activities.

[0029] (b) Second embodiment Figure 2 shows an example of the hardware of the information processing device according to the second embodiment. The information processing device 100 of the second embodiment analyzes measurement data from an underwater drone and generates seaweed bed data showing the distribution of algae. The information processing device 100 visualizes the seaweed bed data by placing algae objects on a digital twin that shows the underwater topography. However, different information processing devices may perform the generation of seaweed bed data and the visualization of seaweed bed data. The information processing device 100 corresponds to the information processing device 10 of the first embodiment.

[0030] The information processing device 100 includes a CPU 101, RAM 102, HDD 103, GPU 104, input interface 105, media reader 106, and communication interface 107. The CPU 101 corresponds to the processing unit 12 of the first embodiment. The RAM 102 or HDD 103 corresponds to the storage unit 11 of the first embodiment.

[0031] The CPU 101 is a processor that executes program instructions. The CPU 101 loads the program and data from the HDD 103 into the RAM 102 and executes the program. The information processing device 100 may have multiple processors.

[0032] RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by CPU 101 and data used for calculations by CPU 101. The information processing device 100 may have a type of volatile memory other than RAM.

[0033] The HDD 103 is a non-volatile storage device that stores software programs such as the operating system (OS), middleware, and application software, as well as data. The information processing device 100 may have other types of non-volatile storage, such as an SSD or flash memory.

[0034] The GPU 104 works in conjunction with the CPU 101 to perform image processing and outputs the image to the display device 111 connected to the information processing device 100. The display device 111 is, for example, a Cathode Ray Tube (CRT) display, a liquid crystal display, an organic EL (Electro Luminescence) display, or a projector.

[0035] Furthermore, the GPU 104 may be used as a General Purpose Computing on Graphics Processing Unit (GPGPU). The GPU 104 may execute programs in response to instructions from the CPU 101. The information processing device 100 may have volatile semiconductor memory other than RAM 102 as GPU memory.

[0036] The input interface 105 receives input signals from an input device 112 connected to the information processing device 100. The input device 112 is, for example, a mouse, a touch panel, or a keyboard. Multiple input devices may be connected to the information processing device 100.

[0037] The media reader 106 is a reading device that reads programs and data recorded on the recording medium 113. The recording medium 113 is, for example, a magnetic disk, an optical disk, or semiconductor memory. Magnetic disks include flexible disks (FDs) and HDDs. Optical disks include compact discs (CDs) and digital versatile discs (DVDs). The media reader 106 copies the programs and data read from the recording medium 113 to other recording media such as RAM 102 or HDD 103. The read programs may be executed by the CPU 101.

[0038] The recording medium 113 may be a portable recording medium. The recording medium 113 may be used for distributing programs and data. The recording medium 113 and the HDD 103 may also be referred to as computer-readable recording media.

[0039] The communication interface 107 communicates with other information processing devices via the network 114. The communication interface 107 may also be a wired communication interface connected to a wired communication device such as a switch or router. Alternatively, the communication interface 107 may also be a wireless communication interface connected to a wireless communication device such as a base station or access point.

[0040] Next, we will explain the process of seaweed bed surveys, generation of seaweed bed data, and visualization of seaweed bed data. In the second embodiment, the seaweed mainly consists of seaweed and seagrass that grow in seawater areas. The user surveys the seaweed beds at regular intervals, such as every six months or every year. The seaweed bed surveys are conducted using an underwater drone equipped with various sensor devices.

[0041] In the second embodiment, the underwater drone has an underwater camera, which is an optical sensor usable underwater. The underwater drone scans the seabed by continuously imaging it from above while moving. The underwater drone, or a surface vessel connected to the underwater drone, also has a positioning device for determining its current position at the time of imaging. The coordinates indicating the current position include latitude and longitude, and may also include depth. The positioning device may include a Global Positioning System (GPS) device.

[0042] The information processing device 100 acquires measurement data from an underwater drone. The measurement data includes image data with associated coordinates. Unprocessed measurement data is sometimes called RAW data. The information processing device 100 uses image processing software to sharpen the underwater image data. The image processing software may be a machine learning model.

[0043] The information processing device 100 uses an image recognition model to recognize algae depicted in the enhanced image data. The image recognition model is a pre-trained machine learning model. The image recognition model is a neural network, such as a Deep Convolutional Neural Network (DCNN). The image recognition model detects bounding boxes indicating the regions where algae are depicted from each image frame contained in the image data and determines the class indicating the type of algae. The image recognition model tracks bounding boxes indicating the same algae over time by comparing the positions of the bounding boxes between adjacent image frames.

[0044] The information processing device 100 divides the underwater space, including the seabed, into multiple unit spaces, each measuring 5 meters x 5 meters x 5 meters. Based on the coordinates included in the measurement data, the information processing device 100 classifies the image recognition results into the multiple unit spaces. Based on the image and class of the bounding box classified into a unit space, the information processing device 100 determines the type of algae growing in that unit space and calculates the density of algae in that unit space. Density is the percentage of the seabed surface covered by algae and is sometimes called cover. The information processing device 100 generates seaweed bed data including the type and density of algae.

[0045] The information processing device 100 has a digital twin representing the seabed topography. The digital twin is a three-dimensional shape model that reproduces the real-world topography in a three-dimensional virtual space. The digital twin includes multiple vertices, each corresponding to a real-world coordinate. The digital twin also includes multiple edges connecting two vertices and multiple faces defined by three or more edges. Therefore, the digital twin represents the topography as a set of vertices, edges, and faces.

[0046] Furthermore, the information processing device 100 has multiple algae objects representing different types of algae. The algae objects are three-dimensional shape models that can be placed in the digital twin. The information processing device 100 places the algae objects in the digital twin based on the algae bed data. For example, for each unit space, the information processing device 100 selects an algae object that represents the type of algae indicated by the algae bed data. The information processing device 100 places the selected algae object in the area on the digital twin corresponding to that unit space at the density indicated by the algae bed data.

[0047] The information processing device 100 places a virtual camera on the digital twin and generates a digital twin image drawn from the position of the virtual camera. The information processing device 100 may allow the user to change the position of the virtual camera. The information processing device 100 displays the generated digital twin image on the display device 111.

[0048] Users can understand changes in seaweed beds by comparing different digital twin images corresponding to seaweed bed data from different survey points. In doing so, users may discover a decrease in seaweed in a given area.

[0049] Figure 3 shows an example of the reduction in algae objects within the digital twin. Digital twin image 131 is a visualization of the algae bed data generated by the previous algae bed survey. Digital twin image 132 is a visualization of the algae bed data generated by the current algae bed survey. This algae bed survey was conducted one year after the previous survey.

[0050] In digital twin image 131, algal objects are distributed at a constant density over a wide area. Therefore, digital twin image 131 indicates that algae are growing at a constant density over a wide area of ​​the seabed. On the other hand, in digital twin image 132, algal objects are missing in some areas. Therefore, digital twin image 132 indicates that algae have disappeared in some areas of the seabed.

[0051] Thus, compared to the previous survey, the density of algae may decrease in some unit areas, and in some unit areas, algae may disappear completely. There are several possible causes for the reduction of algae. In the second embodiment, the main possible causes of reduction are predation by predatory organisms and poor growth due to changes in water quality. Within a certain period since the previous survey (for example, within one year), abnormal algae, such as algal debris, may remain in actual seaweed beds. Abnormal algae have different shapes, colors, or patterns depending on the cause of reduction, and can be useful in identifying the cause of reduction.

[0052] When users discover a decrease in algae, they may want to identify the cause. However, image recognition models are trained to recognize normal algae as "algae," and it is difficult for them to recognize abnormal algae as some kind of object. Therefore, the algae bed data does not contain information about abnormal algae. As a result, digital twin image 132 only shows that the algae have decreased compared to digital twin image 131.

[0053] It is difficult for users to identify the cause of algae decline simply by looking at the digital twin image 132. On the other hand, if additional field surveys by experts are conducted to identify the cause of decline, users will incur long survey times and high costs. Therefore, the information processing device 100 uses sensor data obtainable during the algae bed survey in conjunction with image data to estimate the cause of decline and visualizes the estimated cause of decline on the digital twin image.

[0054] Figure 4 shows an example of seaweed bed data. As mentioned above, the information processing device 100 generates seaweed bed data 141 by analyzing image data from an underwater drone. The seaweed bed data 141 associates date and time, coordinates, seaweed species, density, volume, and carbon dioxide storage amount. The date and time is the date and time when the underwater drone imaged the seabed. The coordinates are the coordinates of the vertices in a unit space. In the example in Figure 4, four pairs of latitude and longitude indicate one rectangular region of the seabed.

[0055] The algal species is the class of algae determined by the image recognition model. The density is the coverage calculated from the image recognition results. The volume is the volume of the algae calculated from the algal species and the image recognition results. The carbon dioxide storage amount is the amount of carbon dioxide fixed by the algae, calculated from the algal species and volume. If two or more types of algae are growing in the same unit space, the information processing device 100 may calculate the density, volume, and carbon dioxide storage amount for each algal species. The information processing device 100 may also calculate the density, volume, and carbon dioxide storage amount by simulation using an algal growth model for each algal species.

[0056] Figure 5 shows an example of environmental data. The information processing device 100 generates environmental data 142 by analyzing sensor data from an underwater drone. Environmental data 142 is generated from the same seaweed bed survey measurement data as seaweed bed data 141. Environmental data 142 associates date and time, coordinates, light intensity, water temperature, nutrient concentration, spatial limit, and number of predators. The date and time is the date and time when the underwater drone imaged the seabed, and is the same as that of seaweed bed data 141. The coordinates are the coordinates of the vertices in a unit space, and are the same as those of seaweed bed data 141.

[0057] Light intensity is the amount of light measured in a unit space. For example, an underwater drone has an illuminance sensor that measures light intensity. Raw measurement data includes the measured light intensity. The light intensity shown in environmental data 142 may be the average of light intensity measured at multiple locations in a unit space. In the example in Figure 5, the unit of light intensity is lux (lx), and this light intensity is sometimes called illuminance. When the light is sunlight, the light intensity [lx] and the light intensity [W / m 2 Between ] and , the light intensity [W / m 2 The relationship is given by ] = luminous intensity [lx] × 0.0079. 2 Between ] and light intensity [lx], there is a difference in light intensity [cal / m 2 ]=Light amount [lx]×1.89×10 -3 There is a relationship of × time [s].

[0058] Water temperature is the water temperature measured in a unit space. For example, an underwater drone has a temperature sensor that measures water temperature. Raw measurement data includes the measured water temperature. The water temperature shown in environmental data 142 may be the average of water temperatures measured at multiple locations in a unit space.

[0059] Nutrient concentration is the nutrient concentration measured in a unit space. Nutrient concentration is the mass of nitrogen per liter of seawater and includes at least one of the masses of ammoniacal nitrogen (NH4-N) and nitrate nitrogen (NO3-N). Nutrient concentration may also be the sum of the masses of ammoniacal nitrogen and nitrate nitrogen. The underwater drone may have a conductivity sensor to measure nutrient concentration. Raw measurement data may include the measured nutrient concentration. Nutrient concentration may also be measured separately from a seawater sample. The nutrient concentration shown in environmental data 142 may be the average of nutrient concentrations measured at multiple locations within a unit space.

[0060] The spatial limit indicates the upper limit on the amount of algae that can grow per square meter of seabed. The spatial limit is determined by the types of algae that grow in a unit space. The spatial limit may also be determined by further considering the seabed topography contained within that unit space. In the example in Figure 5, the unit of the spatial limit is the dry weight per square meter [gDW / m²]. 2 ]

[0061] The number of predators is the quantity of predators detected in a unit area during a seaweed bed survey. In the example in Figure 5, the unit for the number of predators is the number of individuals per cubic meter. Examples of predators that feed on algae include the purple sea urchin or the green sea turtle. The number of predators may also be the sum of the number of purple sea urchins and the number of green sea turtles. The predators, along with the algae, are captured in image data generated by the underwater camera of the underwater drone.

[0062] The information processing device 100 detects predators from image data using an image recognition model. The image recognition model detects bounding boxes indicating the regions where predators are depicted in each image frame contained in the image data, and determines the class indicating the type of predator. The information processing device 100 tracks bounding boxes indicating the same predator over time by comparing the positions of the bounding boxes between adjacent image frames.

[0063] The information processing device 100 may detect predators using the same image recognition model as for detecting algae. Alternatively, the image recognition model may detect algae and predators simultaneously. Similar to algae detection, the information processing device 100 classifies the recognition results of predators into multiple unit spaces based on the coordinates included in the measurement data. Based on the bounding boxes and classes classified into a given unit space, the information processing device 100 determines the types of predators inhabiting that unit space and calculates the number of predators in that unit space.

[0064] Figure 6 shows an example of the data flow for generating a digital twin image. The information processing device 100 stores the seaweed bed data 141 from the current survey, the environmental data 142 from the current survey, the seaweed bed data 143 from the previous survey, and the environmental data 144 from the previous survey. The information processing device 100 generates a digital twin image 131 using the seaweed bed data 143 in the manner described above. The information processing device 100 also generates a digital twin image 133 using the seaweed bed data 141, 143 and the environmental data 142, 144.

[0065] The information processing device 100 generates a digital twin image 133 according to the following method. First, the information processing device 100 places algae objects representing normal algae in the digital twin according to the algae species and density indicated by the algae bed data 141. The information processing device 100 detects unit spaces where the algae have decreased among multiple unit spaces. Unit spaces where the algae have decreased are unit spaces where the density of the algae bed data 141 is lower than the density of the algae bed data 143.

[0066] The information processing device 100 refers to environmental data 142 and 144 to determine the cause of the detected decrease in unit space. The data records referred to in environmental data 142 and 144 are data records relating to unit spaces where algae have decreased.

[0067] In the second embodiment, the information processing device 100 uses the number of predators in environmental data 142 and the number of predators in environmental data 144 to determine whether predation by predators is the cause of the decline. For example, the information processing device 100 determines that predation is the cause of the decline if the number of predators in environmental data 142 is above a threshold and the number of predators in environmental data 142 is above the number of predators in environmental data 144.

[0068] In the second embodiment, the information processing device 100 uses the light intensity, water temperature, nutrient concentration, and spatial limitations of the environmental data 142 to determine whether insufficient growth is the cause of the decrease. For example, the information processing device 100 calculates the algal growth rate at the time of the current investigation from the environmental data 142. If the growth rate is below a threshold (for example, less than 1), the information processing device 100 determines that insufficient growth is the cause of the decrease. Generally, a decrease in light intensity reduces the growth rate. Also, an increase in water temperature reduces the growth rate. Also, a decrease in nutrient concentration reduces the growth rate. Also, a lack of available space reduces the growth rate. Generally, in an underwater environment where the calculated growth rate is less than 1, the algae will wither and die.

[0069] The information processing device 100 may calculate the growth rate according to the formula described in the following non-patent document: Takeshi Kanazawa, Yukio Koibuchi, and Masahiko Isobe, "Development and Field Verification of a Seagrass Growth Prediction Model," Journal of Coastal Engineering, Japan Society of Civil Engineers, Vol. 53, pp. 1021-1025, October 10, 2006. As an example, the information processing device 100 calculates the growth rate μ according to the following formula: μ = Maximum growth rate × f1(L) × f2(T) × f3(N) × f4(s).

[0070] The maximum growth rate is determined by the algae species. Function f1(L) shows the effect of light intensity on the growth rate. Function f1(L) includes a parameter indicating saturation light intensity and outputs different values ​​for different light intensities. Function f2(T) shows the effect of water temperature on the growth rate. Function f2(T) includes a parameter indicating optimal water temperature and outputs different values ​​for different water temperatures.

[0071] The function f3(N) shows the effect of nutrient concentration on the growth rate. Function f3(N) includes parameters indicating the minimum internal nitrogen content and the critical internal nitrogen content, outputting different values ​​for different nutrient concentrations. The function f4(s) shows the effect of available space on the growth rate. Function f4(s) outputs a value corresponding to the difference between the space limit and the current space occupancy. The current space occupancy is calculated, for example, from density or volume.

[0072] If the cause of the decrease is predation, the information processing device 100 additionally places algae objects representing the predated algae into the digital twin. For example, the information processing device 100 additionally places a number of algae objects corresponding to the amount of density decrease, i.e., the difference between the density of the seaweed bed data 143 and the density of the seaweed bed data 141.

[0073] Furthermore, if the cause of the decrease is poor growth, the information processing device 100 additionally places algae objects representing algae that have withered due to poor growth into the digital twin. For example, the information processing device 100 additionally places a quantity of algae objects corresponding to the amount of density decrease. Algae objects representing predated algae and algae objects representing algae that have withered due to insufficient growth will differ in shape, color, or pattern. The information processing device 100 generates a digital twin image 133 from the digital twin to which the algae objects representing abnormal algae have been additionally placed.

[0074] However, the information processing device 100 may add algae objects indicating abnormal algae only in unit spaces where algae have completely disappeared. A unit space where algae have completely disappeared is a unit space where the density was positive in the seaweed bed data 143, but the density is zero in the seaweed bed data 141. In this case, the information processing device 100 may add algae objects indicating abnormal algae according to the density in the seaweed bed data 143. In addition, for unit spaces where algae have decreased or disappeared, the information processing device 100 may add a fixed amount of algae objects regardless of the density in the seaweed bed data 141 and 143.

[0075] Figure 7 shows examples of algal object placement according to the cause of reduction. As shown in Figure 3, digital twin image 132 is the image before algal objects indicating abnormal algae were added. Digital twin image 134 is the image when algal objects indicating algae predated by predators were added. Digital twin image 135 is the image when algal objects indicating poorly growing and withered algae were added. Digital twin images 134 and 135 correspond to digital twin image 133 in Figure 6, respectively.

[0076] By looking only at digital twin image 132, it is difficult for users to identify the cause of the decrease in algae in this area. On the other hand, by looking at digital twin image 134 or digital twin image 135, users can identify the cause of the decrease. Therefore, users can use digital twin images 134 and 135 to analyze the relationship between the seabed topography and the growth of algae. Furthermore, users can use digital twin images 134 and 135 to formulate measures such as environmental conservation activities to protect the algae. Next, the functions of the information processing device 100 and the information processing procedure by the information processing device 100 will be described.

[0077] Figure 8 is a block diagram showing an example of the functions of an information processing device. The information processing device 100 includes an image storage unit 121, a sensor data storage unit 122, a survey result storage unit 123, a shape model storage unit 124, an image recognition unit 125, a simulation unit 126, and a visualization unit 127. The image storage unit 121, the sensor data storage unit 122, the survey result storage unit 123, and the shape model storage unit 124 are implemented using, for example, RAM 102 or HDD 103. The image recognition unit 125, the simulation unit 126, and the visualization unit 127 are implemented using, for example, a CPU 101, a GPU 104, and a program.

[0078] The image storage unit 121 stores image data generated by the underwater camera of the underwater drone. The sensor data storage unit 122 stores sensor data other than image data generated by the sensor devices of the underwater drone.

[0079] The survey results storage unit 123 stores seaweed bed data and environmental data for each field survey. The shape model storage unit 124 stores a three-dimensional shape model showing the seabed topography. The shape model storage unit 124 also stores a three-dimensional shape model of a seaweed object representing normal seaweed for each type of seaweed. Furthermore, the shape model storage unit 124 stores a three-dimensional shape model of a seaweed object representing abnormal seaweed for each type of seaweed and each cause of decline.

[0080] The image recognition unit 125 uses image processing software to sharpen the image data stored in the image storage unit 121. The image recognition unit 125 uses an image recognition model to detect algae from the sharpened image data and determines the class that indicates the type of algae detected. The image recognition unit 125 also uses an image recognition model to detect predators from the sharpened image data and determines the class that indicates the type of predator detected.

[0081] The simulation unit 126 analyzes the distribution of algae and the underwater environment. The simulation unit 126 generates seaweed bed data and environmental data using the image recognition results output by the image recognition unit 125 and the sensor data stored in the sensor data storage unit 122. The simulation unit 126 generates seaweed bed data from the algae recognition results. The simulation unit 126 also generates environmental data from the algae recognition results, the predator recognition results, and the sensor data. The simulation unit 126 stores the generated seaweed bed data and the generated environmental data in the survey results storage unit 123.

[0082] The visualization unit 127 reads a three-dimensional shape model of the terrain from the shape model storage unit 124 in response to a user request and generates a digital twin. The visualization unit 127 reads a three-dimensional shape model of normal algae from the shape model storage unit 124 and places the algae objects of normal algae into the digital twin according to the algae bed data stored in the survey results storage unit 123.

[0083] Furthermore, the visualization unit 127 detects unit spaces where algae are decreasing. The visualization unit 127 uses the environmental data stored in the investigation results storage unit 123 to determine the cause of the decrease in the detected unit space. The visualization unit 127 reads a three-dimensional shape model of the abnormal algae corresponding to the cause of the decrease from the shape model storage unit 124 and places the algae object of the abnormal algae into the digital twin. The visualization unit 127 displays the digital twin image on the display device 111.

[0084] The simulation unit 126 may display at least one of the seaweed bed data and environmental data on the display device 111. The simulation unit 126 may also transmit at least one of the seaweed bed data and environmental data to another information processing device. The visualization unit 127 may save the generated digital twin image to a non-volatile storage device such as an HDD 103. The visualization unit 127 may also transmit the generated digital twin image to another information processing device.

[0085] Figure 9 is a flowchart illustrating an example of the data analysis procedure. In step S10, the information processing device 100 reads image data and sensor data from the underwater drone. In step S11, the image recognition unit 125 recognizes algae and predators from the image data. In step S12, the simulation unit 126 determines the unit space in which the recognized algae and predators exist based on the coordinates associated with the image data.

[0086] In step S13, the simulation unit 126 estimates the density, volume, and carbon dioxide storage capacity of algae for each unit space based on the algae recognition results. In step S14, the simulation unit 126 estimates the spatial limit of algae for each unit space based on the algae recognition results. In step S15, the simulation unit 126 estimates the number of predators for each unit space based on the predator recognition results.

[0087] In step S16, the simulation unit 126 generates seaweed bed data from the estimation results of step S13. The simulation unit 126 also generates environmental data from the sensor data and the estimation results of steps S14 and S15. The simulation unit 126 saves the generated seaweed bed data and environmental data.

[0088] Figure 10 is a flowchart showing an example of the visualization procedure. In step S20, the visualization unit 127 reads the seaweed bed data and environmental data for the current survey. In step S21, the visualization unit 127 places normal algae objects into the digital twin according to the seaweed bed data. In step S22, the visualization unit 127 reads the seaweed bed data and environmental data for the previous survey.

[0089] The visualization unit 127 performs the following steps S23 to S29 for each unit space. In step S23, the visualization unit 127 determines whether the algal density has decreased compared to the previous survey. If the algal density has decreased, the process proceeds to step S24. If the algal density has not decreased, the processing for that unit space is completed. The visualization unit 127 may also determine whether the algae that were present in the previous survey have completely disappeared. In that case, the process proceeds to step S24 only if the algae have completely disappeared in that unit space.

[0090] In step S24, the visualization unit 127 determines whether the previous survey date is within a certain period from the current survey date. For example, the visualization unit 127 determines whether the previous survey date is within one year from the current survey date. If the previous survey date is within a certain period, the process proceeds to step S25. If the previous survey date exceeds a certain period, the processing for that unit space is terminated. Note that if the interval between the previous survey date and the current survey date is excessively long, there is a possibility that no abnormal algae remain on the actual seabed. Also, the accuracy of estimating the cause of the decrease may decrease.

[0091] In step S25, the visualization unit 127 determines whether the number of predators is below a threshold or whether the number of predators has decreased compared to the previous survey. If the number of predators satisfies this condition, the visualization unit 127 determines that predation is not the cause of the decrease. Then, the process proceeds to step S27. If the number of predators does not satisfy this condition, the visualization unit 127 determines that predation is the cause of the decrease. Then, the process proceeds to step S26. In step S26, the visualization unit 127 places the algae object of the predated algae at the position corresponding to that unit space in the digital twin. Then, the processing of that unit space is completed.

[0092] In step S27, the visualization unit 127 calculates the algal growth rate from the environmental data of the current survey. In step S28, the visualization unit 127 determines whether the growth rate is 1 or greater. If the growth rate is 1 or greater, the visualization unit 127 determines that the cause of the decrease is not poor growth. Then, the processing of that unit space is completed. If the growth rate is less than 1, the visualization unit 127 determines that the cause of the decrease is poor growth. Then, the process proceeds to step S29.

[0093] In step S29, the visualization unit 127 places the algae object of the poorly growing and withered algae at the position corresponding to that unit space in the digital twin. Then the processing of that unit space is completed. The visualization unit 127 generates a digital twin image from the digital twin generated through the above steps S20 to S29. The visualization unit 127 displays the digital twin image on the display device 111.

[0094] As described above, the information processing device 100 of the second embodiment analyzes measurement data from an underwater drone to generate seaweed bed data, including statistics on the distribution of seaweed. The information processing device 100 places seaweed objects on a digital twin representing the seabed topography according to the seaweed bed data and displays the digital twin image. This allows the user to visually confirm the distribution of seaweed in a three-dimensional virtual space. Therefore, the user can more easily understand the distribution of seaweed at each survey point than by directly viewing the measurement data.

[0095] Furthermore, the information processing device 100 detects the unit space where algae have decreased and determines the cause of the decrease using image data as well as other sensor data. The information processing device 100 adds algae objects corresponding to the cause of the decrease from among the algae objects indicating abnormal algae to the digital twin. This allows the user to identify the cause of the decrease on the digital twin image without additional work such as expert scrutiny of measurement data or additional field surveys by experts. Therefore, the user can quickly and inexpensively implement measures that utilize the results of regular algae bed surveys, such as algae conservation activities. [Explanation of Symbols]

[0096] 10 Information Processing Devices 11 Storage section 12 Processing Units 13,14 Seagrass bed data 15 Environmental Data 16 Image data 17 Algae Objects

Claims

1. The system acquires first seaweed bed data showing algae in the underwater region at a first time point, second seaweed bed data showing algae in the underwater region at a second time point later than the first time point, and environmental data showing the environment of the underwater region at at least one of the first and second time points. Based on the first and second seaweed bed data, if a decrease in the amount of algae in the underwater area at the second time point is detected compared to the first time point, the cause of the decrease is determined based on the environmental data. Image data showing the topography of the underwater region, which generates image data including an algae object corresponding to the determined cause, from among multiple types of algae objects each representing an abnormal state of algae. An image generation method in which a computer performs the processing.

2. The aforementioned environmental data includes the number of individuals of predatory organisms that feed on algae, and the aforementioned multiple types of algae objects include algae objects that indicate predated algae. The image generation method according to claim 1.

3. The environmental data includes first environmental data showing the number of individuals at the first time point and second environmental data showing the number of individuals at the second time point. The process for making the determination includes, if the number of individuals at the second time point is equal to or greater than a threshold, and the number of individuals at the second time point is equal to or greater than the number of individuals at the first time point, the process for determining that predation by the predator is the cause. The image generation method according to claim 2.

4. The environmental data includes indicator values ​​showing the water quality of the underwater area, and the plurality of types of algal objects include algal objects showing poorly growing and withered algae. The image generation method according to claim 1.

5. The aforementioned indicator value represents the water quality at the second time point, The process for making the determination includes calculating the algal growth rate using the index value, and if the growth rate is below a threshold, determining that poor growth is the cause. The image generation method according to claim 4.

6. The generation process includes placing a three-dimensional shape model representing the corresponding algae object into a digital twin that reproduces the terrain in a three-dimensional virtual space, and generating the image data from the digital twin after placement. The image generation method according to claim 1.

7. The system acquires first seaweed bed data showing algae in the underwater region at a first time point, second seaweed bed data showing algae in the underwater region at a second time point later than the first time point, and environmental data showing the environment of the underwater region at at least one of the first and second time points. Based on the first and second seaweed bed data, if a decrease in the amount of algae in the underwater area at the second time point is detected compared to the first time point, the cause of the decrease is determined based on the environmental data. Image data showing the topography of the underwater region, which generates image data including an algae object corresponding to the determined cause, from among multiple types of algae objects each representing an abnormal state of algae. An image generation program that has a computer perform the processing.

Images