Underwater monitoring system, server device, image processing method, and program

Abstract

To acquire a group level of an aquatic organism.SOLUTION: An underwater monitoring system 1 comprises an underwater photographing system 15 for photographing underwater and a server device 40 capable of communication with the underwater photographing system. The underwater photographing system transmits image data of the underwater photographing to the server device. The server device comprises: a communication unit which receives the image data from the underwater photographing system; and image discrimination means which discriminates the type and the group level of an aquatic organism projected in the image data received from the underwater photographing system.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an underwater monitoring system, a server device, an image processing method, and a program. [Background technology]

[0002] There are cases where it is desirable to know the types and numbers of aquatic organisms present in the water. For example, underwater monitoring systems are known for obtaining information such as the types and numbers of fish raised in fixed nets or aquaculture ponds. There are also cases where it is desirable to know the population of aquatic organisms as natural resources.

[0003] Patent document 1 discloses a configuration that includes a sonar for scanning the area above the seabed and collecting data on the distribution of fish, a control unit that operates the sonar at a predetermined cycle, a recording means that records the scan data obtained from the sonar, and a photographing means that collects data on fish species. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies do not adequately distinguish between the school levels of aquatic organisms. For example, the size of a school of small sardines differs from that of a school of large tuna. That is, even if the number of fish in a school is the same, it is desirable to distinguish between a small school of sardines and a large school of tuna. For this reason, it is conceivable to define the school level (school size) for each fish species, but the technology disclosed in Patent Document 2 collects data on fish species and distribution status using different means, making data analysis a complex process. Furthermore, the technology disclosed in Patent Document 2 collects data using different means, namely, a camera and a sonar (fish finder), which can easily lead to a complex system configuration.

[0005] The present invention aims to provide a technology that can obtain the population level of aquatic organisms. [Means for solving the problem]

[0006] In view of the above-mentioned problems, the present invention provides an underwater monitoring system having an underwater photography system for photographing underwater images and a server device capable of communicating with the underwater photography system, the underwater photography system transmits image data of the underwater photograph to the server device; the server device includes a communication unit that receives the image data from the underwater photography system; and an image discrimination means for discriminating the type and group level of aquatic organisms captured in the image data received from the underwater photography system. [Effects of the Invention]

[0007] The present invention can provide a technology that can obtain the population level of aquatic organisms. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram showing an example of fish school levels defined according to fish species. [Figure 2] 1 is a diagram schematically illustrating the overall configuration of an underwater monitoring system. [Figure 3] FIG. 1 is a schematic diagram showing an example of the configuration of a waterproof buoy in an underwater photography system. [Figure 4] FIG. 2 is a schematic diagram illustrating an example of the configuration of an imaging device. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device and a user terminal device. [Figure 6] FIG. 1 is a diagram illustrating an example of the hardware configuration of a camera capable of capturing 360° image data. [Figure 7] FIG. 2 is a functional block diagram of a server device and a user terminal device; [Figure 8] FIG. 10 is a diagram showing an example of a learning image for fish species discrimination. [Figure 9] FIG. 10 is a diagram showing an example of a learning image for fish number class discrimination. [Figure 10] FIG. 1 is a diagram showing an example of the configuration of a fish species discrimination model using CNN. [Figure 11] FIG. 1 is a diagram illustrating an example of the configuration of a fish count class discrimination model using CNN. [Figure 12] FIG. 1 is a diagram showing an example of the configuration of a fish school level discrimination model using CNN. [Figure 13] FIG. 2 is an example of a functional block diagram of a learning unit. [Figure 14] FIG. 10 is a diagram showing an example of a fish school level discrimination table. [Figure 15] 10 is an example of a sequence diagram illustrating a process in which the underwater monitoring system delivers image discrimination results to a user terminal device. [Figure 16] 10 is an example of a sequence diagram illustrating a process in which the underwater monitoring system delivers image discrimination results to a user terminal device in response to a request from a user. [Figure 17] FIG. 10 is an example of a flowchart illustrating a fish species discrimination process performed by an image discrimination program. [Figure 18] FIG. 10 is an example of a flowchart illustrating a fish school level discrimination process by an image discrimination program. [Figure 19] FIG. 10 is a diagram showing an example of an image discrimination result display screen displayed by a user terminal device. [Figure 20] FIG. 1 is a diagram illustrating a schematic view of a school of fish observed by a spherical camera. DETAILED DESCRIPTION OF THE INVENTION

[0009] An underwater monitoring system and an image processing method performed by the underwater monitoring system will be described below as an example of an embodiment of the present invention with reference to the drawings.

[0010] <Outline of the underwater monitoring system of this embodiment> The definition of the level of fish school density, such as size (hereafter referred to as "fish school level"), varies depending on the type of fish. For example, the definition of the fish school level differs for sardines, which move in large schools, and tuna, which move in small schools. In other words, it is desirable to define the fish school level for each fish species, but in the past, data on fish species and distribution status was collected using different means (cameras and sonar, for example), which made data analysis a complex process. In other words, when a fish school is detected, it is necessary to identify the fish species that make up the school and determine the fish school level according to the fish species. However, because the timing of fish school detection and the timing of fish species identification are not the same, a human must later associate which fish school with which fish species.

[0011] Figure 1 shows an example of school levels defined according to fish species. In Figure 1, school levels are shown for sardines, mackerel, and tuna, but even within the same school level, there is a relationship such as the number of sardines > the number of mackerel > the number of tuna, and the school levels defined according to fish species differ.

[0012] Furthermore, if you want to capture a wide underwater area with a camera that has a narrow angle of view, you need to install cameras to the right, left, front, back, top, and bottom. In other words, you need to install cameras in more locations.

[0013] Therefore, in this embodiment, a limited number of cameras (for example, spherical cameras) capture the entire underwater scene. When the underwater monitoring system analyzes the image data captured by the cameras to determine the fish species and school level, it takes into account that the schooling patterns vary depending on the fish species, and determines the school level based on the fish species information. Therefore, it is possible to provide a more inexpensive underwater monitoring system without complicating the hardware, the entire system, or the data processing method.

[0014] <Terminology> Aquatic organisms (also called underwater organisms or aquatic life) include fish, as well as arthropods such as crabs and shrimp, reptiles such as turtles, amphibians such as frogs and salamanders, mammals such as dolphins, whales, and fur seals, cnidarians such as jellyfish, and echinoderms such as sea cucumbers, sea urchins, and starfish. Aquatic organisms can be monitored if they exist in water. Furthermore, water is not limited to the ocean, and can include rivers, lakes, ponds, and swamps.

[0015] The group level of aquatic organisms is a scale that represents the size of a group of aquatic organisms. The group level of aquatic organisms is determined based on the number of aquatic organisms contained in one group. In this embodiment, the group level of aquatic organisms will be described as the fish school level.

[0016] A fish species is a type of fish. A fish population class is a classification of the number of fish contained in a single image data into several levels. A fish school level is a classification of a fish population class into several levels defined according to the fish species.

[0017] <System configuration example> 2 is a diagram showing a schematic diagram of the overall configuration of the underwater monitoring system 1. The underwater monitoring system 1 has an underwater photography system 15, a server device 30, and a user terminal device 40 as its main components.

[0018] The underwater photography system 15 can communicate with the server device 30 via the communication network N. The user terminal device 40 can also communicate with the server device 30 via the communication network N. The user terminal device 40 may also be able to communicate directly with the underwater photography system 15 via the communication network. The communication network N connecting the underwater photography system 15 and the server device 30 is preferably a communication network for wireless communication that allows relatively long-distance communication, such as a mobile phone network, a communication satellite network, or a long-distance wireless LAN. Note that if the server device 30 is located on a ship or submarine at sea, the server device 30 and the underwater photography system 15 may be connected by wire.

[0019] The communication network N connecting the user terminal device 40 and the server device 30 may be, for example, a mobile phone network, a wireless LAN, or a wired LAN. The server device 30 may be located in a data center or in a designated building. The server device 30 may be installed anywhere, such as on a ship or in an office on land, as long as it is in an area where data communication is possible. Note that a server is a computer or software that performs the function of providing information and processing results in response to requests from clients.

[0020] In this embodiment, the server device 30 receives image data of the school of fish 50 transmitted from the underwater photography system 15, analyzes the image data, and transmits the analysis results to the user terminal device 40. If the user terminal device 40 holds the image data, the user terminal device 40 can also transmit the image data to the server device 30. The server device 30 receives the image data transmitted from the user terminal device 40, analyzes the image data, and transmits the analysis results to the user terminal device 40.

[0021] The server device 30 provides a user interface to the user terminal device 40 using a web application or the like, and displays the analysis results on the user terminal device 40. The server device 30 analyzes image data automatically transmitted from the underwater photography system 15, and transmits image classification results such as fish species and fish school levels to the user terminal device 40. In response to a user request, the server device 30 can also cause the underwater photography system 15 to take images, and transmit the image classification results obtained by analyzing the image data in real time to the user terminal device 40.

[0022] The server device 30 may be realized by cloud computing or by a single information processing device. Cloud computing refers to a form in which resources on a network are used without being aware of specific hardware resources. The server device 30 may exist on the Internet or on-premise.

[0023] Furthermore, the functions of the server device 30 may be distributed across multiple information processing devices, or there may be multiple server devices 30 with the same functions, and the number of information processing systems that deliver videos may be increased or decreased depending on the processing load.

[0024] The user terminal device 40 is, for example, a terminal device used by a user, such as a PC (Personal Computer), a smartphone, or a tablet terminal. A web browser or a native application runs on the user terminal device 40. The user operates the user terminal device 40 to obtain data analysis results and underwater image data from the server device 30. The user terminal device 40 may be any information processing device. Examples of such information processing devices include output devices such as electronic whiteboards and digital signage, HUD (Head Up Display) devices, industrial machinery, sound collection devices, medical equipment, network home appliances, mobile phones, smartphones, car navigation systems, tablet terminals, game consoles, PDAs (Personal Digital Assistants), digital cameras, and wearable PCs.

[0025] The user terminal device 40 may be installed anywhere, such as on a ship or in an office on land, as long as it is within an area where data communication is possible.

[0026] FIG. 3 is a schematic diagram showing an example configuration of a waterproof buoy (floating body) 2 of an underwater photography system 15. The waterproof buoy 2 has a sealed, waterproof space inside, in which a power storage device 6 and a communication device 7 are stored. A power cable 8 is connected to the power storage device 6, and a data communication cable 9 is connected to the communication device 7. In this embodiment, the tip of the data communication cable 9 forms an antenna 11 and is not physically connected to the communication device 7. In other words, the data communication cable 9 communicates wirelessly with the communication device 7 via the antenna 11. This allows the data communication cable 9 to communicate with the communication device 7 even if the communication device 7 does not have a wired cable interface. The communication method may be, for example, wireless LAN, Bluetooth (registered trademark), or the like. Alternatively, the communication device 7 and the data communication cable 9 may be physically connected.

[0027] A power cable 8 and a data communication cable 9 are connected to a camera device 3 installed underwater. A waterproof seal member 10 is fitted into an opening on the side of the waterproof buoy 2, and the power cable 8 and the data communication cable 9 are passed through the waterproof seal member 10 and routed outside the waterproof buoy 2. This allows the waterproof performance of the waterproof buoy 2 to be maintained.

[0028] FIG. 4 is a schematic diagram showing an example configuration of an image capturing device 3. The image capturing device 3 has a waterproof housing case 4 and a camera 5. A power cable 8 and a data communication cable 9 are connected to the camera 5. In this embodiment, the tip of the data communication cable 9 is an antenna 11, which is not physically connected to the camera 5. In other words, the data communication cable 9 communicates with the camera 5 wirelessly via the antenna 11. This allows the data communication cable 9 to communicate with the camera 5 even if the camera 5 does not have a wired cable interface. The communication method may be, for example, wireless LAN, Bluetooth (registered trademark), etc. However, the data communication cable 9 may also be physically connected to the camera 5.

[0029] A waterproof seal member 10 is fitted into the opening on the top surface of the waterproof housing case 4. The power cable 8 and the data communication cable 9 are passed through the waterproof seal member 10 and routed inside the waterproof housing case 4, so that the waterproof performance of the waterproof housing case 4 can be maintained.

[0030] The camera device 3 is installed in the sea by hanging from the waterproof buoy 2 via a hanging rope 13 fixed to the waterproof buoy 2, and photographs aquatic life, mainly fish, and fishing gear such as nets. In this embodiment, a weight 12 is attached to the camera device 3, so that the camera device 3 is installed in the sea without sagging the hanging rope 13. This allows the camera device 3 to maintain its position underwater in a simpler way. However, the camera device 3 may also be installed underwater in other ways, such as by being fixed to fishing gear or the seabed.

[0031] In this embodiment, the waterproof buoy 2 is fixed to one end of a hanging rope 13, and the imaging device 3 is fixed to the other end. This hanging rope 13 is bundled together with the power cable 8 and the data communication cable 9 by a fastener 14. The hanging rope 13 is tensioned by the weight 12 to maintain a straight shape, while the power cable 8 and the data communication cable 9 are kept flexed, thereby preventing the application of large forces to the power cable 8 and the data communication cable 9. However, the hanging rope 13 is not essential, and by increasing the mechanical strength of the power cable 8 and the data communication cable 9, it is possible to hang the imaging device 3 from the waterproof buoy 2 using only the power cable 8 and the data communication cable 9, without using the hanging rope 13.

[0032] <Hardware configuration example> The hardware configurations of the server device 30, the user terminal device 40, and the camera 5 according to this embodiment will be described with reference to FIGS.

[0033] <<Server equipment and user terminal equipment>> Fig. 5 is a diagram showing an example of the hardware configuration of the server device 30 and the user terminal device 40 according to this embodiment. As shown in Fig. 5, the server device 30 and the user terminal device 40 are constructed by a computer 500, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0034] Of these, the CPU 501 controls the overall operation of the computer 500. The ROM 502 stores programs used to drive the CPU 501, such as an IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or image data. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories or printers. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 5.

[0035] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0036] <<Camera>> Figure 6 is an example of a hardware configuration diagram of a camera 5 (omnidirectional camera) capable of capturing 360-degree image data. In the following, camera 5 is assumed to be a device that uses a photographing element to capture 360-degree image data around a device at a predetermined height, but the number of photographing elements may be one, two, or more. Furthermore, it does not necessarily have to be a dedicated device; a PC, digital camera, smartphone, etc. may have substantially the same functionality by attaching a photographing unit with a 360-degree angle of view to it after the fact.

[0037] As shown in FIG. 6, the camera 5 is composed of a photographing unit 601, an image processing unit 604, a photographing control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a concave terminal 621 for Micro USB.

[0038] Of these, the photographing unit 601 includes a wide-angle lens (so-called fisheye lens) 602 with a 360° angle of view for forming a hemispherical image, and photographing elements 603 (image sensors) provided corresponding to each wide-angle lens. The photographing elements 603 include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lens 602 into image data and outputs it as an electrical signal, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers that set various commands and parameters required for the operation of the photographing elements. The photographing unit 601 may also be a 360° camera, an example of a photographing means capable of capturing a 360° view around the camera 5. The camera 5 may also combine multiple pieces of information acquired by multiple photographing elements (e.g., 180° + 180°) to achieve a 360° angle of view.

[0039] The imaging elements 603 (image sensors) of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the imaging elements 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an audio sensor 618, and the like are also connected to the bus 610.

[0040] The image processing unit 604 takes in the image data output from the image pickup element 603 via a parallel I / F bus, performs predetermined processing on each piece of image data, and creates a panoramic image from the fisheye image.

[0041] The imaging control unit 605 generally uses the I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0042] Furthermore, the imaging control unit 605 instructs the imaging elements 603a and 603b to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from a PC. Some cameras 5 have a preview display function or a function corresponding to image display on a display (for example, a PC or smartphone display). In this case, the output of image data from the imaging elements 603 is performed continuously at a predetermined frame rate (frames / minute).

[0043] As will be described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the camera 5 is not provided with a display, but a display unit may be provided.

[0044] The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 via the I / F bus and performs predetermined processing on the sound data.

[0045] The CPU 611 controls the overall operation of the camera 5 and executes necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of processed equirectangular projection images.

[0046] The operation unit 615 is a general term for operation buttons such as a shooting start button 615a. By operating the operation unit 615, the user can start shooting or recording, as well as turn the power on / off, establish a communication connection, and input settings such as various shooting modes and shooting conditions.

[0047] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device may be, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection I / F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. The image data stored in the DRAM 614 is transmitted to an external terminal via the external device connection I / F 616 or recorded on external media. Furthermore, the camera 5 may use multiple external device connection I / Fs 616 to, for example, transmit image data captured by the camera 5 to a PC via USB for recording, while also acquiring image data (e.g., screen information to be displayed in a teleconferencing app) from the PC to the camera 5. The camera 5 may further transmit the image data to other external devices (e.g., a display, a projector, an electronic whiteboard) via HDMI (registered trademark) from the camera 5 for display.

[0048] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the camera 5, and transmit the stored image data to the cloud server. The communication unit 617 may also communicate with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0049] The audio sensor 618 is a sensor that acquires 360° audio information in order to identify from which direction a loud audio is being input within 360° around (horizontal plane) the camera 5. The audio processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction within 360°.

[0050] Note that other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use this information for image correction or adding location information.

[0051] The image processing unit 604 also performs the following processes. The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB interpolation processing) on ​​the RAW data input from the image sensor that inputs the spherical image to create a fisheye image (curved image). The CPU 611 then performs flattening processing such as DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image that captures 360 degrees around the camera 5.

[0052] <About the function> 7 is a functional block diagram of server device 30 and user terminal device 40. Server device 30 has automatic processing unit 301 that automatically performs various processes, and user request processing means 329 that performs processes based on requests from users.

[0053] The automatic processing unit 301 includes a camera control unit 312, a captured image data calling unit 315, an image discrimination unit 316, an image discrimination result saving unit 317, and a memory unit 310. The automatic processing unit 301 implements the camera control unit 312 based on a camera control program 311 stored in the memory unit 310. The camera control unit 312 automatically controls the camera 5 of the image capture device 3. Automatic control refers to performing predetermined control at a predetermined time. The predetermined control includes, for example, starting the camera 5, starting shooting with the camera 5, stopping shooting with the camera 5, saving image data 313, and pausing the camera 5. The camera control unit 312 can use various web applications and client software, such as live streaming software and remote operation applications. In this embodiment, RICOH Remote Field (registered trademark) may be used as an example.

[0054] The automatic processing unit 301 also implements a photographic data recall means 315, an image discrimination means 316, and an image discrimination result storage means 317 based on an image discrimination program 314 stored in the storage unit 310. The photographic data recall means 315 acquires image data 313 from the storage unit 310 and passes it to the image discrimination means 316. The image discrimination means 316 extracts feature quantities of objects from the image data 313 and performs various discriminations, such as object classification. While various artificial intelligence technologies and discriminant analysis technologies can be used as the image discrimination means 316, this description will be given assuming that it uses models (a fish species discrimination model, a fish population class discrimination model, and a fish school level discrimination model, which will be described later) constructed using a convolutional neural network (hereinafter referred to as CNN). The person in charge converts the models learned by the CNN into a programming language and implements it in the image discrimination means 316. The image discrimination result storage means 317 stores the image discrimination result 318 obtained by the image discrimination means 316 in the storage unit 310. The image discrimination result 318 includes, for example, at least one of the fish species and the fish school level.

[0055] Next, the user request processing means 329 will be described. The user request processing means 329 performs processing based on a user request transmitted from the user terminal device 40. The user request processing means 329 includes a photographed data calling means 315, an image discrimination means 316, an image discrimination result storage means 317, a user request processing means 329, a transmission / reception means 330, and a memory unit 320. Of these, the functions of the photographed data calling means 315, the image discrimination means 316, and the image discrimination result storage means 317 may be the same as those of the automatic processing unit 301. Furthermore, the image data 313, the image discrimination result 318, and the image discrimination program 314 stored in the memory unit 320 may be the same as those of the automatic processing unit 301.

[0056] The user request processing means 329 processes a user request received by the transmission / reception means 330 from the user terminal device 40. The transmission / reception means 330 receives a user request from the user terminal device 40, and also transmits image data 313, 323 and image discrimination results 318, 328 to the user terminal device 40.

[0057] The user request processing means 329 may receive image data of the fish from the user terminal device 40 via the transmission / reception means 330. The user request processing means 329 stores this image data in the storage unit 320. The photographed data calling means 315 acquires the image data from the storage unit 320 and passes it to the image discrimination means 316. The image discrimination result saving means 317 saves the image discrimination result obtained by the image discrimination means 316 in the storage unit 320. The user request processing means 329 transmits the image data and the image discrimination result to the user terminal device 40 via the transmission / reception means 330.

[0058] There are two main user requirements: The first is to display the automatically processed image data 313 and the image discrimination result 318, which is the result of the analysis, on the user terminal device 40. The second is that the user manually operates the camera control means 312 and the image discrimination program 314, and displays the image data 323 that is the output of these and the image discrimination result 328 that is the analysis result thereof on the user terminal device 40.

[0059] In other words, requests issued from the user terminal device 40 include the selection and display of image data 313 captured by automatic processing, the selection and display of image discrimination results 318 discriminated by automatic processing, the activation of the camera control means 312, the selection and display of image data 323 captured manually by the user, the activation of the image discrimination program 314, and the selection and display of image discrimination results 328 performed manually by the user.

[0060] <<User terminal device>> The user terminal device 40 is used by a user who wants to analyze fish species and fish school levels. The user terminal device 40 has a communication unit 41, a display control unit 42, and an operation reception unit 43. Each of these functional units is a function or means realized by the CPU 501 executing instructions contained in one or more programs installed in the user terminal device 40. Note that this program may be a web application executed by a web browser, or a dedicated native application.

[0061] The communication unit 41 transmits and receives various types of information to and from the server device 30. In this embodiment, the communication unit 41 receives screen information of a Web application and an image discrimination result display screen (described later) from the server device 30, and transmits user operation details and instructions to the server device 30.

[0062] The display control unit 42 interprets screen information of various screens and displays it on the display 506. The operation accepting unit 43 accepts various operations on the various screens displayed on the display 506 by the user.

[0063] <About learning> Next, CNN learning will be described with reference to FIG. 8 and other figures. FIG. 8 is an example of a learning image for fish species discrimination. As described above, when CNN is used as the model used by the image discrimination means 316, the learning data is image data of fish whose species is known. That is, multiple sets of learning data are prepared, each set consisting of fish image data, which is input data to the CNN, and the fish species, which is training data. The image data used for the learning data are images of aquatic organisms taken in various environments, and image data of various image qualities, with brightness, contrast, hue, etc., changed, are used to increase robustness.

[0064] As an example, Figure 8 shows image data of sardines, mackerel, and Japanese jack mackerel. Since a school of fish is almost always made up of the same type of fish, the image data used for training to identify fish species contains only one type of fish. The image data used for training to identify fish species can be either of the following: (1) Image data of a fish recognized by object recognition from the image data, cropped Object recognition is image processing using AI to detect pre-specified (learned) objects. In this embodiment, the objects are fish. The person in charge annotates (labels the fish species) the fish trimmed by object recognition. (2) Original image data captured by camera 5 The original image data captured by camera 5 is assumed to contain one or more fish. This image data may also contain a school of fish. The person in charge annotates the image data. Since different species of fish rarely gather together in schools, even if a school of fish is captured, only one species of fish is annotated. Therefore, in (2), the image data used for training to identify fish species and the image data used for training to identify fish number classes may be the same.

[0065] FIG. 9 shows image data used for learning fish number class discrimination. Image data used for learning fish number class discrimination is an image of a school of fish classified according to predefined fish number classes. A fish number class indicates, in several stages, how many fish are present in a single image data. FIG. 9 shows four stages, but this is just one example. For the purpose of discriminating between fish school levels, which will be described later, it is preferable that the number of fish number classes is greater than the number of fish school levels.

[0066] Image data for learning to distinguish fish number classes is prepared regardless of the fish species. In other words, if there are 10 fish, sardines and mackerel are classified into the same fish number class. This allows for a single fish number class discrimination model regardless of the fish species. However, the learning unit may also generate a fish number level discrimination model for each fish species.

[0067] 8 and 9 show only a few pieces of image data for the sake of convenience, but in reality, a model is generated using more than 500 pieces of learning image data. In the example of Fig. 9, there are five fish school levels, but it is possible to define more or fewer than four levels.

[0068] Furthermore, in this embodiment, a CNN has been described, but other machine learning methods such as a support vector machine may be used to discriminate image data.

[0069] <<CNNについて> > In this embodiment, an existing CNN or an appropriately improved CNN may be used. A brief supplementary note about CNN will be provided.

[0070] 10 shows an example of the configuration of a fish species discrimination model using a CNN 60. As an example, the CNN 60 has convolutional layers 62 and 64, pooling layers 63 and 65, and a fully connected layer 70. An input image 61 is image data captured by a camera 5. The input image 61 is processed in the order of the convolutional layer 62, pooling layer 63, convolutional layer 64, pooling layer 65, and fully connected layer 70.

[0071] The convolutional layers 62 and 64 convert lattice-like numerical data called kernels (or filters) and numerical data of a partial image (called a window) of the same size as the kernel into a single numerical value by calculating the sum of the product of each element. The convolutional layers 62 and 64 convert this conversion process into small lattice-like numerical data (i.e., tensors) by shifting the window slightly. The lattice-like numerical data is activated by an activation function and input to the pooling layers 63 and 65.

[0072] The pooling layers 63 and 65 are used to create a single numerical value from the activated numerical data. Examples include maximum pooling, which selects the maximum value within a window, and average pooling, which selects the average value within a window. The convolutional layers 62 and 64 extract features from the image data, and the pooling layers 63 and 65 blur the precise location of the object. The activation function is a function that nonlinearly transforms (activates) the input (for example, ReLU, tanh, sigmoid, etc.).

[0073] The output of the pooling layer 65 is input to the fully connected layer 70. The fully connected layer 70 is called a neural network. In a neural network, L layers are fully connected from the nodes in the input layer 66 to the nodes in the output layer 68. A neural network with a deep hierarchy is called a DNN. The layer between the input layer 66 and the output layer 68 is called an intermediate layer 67. The number of intermediate layers 67 and the number of nodes are merely examples.

[0074] Weights are set for the connections between nodes, and the output from a node multiplied by the weight is transmitted to the node in the next layer. The node in the next layer receives the output of all nodes in the previous layer, so the node in the next layer sums the output of all nodes in the previous layer. The node in the next layer activates the summed output using an activation function and transmits it to the next node. This process is repeated until the value is transmitted to the output layer.

[0075] In this embodiment, since it is desired to distinguish between fish species, fish population classes, or fish school levels, a classification model is generated (another model is a regression model). For this reason, in a model for distinguishing fish species, the output layer 68 is provided with nodes equal to the number of fish species to be distinguished. For example, if it is desired to distinguish between 10 types of fish species, the number of nodes in the output layer 68 is 10. For example, if it is desired to distinguish between four levels of fish population classes, the number of nodes in the output layer 68 is 4. Figure 10 shows a model for distinguishing fish species, and Figure 11 shows a model for distinguishing between fish population classes.

[0076] In a classification model, it is common for each node in the output layer 68 to output the probability of being classified into that node. Therefore, in Figure 10, the output layer 68 outputs the probability of the fish species associated with each node, such as node 71 being the "probability of sardine," node 72 being the "probability of mackerel," and node 73 being the "probability of Japanese jack mackerel."

[0077] In the model learning phase, image data of known fish species is prepared, so the training data is a vector in which only the node corresponding to the fish species is "1" and the other nodes are zero. For example, in the case of image data known to be of sardines, only node 71 is "1" and the other nodes are "0". The learning unit, which will be described later, calculates the difference between the output (probability) of each node in the output layer 68 and the training data using a loss function, and transmits this to the input layer side using the backpropagation method. The weights between the nodes are learned using the backpropagation method, and gradually nodes 71 to 73 in the output layer begin to output the correct probabilities.

[0078] In the inference phase using the model, for example, when image data of sardines is input, it is expected that node 71 in output layer 68 corresponding to sardines will output a probability close to "1," and nodes 72 and 73 in output layer 68 corresponding to other fish species will output probabilities close to "0." The image discrimination means 316 discriminates (infers) that the image data is of the fish species corresponding to the node with the highest probability. Note that when image data of a fish species that the model has not learned is input, each node in output layer 68 will output roughly the same probability, and therefore, if the highest probability is equal to or less than a threshold, the image discrimination means 316 will discriminate the image as an unclassified fish species.

[0079] The same is true for the CNN 80 shown in Figure 11. The CNN 80 in Figure 11 is a model that distinguishes fish population classes, and each node 75 to 78 in the output layer 68 outputs the probability of fish population classes 0 to 3. That is, node 75 outputs the "probability of fish population class 0," node 76 outputs the "probability of fish population class," node 77 outputs the "probability of fish population class 2," and node 78 outputs the "probability of fish population class 3." In the learning phase of the CNN 80 in Figure 11, multiple image data containing different numbers of fish are used, prepared regardless of fish species. In addition, each image data is annotated with the fish population class.

[0080] In the inference phase, the image discrimination means 316 infers the fish population class using a model for discriminating the fish population class, and then further discriminates the fish school level using a fish school level discrimination table (see FIG. 14).

[0081] If a fish school level is defined for each fish species, a large number of patterns will be generated by multiplying the number of fish species by the number of fish school level definitions, which will take a long time to learn. Therefore, in this embodiment, the image discrimination means 316 first discriminates fish species. Next, the image discrimination means 316 discriminates fish count classes. Then, the image discrimination means 316 narrows down the fish species and discriminates fish school levels based on the fish count classes. This reduces the amount of data processing. Therefore, it is preferable to discriminate fish species before discriminating fish school levels.

[0082] However, it is also possible to create a model trained on training data containing a mixture of fish species and school levels, such as sardine school levels 0 to 3, mackerel school levels 0 to 3, and yellowtail school levels 0 to 3, and use this model to output which fish and which school level a given image data belongs to in a single inference. However, this is expected to take some time to train.

[0083] Note that the numbers of convolutional layers 62 and 64, pooling layers 63 and 65, and fully connected layers 70 shown in the figure are merely examples, and the order of processing is also merely an example.

[0084] <> The image discrimination means 316 may directly discriminate the fish school level. In this case, the CNN 81 is a model that outputs the probability of the fish school level. The CNN 81 in FIG. 12 is a model that discriminates the fish school level, and each node 82 to 85 of the output layer 68 outputs the probability of fish school levels 0 to 3. That is, node 82 outputs the "probability of fish school level 0," node 83 outputs the "probability of fish school level 1," node 84 outputs the "probability of fish school level 2," and node 85 outputs the "probability of fish school level 3." In the learning phase of the CNN 81 in FIG. 12, multiple image data prepared for each fish species and containing different numbers of fish are used. In this case, the fish school level defined for each fish species is annotated to the image data.

[0085] Therefore, because a fish school-level discrimination model is generated for each fish species, the workload for preparing training data and other tasks increases, but the time required to train each individual fish school-level discrimination model can be shortened.

[0086] Note that the numbers of convolutional layers 62 and 64, pooling layers 63 and 65, and fully connected layers 70 shown in the figure are merely examples, and the order of processing is also merely an example.

[0087] <<About the learning section functions>> FIG. 13 is a functional block diagram of the learning unit 624. The learning unit 624 has a function of generating a fish species discrimination model and a fish school level discrimination model. FIG. 13(a) is a functional block diagram of the learning unit 624 that generates a fish species discrimination model, and FIG. 13(b) is a functional block diagram of the learning unit 625 that generates a fish population class discrimination model. Furthermore, FIG. 13(c) is a functional block diagram of the learning unit 626 that generates a fish school level discrimination model. These have the same structure except for the learning data and the generated model. Furthermore, learning may be performed on any computer other than the server device 30, but the server device 30 may also be used for learning.

[0088] The learning units 624, 625, and 626 each have a learning data acquisition unit 641, a learning data storage unit 642, and a model generation unit 643. The learning data acquisition unit 641 acquires learning data. The learning data is as follows: Fish species discrimination model Image data (input data) classified by fish species, fish species (training data) Fish population class discrimination model Image data (input data) not classified by fish species, fish number class (training data) Fish school level discrimination model Image data classified by fish species (input data), fish school level (training data) The learning data acquisition unit 641 acquires learning data and stores it in the learning data storage unit 642. The learning data is a set of input data and teacher data, and multiple sets are prepared (for example, 500 sets).

[0089] The training data storage unit 642 stores the training data acquired by the training data acquisition unit 641. The model generation unit 643 trains the training data using various machine learning algorithms to generate a fish species discrimination model, a fish number class discrimination model, and a fish school level discrimination model. The fish species discrimination model can also be expressed as correspondence information that associates image data with fish species. The fish number class discrimination model can also be expressed as correspondence information that associates image data with fish number classes. The fish school level discrimination model can also be expressed as correspondence information that associates image data with fish school levels. The model of this embodiment is a classification model that classifies training data. Note that classification models used in supervised learning include support vector machines, logistic regression, decision trees, random forests, etc. in addition to CNN.

[0090] An image classification program 314 is created based on the fish species classification model, fish population class classification model, and fish school level classification model generated through machine learning as described above. There are various methods for training and creating programs, but in this embodiment, training is started based on VGG16, a CNN technique, and recursive training is performed while automatically changing the configuration and parameters of VGG16 to improve the accuracy rate, which is one of the indicators for evaluating the classification results. VGG16 is a convolutional neural network with a depth of 16 layers, and is characterized by convolutional layers using small 3 x 3 filters.

[0091] One method for determining the level of a school of fish is to count the number of fish using object recognition technology. However, it is difficult to accurately count the number of fish from image data taken underwater due to factors such as overlapping fish and unclear images. On the other hand, determining the level of a school of fish often does not require an exact number of fish, but rather only requires an understanding of the size of the school. Therefore, in this embodiment, rather than having the server device 30 count the number of fish, the server device 30 first classifies the number of fish as shown in Figure 9 and then determines the level of the school of fish using the fish school level determination table in Figure 14. In other words, the level of the school of fish is determined based on the appearance of the school of fish. This reduces the amount of data processing required for the determination.

[0092] <Fish school level discrimination table> Figure 14 shows an example of a fish school level discrimination table. In the fish school level discrimination table, a fish school level is defined for each fish species, taking into account that schooling patterns vary depending on the fish species. The image discrimination means 316 discriminates the fish school level by referring to the fish school level discrimination table. That is, the image discrimination means 316 determines the row of the fish school level discrimination table based on the fish species output by the fish species discrimination model. The image discrimination means 316 also determines the column of the fish school level discrimination table based on the fish number class inferred by the fish number class discrimination model. The fish school level set in the square where the determined row and column intersect is a fish school level that takes into account that schooling patterns vary depending on the fish species.

[0093] For example, even if the number of fish in the image is approximately the same, fish species A can be judged as fish school level 1, while fish species B can be judged as fish school level 3, which will allow for more accurate decisions such as deciding whether to go fishing.

[0094] Furthermore, in addition to the number of fish, information such as season and region may also be added to define the school level in more detail. In this case, a school level discrimination table is prepared for each fish species, school level, and season (or region). Furthermore, a school level discrimination table is prepared for each fish species, school level, season, and region. <Action or Processing> 15 is a sequence diagram illustrating the process by which the underwater monitoring system delivers the image discrimination results to the user terminal device 40. First, the flow of the process when underwater photography is automatically performed will be described.

[0095] S1: The camera 5 of the underwater photography system 15 periodically photographs the surroundings in response to a request issued at a predetermined time interval from the automatic processing unit 301. The time interval can be set by the user operating the user terminal device 40. It does not necessarily have to be periodic, and photographs may be taken repeatedly.

[0096] S2: The camera 5 of the underwater photography system 15 transmits image data to the server device 30.

[0097] S3: The communication means 319 of the server device 30 receives the image data. The communication means 319 stores the image data 313 in the storage unit 310.

[0098] S4: When image data 313 is saved in memory unit 310, the photographing data retrieval means 315 uses this as an opportunity to retrieve the image data 313 from memory unit 310. Image discrimination means 316 discriminates the fish school level using a fish species discrimination model, a fish number class discrimination model, and a fish school level discrimination table. Image discrimination result storage means 317 stores the fish school level in memory unit 310 as image discrimination result 318. Details of step S4 will be explained using the flowcharts in Figures 17 and 18.

[0099] S5: At any time, the user connects the user terminal device 40 to the server device 30. The user displays a specific web page on the user terminal device 40 and inputs an operation to the user terminal device 40 requesting image discrimination results. The operation reception unit 43 receives the input and operation. The user can specify, for example, at least one of the image discrimination results for fish species and the image discrimination results for fish school level.

[0100] S6: The transmission / reception means 330 of the server device 30 receives the request, and the user request processing means 329 acquires the image discrimination means 316 and the image data 313 from the storage unit 320. The transmission / reception means 330 transmits the image discrimination means 316 and the image data 313 to the user terminal device 40.

[0101] S7: Communication unit 41 of user terminal device 40 receives image discrimination means 316 and image data 313, and display control unit 42 displays them on display 506. Note that image discrimination result display screen 200 on which user terminal device 40 displays image discrimination means 316 and image data 313 is shown in FIG. 19.

[0102] FIG. 16 is a sequence diagram illustrating the process in which the underwater monitoring system 1 delivers the image discrimination results to the user terminal device 40 in response to a request from the user.

[0103] S11: At any timing, the user connects user terminal device 40 to server device 30. The user displays a specific web page on user terminal device 40 and inputs an operation to request photography into user terminal device 40. Operation accepting unit 43 accepts the operation or input. Communication unit 41 transmits the photography request to server device 30.

[0104] S12: The transmitting / receiving means 330 of the server device 30 receives the photography request, and the user request processing means 329 passes the photography request to the automatic processing unit 301. The camera control means 312 of the automatic processing unit 301 transmits the photography request to the underwater photography system 15 via the communication means 319.

[0105] S13: The camera 5 of the underwater photography system 15 receives the photography request and photographs the surroundings.

[0106] S14: The camera 5 of the underwater photography system 15 transmits image data to the server device 30.

[0107] S15: The communication means 319 of the server device 30 receives the image data. The communication means 319 passes the image data to the user request processing means 329. The user request processing means 329 stores the image data as image data 323 in the storage unit 320.

[0108] S16: When image data 323 is saved in memory unit 320, the photographing data retrieval means 315 uses this as an opportunity to retrieve the image data 323 from memory unit 320. The image discrimination means 316 discriminates the fish school level using a fish species discrimination model, a fish number class discrimination model, and a fish school level discrimination table. The image discrimination result storage means 317 stores the fish school level in memory unit 320 as an image discrimination result 328. Details of step S16 will be explained using the flowcharts in Figures 17 and 18.

[0109] S17: The user request processing means 329 acquires the image discrimination result 328 and the image data 323 from the storage unit 320. The transmission / reception means 330 transmits the image discrimination result 328 and the image data 323 to the user terminal device 40.

[0110] S18: Communication unit 41 of user terminal device 40 receives image discrimination result 328 and image data 323, and display control unit 42 displays them on display 506. Note that image discrimination result display screen 200 on which user terminal device 40 displays image discrimination result 328 and image data 323 is shown in FIG. 19 .

[0111] 17 is a flowchart illustrating the fish species discrimination process by the image discrimination program 314 (photographed data calling means 315, image discrimination means 316, image discrimination result storage means 317). The process in FIG. 17 is started in response to a request issued at a predetermined time interval from the automatic processing unit 301 or a user request transmitted from the user terminal device 40.

[0112] First, the photographed data calling means 315 reads the target image data (S101). The target image data is image data that has been photographed but for which no image discrimination results have yet been obtained. If the predetermined time interval is sufficiently longer than the time required for image discrimination, the target image data is the image data that was last photographed. The target image data may also be image data that has been photographed at the request of the user.

[0113] Next, the image discrimination means 316 inputs the image data into a fish species discrimination model to discriminate the fish species (S102). The image discrimination means 316 may input the wide-angle image data directly into the fish species discrimination model, or may input a portion of the image in which the circumscribing rectangle of the fish has been trimmed by object recognition into the fish species discrimination model.

[0114] Next, the image discrimination result storage means 317 stores the image discrimination result 318 of the fish species in the storage unit 310 or 320 (S103). In this way, the image discrimination result 318 or 328 in which the fish species is associated with one image data 313 or 323 is stored.

[0115] Next, the photographing data calling means 315 reads image data that is to be used to determine the school of fish level (S104). This image data may be the same as that in step S101.

[0116] Next, the image discrimination means 316 inputs the image data into a discrimination model for fish number classes to discriminate the fish number classes (S105). The image discrimination means 316 inputs the wide-angle image data directly into the discrimination model for fish number classes.

[0117] Next, the image discrimination means 316 determines the fish school level associated with the fish species discriminated in step S102 and the fish number class discriminated in step S105 from the fish school level discrimination table (S106).

[0118] Next, the image discrimination result storage means 317 stores the fish school level image discrimination result 318 or 328 in the memory unit 310 or 320 (S107). In this way, the image discrimination result 318 or 328 in which the fish species and the fish school level are associated with one image data is stored.

[0119] <<Discrimination of fish school level using a model that directly discriminates fish school level>> Figure 18 is a flowchart explaining the fish school level discrimination process by the image discrimination program 314 (photographed data calling means 315, image discrimination means 316, image discrimination result storage means 317). Figure 18 explains the case where a fish school level discrimination model is used instead of a fish number class discrimination model. In this case, the fish school level discrimination table is not used.

[0120] First, steps S201 to S203 may be the same as steps S101 to S103 in FIG.

[0121] Next, the photographing data calling means 315 reads the image data to be subjected to fish school level discrimination (S204a). In parallel, the photographing data calling means 315 also reads the fish species image discrimination result 318 or 328 (S204b).

[0122] Based on the fish species image discrimination result 318 or 328, the image discrimination means 316 selects a fish school level discrimination model corresponding to the fish species, and inputs the image data into the selected fish school level discrimination model to discriminate the fish school level (S205).

[0123] Next, the image discrimination result storage means 317 stores the fish school level image discrimination result 318 or 328 in the memory unit 310 or 320 (S206). In this way, the image discrimination result 318 or 328 in which the fish species and the fish school level are associated with one image data is stored.

[0124] <Image discrimination result display screen> 19 is an example of an image discrimination result display screen 200 displayed by the user terminal device 40. The image discrimination result display screen 200 has an image folder selection button 231, a discrimination type setting field 201, a discrimination execution button 202, a discrimination result display button 203, a real-time video button 204, and an image display field 205.

[0125] The image folder selection button 231 is a button that allows the user to perform image discrimination on any image. One of the arbitrary images is an image captured using the real-time video button 204. When the user selects the real-time video button 204, the camera control means 312 is activated. The user can manually save the real-time video captured under the control of the camera control means 312. This real-time video is saved in the folder that the user selects using the image folder selection button 231.

[0126] The discrimination type setting field 201 is a selection field in which the user selects the analysis content. The discrimination type setting field 201 has radio buttons corresponding to fish species 209, fish school level 210, and fish species + fish school level 211. Fish species 209 is selected when the user requests the server device 30 to discriminate the fish species. Fish school level 210 is selected when the user requests the server device 30 to discriminate the fish school level. Fish species + fish school level 211 is selected when the user requests the server device 30 to discriminate the fish species and fish school level.

[0127] The discrimination execution button 202 is a button that accepts real-time photography and discrimination of at least one of the fish species and fish school level. When the user presses the discrimination execution button 202, the user terminal device 40 transmits the selection made in the discrimination type setting field 201 and a photography request to the server device 30. This allows the user terminal device 40 to receive and display the image discrimination result and image data in which at least one of the fish species and fish school level has been discriminated according to the selection. For example, if the radio button associated with the fish species + fish school level 211 is selected in the discrimination type setting field 201, the user terminal device 40 can display the image discrimination result and image data in which the fish species and fish school level have been discriminated.

[0128] The discrimination result display button 203 is a button that accepts processing to display the image discrimination result 318 that has already been saved in the server device 30 on the user terminal device 40. In this case, the user terminal device 40 can display the image discrimination result in which the fish species and fish school level have been discriminated, as well as the image data. The real-time video button 204 is a button that accepts processing to request the user to take an image in real time from the camera 5, and to display the captured image data on the user terminal device 40. In this case, the image discrimination result is not attached, but the user terminal device 40 can display the image data like a video.

[0129] In the image display field 205 in Figure 19, when the fish species + fish school level 211 is pressed, multiple image data 206, 207 are displayed. The image data 206 displays "Level 3: Sardine", and the image data 207 displays "Level 2: Yellowtail". The image data 213 displays "Level 2: Mackerel", and the image data 214 displays "Level 2: Sardine". The user can grasp the fish school level and fish species.

[0130] <Major Effects> As described above, the underwater monitoring system 1 of this embodiment can obtain two types of information from images captured only by the camera 5: information about individual fish, such as the fish species, and information about the entire school of fish, such as the fish school level. When analyzing image data captured by the camera to determine the fish species and school level, the underwater monitoring system 1 takes into account that different fish species school in different ways, and determines the school level based on the fish species information. Therefore, a more inexpensive underwater monitoring system can be provided without complicating the hardware, the entire system, or the data processing method.

[0131] In order to perform image discrimination at the fish species or fish school level with less data processing, it is preferable to use an omnidirectional camera as the camera 5.

[0132] Fig. 20 is a diagram that shows a model of the observation of a school of fish by an omnidirectional camera 5. As shown in Fig. 20, an image can be captured in a 360-degree range centered on the omnidirectional camera 5, so two types of information can be obtained with only one camera 5 and one capture: information on individual fish 51 such as the fish species, and information on the entire school of fish 52 such as the fish school level.

[0133] Furthermore, because the focus position can be set to infinity, information about the entire school of fish, from near to far, can be obtained with just one camera 5 and one shot, making it possible to accurately determine the level of the school of fish. In other words, by using an omnidirectional camera, more information can be obtained in both the horizontal and vertical directions with just one camera 5 and one shot.

[0134] To obtain the same information using a conventional camera with a narrow angle of view, it would be necessary to install cameras in each direction: right, left, front, back, top, and bottom, and further increase the number of camera locations to match the camera's depth of field. Therefore, by using a spherical camera, it is possible to realize a small and inexpensive imaging device 3. The spherical image data captured by the spherical camera is converted into an equirectangular image before undergoing machine learning and image classification.

[0135] 2 and 20, only one camera 5 is installed, but it is also possible to install two or more cameras 5. This is to accommodate the fact that the water depth at which different fish species live varies, and multiple cameras 5 can be installed at different water depths.

[0136] By using the underwater monitoring system 1 of this embodiment, a user can grasp the type of fish in the water and the level of the schools even while they are far away. For example, in the case of fixed net fishing, the user can determine whether to go out fishing or not and know the amount of ice that should be loaded on the boat before going out, thereby improving work efficiency, reducing costs, and stabilizing management.

[0137] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0138] For example, in this embodiment, the case where fish are captured in the image data has been mainly described, but the server device 30 can perform similar processing even if shrimp or other creatures are captured in the image data. The image discrimination means 316 of the server device 30 can recognize shrimp from the image data, discriminate the fish population class, and discriminate the fish school level using a fish school level discrimination table created for shrimp.

[0139] In addition, although the present embodiment distinguishes between fish population classes, the server device 30 may recognize fish by object recognition and count the number of fish. The server device 30 may classify the number of fish into fish population classes. Alternatively, since the server device 30 knows the fish species, it may distinguish between fish school levels from the fish species and the number of fish.

[0140] Furthermore, the underwater photography system 15 does not have to be installed underwater; for example, it may be installed floating on the water surface with its optical axis directed underwater.

[0141] Furthermore, in this embodiment, the server device 30 and the user terminal device 40 communicate with each other via the communication network N, but the user may operate a console connected to the server device 30. The console refers to a keyboard and a display connected to the server device 30. The user operates the keyboard of the console to display the image discrimination result display screen 200 on the display.

[0142] Furthermore, in this embodiment, the underwater photography system 15 and the server device 30 are separate entities, but the underwater photography system 15 and the server device 30 may also be integrated. That is, the underwater photography system 15 may operate as a standalone system. In this case, the underwater photography system 15 includes the storage unit 302, photography data retrieval means 315, image discrimination means 316, and image discrimination result storage means 317 of the server device 30 shown in FIG. 7. The user terminal device 40 communicates with the standalone underwater photography system 15 via the communication network N.

[0143] Furthermore, the underwater photography system 15, server device 30, and user terminal device 40 may be integrated. In this case, the underwater photography system 15 has the functions of the server device 30, as well as the display control unit 42 and operation reception unit 43 of the user terminal device 40 shown in Fig. 7. The user can operate the underwater photography system 15 directly.

[0144] In addition, the configuration examples in Fig. 7 and the like are divided according to main functions to make it easier to understand the processing by the server device 30. The method of dividing the processing units and their names do not limit the present invention. The processing by the server device 30 can be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

[0145] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0146] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, which are ways of storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0147] The above description is one example, and other embodiments, additions, modifications, deletions, and other changes can be made within the scope of what a person skilled in the art can conceive. Any of the following aspects is within the scope of the present invention as long as it achieves the effects of the present invention.

[0148] <Aspect> [Appendix 1] An underwater monitoring system having an underwater photography system for photographing underwater scenes and a server device capable of communicating with the underwater photography system, The underwater photography system includes: Transmitting the image data of the underwater photograph to the server device, The server device a communication unit that receives the image data from the underwater photography system; an image discrimination means for discriminating the type and population level of aquatic organisms captured in the image data received from the underwater photography system; An underwater monitoring system comprising: [Appendix 2] The image discrimination means, based on a discrimination model of types of aquatic life that has learned the correspondence between image data and types of aquatic life that appear in the image data, The underwater monitoring system described in Appendix 1, characterized in that it determines the type of aquatic organisms captured in the image data received from the underwater photography system. [Appendix 3] After determining the type of aquatic organism, The image discrimination means discriminating the several classes of aquatic organisms captured in the image data received from the underwater photography system based on a several-class discrimination model that has learned correspondences between the image data and the several classes of aquatic organisms captured in the image data; The underwater monitoring system described in Appendix 2 is characterized in that the group level of aquatic organisms depicted in the image data received from the underwater photography system is determined based on a group level discrimination table in which the group level is associated with the type of aquatic organism and the several classes. [Appendix 4] After determining the type of aquatic organism, The underwater monitoring system described in Appendix 2 is characterized in that the image discrimination means discriminates the group level of the aquatic organisms depicted in the image data received from the underwater photography system based on a group level discrimination model learned for each type of aquatic organism, based on the correspondence between the image data and the group level of the aquatic organisms depicted in the image data. [Appendix 5] The underwater monitoring system according to any one of Supplementary Note 1 to 4, wherein the underwater photography system includes a spherical camera. [Appendix 6] 6. The underwater monitoring system according to any one of claims 1 to 5, wherein the underwater photography system further comprises a buoy equipped with a power storage device and a communication device. [Appendix 7] The underwater monitoring system described in Appendix 3, characterized in that the discrimination model for several classes of aquatic organisms is trained using image data classified into several different classes regardless of the type of aquatic organism. [Appendix 8] The underwater monitoring system described in Appendix 4, characterized in that the aquatic organism group-level discrimination model is trained using image data in which the group level is classified by type of aquatic organism. [Appendix 9] The server device requesting the underwater photography system to take underwater photographs at predetermined time intervals; The underwater monitoring system described in any one of Appendices 1 to 8, characterized in that the underwater photography system takes underwater photographs in response to a request from the server device and transmits the captured image data to the server device. [Appendix 10] a user terminal device that can communicate with the server device via a network; The server device a user request processing means for processing a request from the user terminal device; An underwater monitoring system described in any one of Appendices 1 to 8, characterized in that the user request processing means receives the request from the user terminal device and transmits at least one of the aquatic organism type or the group level discrimination result to the user terminal device. [Appendix 11] The server device a camera control means for controlling a camera included in the underwater photography system; When the user request processing means receives a photographing request from the user terminal device, 11. The underwater monitoring system according to claim 10, wherein the camera control means controls the underwater photography system to photograph underwater. [Appendix 12] the user terminal device displays a selection field for accepting either discrimination of the type of aquatic organism, discrimination of the group level, or discrimination of both the type of aquatic organism and the group level, and a discrimination execution button; When the determination execution button is pressed, a photographing request is sent to the server device together with the selected content of the selection field; The underwater monitoring system described in Appendix 10 or 11, characterized in that it receives and displays the determination result determined based on the selection content of the selection field, and the image data captured in real time by the underwater photography system. [Appendix 13] the user terminal device displays a determination result display button, When the determination result display button is pressed, a request for an image determination result regarding the image data is sent to the server device; The underwater monitoring system described in Appendix 10 or 11 is characterized in that the group level of aquatic organisms captured in the image data captured by the underwater photography system, which has already been determined by the image discrimination means, and the image data already captured by the underwater photography system are received and displayed from the server device. [Appendix 14] A server device capable of communicating with an underwater photography system for photographing underwater scenes, a communication unit that receives image data of underwater images from the underwater photography system; an image discrimination means for discriminating the type and population level of aquatic organisms captured in the image data received from the underwater photography system; A server device comprising: [Appendix 15] An image processing method performed by a server device capable of communicating with an underwater photography system that takes pictures underwater, comprising: receiving image data of underwater images from the underwater photography system; a process of determining the type and population level of aquatic organisms captured in the image data received from the underwater photography system; An image processing method comprising: [Appendix 16] A server device that can communicate with an underwater photography system that takes underwater photos. a communication unit that receives image data of underwater images from the underwater photography system; an image discrimination means for discriminating the type and population level of aquatic organisms captured in the image data received from the underwater photography system; A program to function as a [Appendix 17] a storage unit that acquires and stores image data of underwater images; an image discrimination means for discriminating the type and population level of aquatic organisms captured in the image data; An information processing device comprising: [Appendix 18] An image processing method performed by an information processing device, A process of acquiring image data of the underwater image; A process of determining the type and population level of aquatic organisms captured in the image data; An image processing method comprising: [Appendix 19] An information processing device a storage unit that acquires and stores image data of underwater images; an image discrimination means for discriminating the type and population level of aquatic organisms captured in the image data; A program to function as a [Explanation of symbols]

[0149] 1. Underwater monitoring system 5. Camera 15 Underwater photography system 30 Server equipment 40 User terminal equipment [Prior art documents] [Patent documents]

[0150] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-71806

Claims

1. An underwater monitoring system having an underwater photography system for photographing underwater scenes and a server device capable of communicating with the underwater photography system, The underwater photography system includes: Transmitting the image data of the underwater photograph to the server device, The server device a communication unit that receives the image data from the underwater photography system; an image discrimination means for discriminating the type and population level of aquatic organisms captured in the image data received from the underwater photography system; An underwater monitoring system comprising:

2. The image discrimination means, based on a discrimination model of types of aquatic life that has learned the correspondence between image data and types of aquatic life that appear in the image data, 2. The underwater monitoring system according to claim 1, wherein the type of aquatic organisms captured in the image data received from the underwater photography system is identified.

3. After determining the type of aquatic organism, The image discrimination means discriminating the several classes of aquatic organisms captured in the image data received from the underwater photography system based on a several-class discrimination model that has learned correspondences between the image data and the several classes of aquatic organisms captured in the image data; The underwater monitoring system described in claim 2, characterized in that the group level of the aquatic organisms depicted in the image data received from the underwater photography system is determined based on a group level discrimination table in which the group level is associated with the type of aquatic organism and the several classes.

4. After determining the type of aquatic organism, The image discrimination means determines a correspondence between image data and a group level of aquatic organisms shown in the image data based on a group level discrimination model learned for each type of aquatic organism, 3. The underwater monitoring system according to claim 2, wherein the level of the group of fish captured in the image data received from the underwater photography system is determined.

5. 5. The underwater monitoring system according to claim 1, wherein the underwater photography system includes a spherical camera.

6. 2. The underwater monitoring system according to claim 1, wherein the underwater photography system further comprises a buoy equipped with a power storage device and a communication device.

7. The underwater monitoring system described in claim 3, characterized in that the discrimination model for several classes of aquatic organisms is trained using image data classified into several different classes regardless of the type of aquatic organism.

8. The underwater monitoring system of claim 4, wherein the aquatic organism group-level discrimination model is trained using image data in which the group level is classified by type of aquatic organism.

9. The server device requesting the underwater photography system to take underwater photographs at predetermined time intervals; 2. The underwater monitoring system according to claim 1, wherein the underwater photography system takes underwater photographs in response to a request from the server device, and transmits the photographed image data to the server device.

10. a user terminal device that can communicate with the server device via a network; The server device a user request processing means for processing a request from the user terminal device; The underwater monitoring system described in claim 1, characterized in that the user request processing means receives the request from the user terminal device and transmits at least one of the aquatic organism type or group level discrimination results to the user terminal device.

11. The server device a camera control means for controlling a camera included in the underwater photography system; When the user request processing means receives a photographing request from the user terminal device, 11. The underwater monitoring system according to claim 10, wherein the camera control means controls the underwater photography system so that the underwater photography system takes pictures underwater.

12. the user terminal device displays a selection field for accepting either discrimination of the type of aquatic organism, discrimination of the group level, or discrimination of both the type of aquatic organism and the group level, and a discrimination execution button; When the determination execution button is pressed, a photographing request is sent to the server device together with the selected content of the selection field; An underwater monitoring system as described in claim 10 or 11, characterized in that the judgment result judged according to the selection content of the selection field and the image data captured in real time by the underwater photography system are received and displayed.

13. the user terminal device displays a determination result display button, When the determination result display button is pressed, a request for an image determination result regarding the image data is sent to the server device; The underwater monitoring system described in claim 10 or 11, characterized in that the group level of aquatic organisms captured in the image data captured by the underwater photography system, which has already been determined by the image discrimination means, and the image data already captured by the underwater photography system, are received from the server device and displayed.

14. A server device capable of communicating with an underwater photography system for photographing underwater scenes, a communication unit that receives image data of underwater images from the underwater photography system; an image discrimination means for discriminating the type and population level of aquatic organisms captured in the image data received from the underwater photography system; A server device comprising:

15. An image processing method performed by a server device capable of communicating with an underwater photography system that takes pictures underwater, comprising: receiving image data of underwater images from the underwater photography system; a process of determining the type and population level of aquatic organisms captured in the image data received from the underwater photography system; An image processing method comprising:

16. A server device that can communicate with an underwater photography system that takes underwater photos. a communication unit that receives image data of underwater images from the underwater photography system; an image discrimination means for discriminating the type and population level of aquatic organisms captured in the image data received from the underwater photography system; A program to function as a

Images