Fish species identification system, server, fish species identification method, and program

Abstract

To provide a fish kind determination system, a server, a fish kind determination method, and a program that can enhance the precision of a fish kind determination result using a machine learning model.SOLUTION: A fish kind determination system 1 comprises an echo data acquisition part 110 which acquires underwater echo data, a storage part 202, and a control part 201. The storage part 202 is stored with a machine learning model which outputs a prediction probability for each fish kind based upon the echo data, know-how information on each fish kind associated with a fish school set by a user, and a know-how model which corrects the prediction probability of each fish kind based upon the know-how information. The control part 201 determines the fish kind of a fish school based upon a correction result obtained by correcting the prediction probability of each fish kind, acquired with the machine learning model, with the know-how model.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a fish species discrimination system, a server, and a fish species discrimination method for performing fish species discrimination using a machine learning model (machine learning algorithm), and a program for causing a computer to execute a function of performing fish species discrimination using a machine learning model.

Background Art

[0002] Conventionally, a fish school detection device for detecting a fish school in water is known. In this type of fish school detection device, ultrasonic waves are transmitted into water, and the reflected waves are received. Echo data corresponding to the intensity of the received reflected waves is generated, and an echo image is displayed based on the generated echo data. A user can confirm a fish school from the echo image and smoothly proceed with the capture of the fish school.

[0003] In this case, it is preferable that the fish species of the fish school on the echo image can be further discriminated. Thereby, the user can efficiently capture the fish of the fish species he desires.

[0004] For the discrimination of fish species, for example, a machine learning model can be used. In this case, the echo data output from the fish school detector is used as input data, and the fish species of the fish school on the echo data is used as teacher data, and learning for the machine learning model is performed to generate a learned model. The fish species (teacher data) of the fish school on the echo data is input by the user based on actual fishing, for example. The following Patent Document 1 describes this type of fish species estimation system.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The method described above can generate a unique machine learning model for each user. However, because the training data is input by the user based on actual catches, the amount of training data is limited. Therefore, it is difficult to improve the accuracy of the fish species classification results by the machine learning model.

[0007] One possible solution to this problem is to aggregate standard data generated by experts for machine learning purposes and use this aggregated standard data as training data for training machine learning models.

[0008] However, the underwater characteristics of each fish species, such as swimming depth, swimming speed, and schooling patterns, can vary depending on the region. Furthermore, the sea conditions suitable for each fish species, such as water temperature, salinity, and current speed, can also vary depending on the region. Moreover, the best time and location to catch each fish species can also vary depending on the region. For these reasons, if fish species in a school are identified using a machine learning model based on standard data as described above, it is possible that highly accurate identification results may not be obtained in each region.

[0009] In view of these challenges, the present invention aims to provide a fish species identification system, server, fish species identification method, and program that can improve the accuracy of fish species identification results using machine learning models. [Means for solving the problem]

[0010] A first aspect of the present invention relates to a fish species identification system. The fish species identification system according to this aspect comprises an echo data acquisition unit that acquires echo data in water, a storage unit, and a control unit. The storage unit is The system is trained using training data that combines echo data of the range of the fish school and the fish species in that school, and the echo data acquired by the echo data acquisition unit is of the range of the fish school. Based on the aforementioned echo data , against the school of fish By fish species of A machine learning model that outputs predicted probabilities, and the user For each of the aforementioned fish species Set fish school Features Regarding Runo The Uha information and the predicted probability for each of the aforementioned fish species , for each of the aforementioned fish species The aforementioned know-how information The characteristics of the fish school defined in and the characteristics of the fish school within the range of the fish school from which the predicted probability was obtained. Based on By calculationThe control unit stores the know-how model to be modified. Based on the modification result obtained by modifying the predicted probability for each fish species acquired by the machine learning model with the know-how model, The predicted probability was obtained using the aforementioned machine learning model. Identify the species of fish in the aforementioned school.

[0011] According to the fish species identification system of this embodiment, the predicted probability of each fish species by the machine learning model is modified based on a know-how model using know-how information from the user. As a result, the modified predicted probability of each fish species is more likely to fit the region and fishing area to which the user belongs. Therefore, the accuracy of the fish species identification results can be improved.

[0012] In the fish species identification system according to this embodiment, the know-how information may include information regarding the underwater characteristics of the fish of the said species.

[0013] The underwater characteristics of each fish species, such as swimming depth, swimming speed, and schooling patterns, can vary depending on the region. Therefore, by including the underwater characteristics of each fish species in the know-how information, the revised results obtained by correcting the machine learning model's predicted probability for each fish species using the know-how model can be brought closer to the predicted probability that is appropriate for the regional characteristics of each fish species. Thus, the accuracy of the fish species identification results can be improved.

[0014] In the fish species identification system according to this embodiment, the know-how information may include information on sea conditions suitable for the fish of the said species.

[0015] The optimal sea conditions for each fish species, such as water temperature, salinity, and current speed, can vary depending on the region. Therefore, by including the optimal sea conditions for each fish species in the know-how information, the revised results of the machine learning model's predicted probability for each fish species, corrected by the know-how model, can be brought closer to the predicted probability that is appropriate for the regional characteristics of each fish species. Thus, the accuracy of the fish species identification results can be improved.

[0016] In the fish species discrimination system according to this aspect, the know-how information may include information regarding at least one of the capture time and capture location of the fish of the fish species.

[0017] The capture time and capture location where fish of each fish species can be captured may vary depending on the region. Thus, by including know-how information regarding at least one of the capture time and capture location of the fish of each fish species, the corrected result obtained by correcting the prediction probability of each fish species by the machine learning model using the know-how model can be made closer to the prediction probability according to the regional characteristics of the fish of each fish species. Therefore, the accuracy of the fish species discrimination result can be improved.

[0018] In the fish species discrimination system according to this aspect, the control unit may be configured to change the degree of correction of the prediction probability in the know-how model based on feedback information indicating the content of the user's correction to the discrimination result.

[0019] According to this configuration, since the degree of correction of the prediction probability in the know-how model is changed at any time according to the content of the user's correction to the discrimination result, for example, even when the user has erroneously set inappropriate know-how information, the corrected prediction probability can be made closer to the prediction probability according to the actual fish species.

[0020] In the fish species discrimination system according to this aspect, the control unit may be configured to change the degree of correction of the prediction probability in the know-how model based on the learning progress of the machine learning.

[0021] According to this configuration, for example, when the learning progress of the machine learning model is low and the accuracy of the prediction probability of each fish species is low, the degree of correction of the prediction probability in the know-how model is increased, and then, as the learning progress of the machine learning model increases and the accuracy of the prediction probability of each fish species increases, the degree of correction of the prediction probability in the know-how model is reduced. In this way, by the complementary action of the machine learning model and the know-how model, the accuracy of the corrected prediction probability can be efficiently improved, and the accuracy of the fish species discrimination result can be improved.

[0022] The fish species discrimination system according to this aspect may include an underwater detection device that detects a fish school in water, and a server that can communicate with the underwater detection device. Here, the echo data acquisition unit may be arranged in the underwater detection device, and the storage unit and the control unit may be arranged in the server.

[0023] According to this configuration, mainly in the server, construction of a machine learning model and a know-how model necessary for fish species discrimination, and fish species discrimination processing using these are executed. Therefore, the load on the underwater detection device installed on a ship or the like can be reduced, and the fish species discrimination processing can be efficiently executed.

[0024] A second aspect of the present invention relates to a server that can communicate with an underwater detection device that detects a fish school in water. The server according to this aspect includes a storage unit and a control unit. The storage unit stores It has been trained using training data that combines echo data of the range of the fish school and the fish species in that school. echo data received from the underwater detection device The echo data of the range of the fish school based on , against the school of fish for each fish species of a machine learning model that outputs a prediction probability, know-how information for each fish species regarding a fish school set by a user, and a know-how model that corrects the prediction probability for each fish species by the machine learning model based on the know-how information. The control unit discriminates the fish species of the fish school based on the correction result obtained by correcting the prediction probability for each fish species acquired by the machine learning model with the know-how model. For each of the aforementioned fish species set fish school Features by the user , for each of the aforementioned fish species for each fish species The characteristics of the fish school defined in and the characteristics of the fish school within the range of the fish school from which the predicted probability was obtained. for each fish species The predicted probability was obtained using the aforementioned machine learning model. and the know-how information

[0025] A third aspect of the present invention relates to a fish species discrimination method. The fish species discrimination method according to this aspect acquires underwater echo data, calculates a prediction probability for each fish species based on the echo data by a machine learning model, stores know-how information from a user regarding a fish school for each fish species, and A machine learning model trained on training data combining echo data of the fish school range and the fish species of the fish school is used to determine the range of the fish school from the echo data. based on the echo data , against the school of fish for each fish species of calculates a prediction probability for each fish species by a machine learning model, stores know-how information from a user regarding a fish school for each fish species, and Features for each fish species , for each of the aforementioned fish species corrects the prediction probability for each fish species based on the know-how information. The characteristics of the fish school defined in and the characteristics of the fish school within the range of the fish school from which the predicted probability was obtained. based on By calculationCorrected, Repair Based on the corrected predicted probabilities for each fish species The predicted probability was obtained using the machine learning model. Identify the species of fish in the aforementioned school.

[0026] A fourth aspect of the present invention relates to a program that causes a computer to perform a predetermined function. The program according to this aspect is: A machine learning model trained on training data combining echo data of the fish school's range and the fish species in that school is used. Echo data acquired from underwater The echo data of the range of the fish school Based , against the school of fish Predicted probability for each fish species Calculate The function of releasing and the school of fish Features A function to store user know-how information regarding each fish species, and the predicted probability for each fish species. , for each of the aforementioned fish species The aforementioned know-how information The characteristics of the fish school defined in and the characteristics of the fish school within the range of the fish school from which the predicted probability was obtained. Based on By calculation The function to make corrections, Repair Based on the corrected predicted probabilities for each fish species The predicted probability was obtained using the machine learning model. This includes a function for identifying the species of fish in the aforementioned school of fish.

[0027] According to the second and third embodiments described above, the same effects as those of the first embodiment are achieved. [Effects of the Invention]

[0028] As described above, the present invention provides a fish species identification system, server, fish species identification method, and program that can improve the accuracy of fish species identification results using machine learning models.

[0029] The effects and significance of the present invention will become even clearer from the description of the embodiments shown below. However, the embodiments shown below are merely examples of how to implement the present invention, and the present invention is not limited in any way to those described in the embodiments below. [Brief explanation of the drawing]

[0030] [Figure 1] Figure 1 is a diagram showing the configuration of a fish species identification system according to an embodiment. [Figure 2] Figure 2 is a block diagram showing the configuration of a fish species identification system according to an embodiment. [Figure 3] Figure 3 is a diagram showing the management status of various types of information in the storage unit of a server according to this embodiment. [Figure 4] Figures 4(a) to 4(c) are diagrams showing the configuration of individual data according to each embodiment. [Figure 5] Figure 5 is a schematic diagram illustrating a fish species discrimination process using a neural network according to an embodiment. [Figure 6] Figure 6 shows a screen for inputting know-how information according to the embodiment. [Figure 7] Figure 7 illustrates an example of a process in which the predicted probability of each fish species, calculated by a machine learning model, is corrected using a know-how model, according to the embodiment. [Figure 8] Figure 8 is a flowchart showing the fish species identification process according to this embodiment. [Figure 9] Figure 9 is a schematic diagram showing an example of displaying an echo image including the fish species identification result according to the embodiment. [Figure 10] Figure 10(a) is a flowchart showing the feedback information transmission process performed by the control unit of the underwater detection device according to the embodiment. Figure 10(b) is a flowchart showing the feedback information reception process performed by the control unit of the server according to the embodiment. [Figure 11] Figure 11 is a schematic diagram showing a screen for receiving corrections of fish species from a user, according to an embodiment. [Figure 12] Figure 12(a) is a flowchart showing the modification process of the know-how model executed by the control unit of the server according to the embodiment. Figure 12(b) is a flowchart showing an example of the process in step S312 of Figure 12(a) according to the embodiment. [Figure 13] Figures 13(a) and 13(b) show examples of modifications to the know-how model according to the embodiment. [Figure 14]Figure 14(a) is a flowchart illustrating the process for modifying the degree of correction of the prediction probability in the know-how model based on the learning progress of machine learning, as per Modification Example 1. Figure 14(b) is a diagram showing the structure of the table used in the process in Figure 14(a), as per Modification Example 1. [Figure 15] Figure 15 shows an example of how to modify know-how information related to Change Example 2. [Modes for carrying out the invention]

[0031] Figure 1 shows the configuration of the fish species identification system 1.

[0032] The fish species identification system 1 comprises an underwater detection device 10 and a server 20. The underwater detection device 10 is a fish finder installed on the ship 2. The underwater detection device 10 can communicate with the server 20 via an external communication network 30 (for example, the internet) and a base station 40. The underwater detection device 10 and the server 20 each hold address information for communicating with each other. This address information is set in the underwater detection device 10 and the server 20 during initial setup.

[0033] The underwater detection device 10 comprises a transducer 11 and a control unit 12. The transducer 11 is installed on the bottom of the ship 2, and the control unit 12 is installed in the ship's wheelhouse or elsewhere. The transducer 11 and the control unit 12 are connected by a signal cable (not shown). The transducer 11 is equipped with an ultrasonic transducer for transmitting and receiving waves. In response to control from the control unit 12, the transducer 11 transmits ultrasonic waves 3 (transmitted waves) toward the seabed 4 using the ultrasonic transducer and receives the reflected waves. The transducer 11 transmits a received signal based on the received reflected waves to the control unit 12.

[0034] The control unit 12 processes the received signal and generates echo data showing the echo intensity at each depth. The control unit 12 arranges the echo intensity at each depth based on the echo data in chronological order to generate an echo screen for one display. The control unit 12 displays the generated echo screen on the display unit. The control unit 12 updates the echo screen for each ultrasonic wave transmission and reception. By referring to the echo screen, the user can understand the presence and location of the school of fish 5.

[0035] Furthermore, the control unit 12 transmits the generated echo data to the server 20 as needed. The server 20 stores the received echo data and generates echo images similar to those of the control unit 12. The server 20 uses a machine learning model to calculate the predicted probability (the probability that the fish is of that species) for each fish species included in the school of fish in the echo image. Furthermore, the server 20 modifies the calculated predicted probability for each fish species using a know-how model based on know-how information set by the user.

[0036] Here, the know-how information refers to information specific to each fish species regarding schools of fish, and may include the underwater characteristics of each fish species (swimming depth, swimming speed, schooling pattern, etc.), information on suitable sea conditions for each fish species (water temperature, salinity, current speed, etc.), or the timing and location of capture for each fish species.

[0037] Server 20 modifies the predicted probability for each fish species calculated by the machine learning model using a know-how model based on know-how information set by the user, calculates the modified predicted probability for each fish species, and obtains the fish species identification result for the fish school based on the modified predicted probability for each fish species. Server 20 transmits the fish species identification result obtained in this way, along with the range (depth, time) of the fish school to be identified, to the underwater detection device 10 that receives the echo data.

[0038] The underwater detection device 10 overlays the fish species identification results onto the corresponding area of ​​the echo image based on the received identification results and the range (depth, time) of the fish school. This allows the user to confirm the fish species of each fish school on the echo image, enabling them to smoothly proceed with catching the desired fish.

[0039] If the user's fish species identification result provided by the server 20 differs from the actual fish species caught, the user sends feedback information to the server 20 to correct the fish species. For example, after finishing fishing for the day, the user performs an operation to acquire echo data for a predetermined time period of that day from the server 20 via the input unit of the underwater detection device 10. As a result, the server 20 transmits the echo data for the specified day and time period, along with the fish species identification result (including the range of the fish school to be identified), to the underwater detection device 10. Based on the received echo data and identification result, the underwater detection device 10 displays an echo image including the identification result on its display unit.

[0040] The user performs an operation via the input unit to correct the fish species identification result displayed on the echo image to the species of the fish they have caught. As a result, the underwater detection device 10 transmits feedback information to the server 20, including the user's correction to the identification result and the range (depth, time) of the fish school corresponding to that identification result. Based on the received feedback information, the server 20 changes the degree of correction of the prediction probability in the know-how model described above. This optimizes the know-how model so that it reflects the user's actual fishing results.

[0041] Although only one underwater detection device 10 is shown in Figure 1, in reality, numerous underwater detection devices 10 can communicate with the server 20 via the external communication network 30 and the nearest base station. Furthermore, the underwater detection devices 10 that communicate with the server 20 include not only those installed on the ship 2 as shown in Figure 1, but also several types of underwater detection devices used in different fishing methods, such as those installed on fixed nets.

[0042] Figure 2 is a block diagram showing the configuration of the fish species identification system 1.

[0043] The underwater detection device 10 comprises a control unit 101, a display unit 102, an input unit 103, a transmitter / receiver unit 104, a signal processing unit 105, a communication unit 106, and a position detection unit 107. The transmitter / receiver unit 104 and the signal processing unit 105 constitute an echo data acquisition unit 110 that acquires underwater echo data.

[0044] The control unit 101 consists of a microcomputer and memory, etc. The control unit 101 controls each part of the underwater detection device 10 according to a program stored in memory. This program includes functions for receiving and displaying fish species identification results, as well as functions for receiving and transmitting feedback information and know-how information, as described later.

[0045] The display unit 102 includes a monitor and displays a predetermined image under control from the control unit 101. The input unit 103 includes a trackball for moving a cursor on the image displayed on the display unit 102, operation keys, etc., and outputs signals to the control unit 101 in response to user operations. The display unit 102 and the input unit 103 may be integrated using a liquid crystal touch panel or the like.

[0046] The transmitter / receiver unit 104 includes a transducer 11 as shown in Figure 1, a transmitting circuit for supplying a transmission signal to the transducer 11, and a receiving circuit for processing the received signal output from the transducer 11 and outputting it to the signal processing unit 105. The transmitting circuit and the receiving circuit are included in the control unit 12 shown in Figure 1.

[0047] The transmitter / receiver unit 104 transmits a transmission wave (ultrasound) according to the control unit 101. In this sequence, two types of transmission waves with different frequencies are transmitted. The transmitter / receiver unit 104 receives the reflected waves of each transmitted frequency and outputs a received signal. The receiving circuit extracts the received signal for each transmission frequency and outputs it to the signal processing unit 105.

[0048] The reason for transmitting and receiving signals at two different frequencies is to enable more accurate fish species identification, as will be explained later. For example, the presence or absence of a swim bladder causes differences in the echo intensity at each frequency. Therefore, by referring to the differences in echo intensity from a school of fish, the species of fish in that school can be identified with high accuracy.

[0049] The signal processing unit 105 generates echo data indicating the intensity of reflected waves according to depth from the received signals of each frequency input from the transmitter / receiver unit 104, and outputs the two types of generated echo data to the control unit 101. The elapsed time from the timing of transmission of each frequency wave corresponds to the depth. Here, the intensity of the reflected wave attenuates as the depth increases. Therefore, in order to handle the echo data quantitatively regardless of the difference in depth, the signal processing unit 105 corrects the intensity of the reflected wave that attenuates according to the elapsed time, and outputs the echo data with the corrected intensity to the control unit 101.

[0050] The control unit 101 generates an echo image based on the received echo data and displays it on the display unit 102. The control unit 101 generates echo data using echo data corresponding to either one of the frequencies. The user may switch as appropriate which frequency of echo data is used to generate the echo image. The control unit 101 generates a single row of images in the depth direction from the echo data, representing the echo intensity at each depth in gradation using a color scale. The control unit 101 integrates the images in each row from the present time to a predetermined time ago in the time direction to generate a single screen of echo image.

[0051] In the following explanation, when we refer to echo data (including echo data included in feedback information), we mean echo data of two different frequencies unless otherwise specified.

[0052] The communication unit 106 is a communication module capable of wireless communication with the base station 40. The position detection unit 107 is equipped with GPS and detects the position of the underwater detection device 10. The position detection unit 107 outputs the detected position information to the control unit 101.

[0053] As explained with reference to Figure 1, the control unit 101 periodically transmits echo data, feedback information, and know-how information to the server 20 via the communication unit 106. The control unit 101 also receives the fish species identification result from the server 20 via the communication unit 106. Furthermore, the control unit 101 transmits the location information detected by the location detection unit 107 to the server 20.

[0054] As shown in Figure 2, in addition to the underwater detection device 10, numerous other underwater detection devices 10a, 10b, ... can communicate with the server 20 via the external communication network 30 and the nearest base stations 40a, 40b, .... As described above, the underwater detection devices 10 that communicate with the server 20 include not only those installed on the ship 2 as shown in Figure 1, but also several types of underwater detection devices used for different fishing methods, such as underwater detection devices installed on fixed nets. The basic configuration of the other underwater detection devices is the same as that of the underwater detection device 10 in Figure 2.

[0055] However, an underwater detection device installed in a fixed net may consist of an offshore unit installed in the fixed net and a terminal that can communicate with this offshore unit via an external communication network, allowing the user to remotely monitor the condition of fish within the net. Echo data acquired by the offshore unit is transmitted to the terminal via the external communication network. This displays the echo image on the terminal. The terminal may be a personal computer, a mobile phone, a tablet, or any other portable device owned by the user.

[0056] In this case, the terminal may transmit echo data to the server 20, or the offshore unit may transmit echo data to the server 20 in parallel with transmitting echo data to the terminal. Feedback information and know-how information may be input via the terminal and transmitted from the terminal to the server 20. The fish species identification result may be transmitted directly from the server 20 to the terminal without going through the offshore unit. Furthermore, the server 20 may transmit echo data to the terminal. That is, the server 20 may receive echo data from the offshore unit and transmit the received echo data to the terminal.

[0057] The server 20 comprises a control unit 201, a storage unit 202, and a communication unit 203. The control unit 201 is composed of a CPU, etc. The storage unit 202 is composed of ROM, RAM, hard disk, etc. The storage unit 202 stores a program for fish species identification. The control unit 201 controls each part according to the program stored in the storage unit 202. The communication unit 203 communicates with the underwater detection device 10 via the external communication network 30 and base station 40 under control from the control unit 201.

[0058] The control unit 201 generates a know-how model applicable to each underwater detection device 10 using the above program. The control unit 201 also stores the echo data, feedback information, and know-how information received from each underwater detection device 10 in the storage unit 202, associating them with each underwater detection device 10. The control unit 201 generates a know-how model applicable to each underwater detection device 10 using the know-how information received from each underwater detection device 10, and further updates the know-how model applicable to each underwater detection device 10 using the feedback information received from each underwater detection device 10.

[0059] Furthermore, the echo data transmitted from each underwater detection device 10 to the server 20 may be thinned down to a predetermined granularity before transmission in order to reduce communication traffic and the capacity load on the server 20. In this case, the server 20 uses the echo data corrected for thinning by interpolation to perform fish species discrimination and machine learning. Alternatively, fish species discrimination and machine learning may be performed using the thinned echo data. However, in order to perform fish species discrimination and machine learning with higher accuracy, it is preferable to use echo data corrected for thinning by interpolation for fish species discrimination and machine learning.

[0060] Furthermore, in order to quantitatively process the echo data received from each underwater detection device 10, the server 20 may perform corrections on the echo data received from each underwater detection device 10 based on underwater acoustic theory, taking into account the characteristics of the underwater detection device 10 and the transducer 11 (e.g., sensitivity, amplification factor, etc.) to perform fish species discrimination and machine learning. This makes it possible to perform fish species discrimination by machine learning models and machine learning on machine learning models with greater accuracy.

[0061] Figure 3 shows the management status of various types of information in the storage unit 202 of the server 20.

[0062] The memory unit 202 stores standard data 301, machine learning models 302, ocean condition data 303, individual data 311 and 321, and know-how models 312 and 322.

[0063] Standard Data 301 is standard training data for machine learning. Standard Data 301 combines echo data of fish school ranges (depth, time) with data on the fish species within those schools. Standard Data 301 is continuously generated by experts and registered by administrators. As a result, the amount of data in the standard dataset gradually increases.

[0064] Machine learning model 302 is a machine learning model generated by machine learning using standard data 301. Whenever standard data 301 is updated, machine learning is performed on machine learning model 302, and machine learning model 302 is updated. In this way, the learning progress of machine learning model 302 increases.

[0065] In this embodiment, machine learning using a neural network is applied. For example, a neural network using deep learning, which combines neurons in multiple stages, is applied. However, the machine learning applied is not limited to this, and other machine learning methods such as support vector machines and decision trees may also be applied.

[0066] The ocean condition data 303 is data relating to ocean conditions such as water temperature, salinity, and current speed. The ocean condition data 303 is detected by detectors installed on buoys and other objects at sea, and is periodically transmitted from each detector to the server 20 via wireless communication. The detectors are equipped with GPS and transmit the location information detected by GPS along with the ocean condition data to the server 20. The server 20 stores the ocean condition data in the storage unit 202 for each buoy (detector) location. If the underwater detection device 10 is equipped with detectors for acquiring ocean condition data, the ocean condition data may be transmitted from the underwater detection device 10 to the server 20 along with the location information.

[0067] Individual data 311 and 321 are data acquired from each user's underwater detection device 10. Know-how models 312 and 322 are models for correcting the predicted probability for each fish species calculated by the machine learning model 302, as described above, based on each user's know-how information.

[0068] In Figure 3, individual data 311 and know-how model 312 are for user U1, and individual data 321 and know-how model 322 are for user U2. Similarly, individual data, know-how information, and know-how models are managed for each user other than U1 and U2.

[0069] Figures 4(a) to 4(c) show the structure of individual data.

[0070] As shown in Figures 4(a) to 4(c), various types of individual data are managed in association with the user ID. The user ID is information used to identify the user (underwater detection device 10). For example, the product code of the underwater detection device 10 may be used as the user ID, or a randomly assigned code may be used as the user ID. The user ID is transmitted and received as needed when information is transmitted and received between the underwater detection device 10 and the server 20.

[0071] Figure 4(a) shows individual data related to echo data. The underwater detection device 10 sequentially transmits the echo data obtained by one sequence of transmit and receive waves to the server 20 along with the date and time of acquisition. The storage unit 202 of the server 20 stores the start date and time and end date and time of acquisition of the echo data, as well as a group of echo data acquired during that period and their acquisition dates and times, associated with the user ID.

[0072] Furthermore, the fish species classification results obtained by a machine learning model from a group of echo data from the start date to the end date are further associated with the echo data of each group. If multiple fish species classification results are obtained, all of these results are associated with a group of echo data. Each fish species classification result consists of the range of the fish group (depth, time) and the classification result (fish species).

[0073] Figure 4(b) shows individual data related to feedback information. Here, feedback information acquired from the underwater detection device 10 corresponding to the user ID is stored in chronological order. Figure 4(c) shows individual data related to know-how information. Here, know-how information acquired from the underwater detection device 10 corresponding to the user ID is stored in chronological order.

[0074] Figure 5 schematically illustrates the fish species discrimination process using a neural network.

[0075] The control unit 201 of server 20 extracts the range (depth, time) of a school of fish from the echo data for one screen to be processed. The range of a school of fish is extracted as a range on the echo image where the echo intensity is above a predetermined threshold and there is a connection between the echo intensities. The method for extracting the range of a school of fish may be incorporated by reference from the description in International Publication No. 2019 / 003759, which was previously filed by the applicant.

[0076] The control unit 201 applies the echo data of the extracted fish school range to the input 302a of the machine learning model (machine learning algorithm using a neural network) 302 shown in Figure 5.

[0077] The output 302b of the machine learning model 302 is assigned items for fish species such as sardines, horse mackerel, and mackerel. When echo data of the range of the fish school is applied to the input 302a of the machine learning model 302, the probability (prediction probability) that the fish species in the fish school is the species of the item is output from each item in the output 302b of the machine learning model 302. In the example in Figure 5, the mackerel item outputs a prediction probability of 85%, the sardine item outputs a prediction probability of 70%, and the tuna item outputs a prediction probability of 10%.

[0078] The predicted probability for each item is modified by the know-how model 312. As described above, the know-how model 312 is a model (algorithm) that modifies the predicted probability for each fish species (item) based on the user's know-how information. In the example in Figure 5, the predicted probability for the mackerel item is modified from 85% to 25%, the predicted probability for the sardine item is modified from 70% to 94%, and the predicted probability for the tuna item is modified from 10% to 1%.

[0079] The corrected predicted probability for each item is compared against output condition 401. Output condition 401 applies, for example, the condition that the fish species whose corrected predicted probability is above a predetermined lower limit and ranks highest is output as the classification result 402. The lower limit is set to prevent fish species with low accuracy from being output as classification results. In the example in Figure 5, sardines, whose corrected predicted probability is 94%, are output as the fish species classification result 402.

[0080] Machine learning for the machine learning model 302 is performed by sequentially applying a series of training data to the input 302a and output 302b of the machine learning model 302. Specifically, the input 302a of the machine learning model 302 is input to the school of fish echo data contained in one training data set, and the output 302b of the machine learning model 302 is set to 100% for the item corresponding to the fish species contained in this training data, and to 0% for the other items, and machine learning is performed.

[0081] The machine learning model 302 shown in Figure 3 is generated by sequentially setting the standard data 301 (echo data of the fish school range, fish species) as input 302a and output 302b of the machine learning model 302 and performing machine learning.

[0082] Furthermore, in addition to the echo data of the fish school, the input 302a of the machine learning model 302 may also contain other information that can be used for fish species discrimination, such as the location where the echo data was obtained and oceanographic data for that location. In this case, the standard data may also include this additional information.

[0083] Figure 6 shows an example of the configuration of the input screen 500 for know-how information.

[0084] The input screen 500 includes a fish species setting item 501, a know-how input area 502, a reliability input area 503, and a confirmation key 504. The input screen 500 is displayed on the display unit 102 in Figure 2 and accepts input from the user via the input unit 103.

[0085] Fish species setting item 501 is an item for the user to set the fish species to which the know-how information will be set. When fish species setting item 501 is selected, a dropdown list of candidate fish species will be displayed directly below it. The user selects the fish species in which they intend to input the know-how information from the displayed candidate species. As a result, the selected fish species will be displayed in fish species setting item 501. In the example in Figure 6, mackerel is shown as selected.

[0086] The know-how input area 502 is an item for users to input their own know-how information. The know-how input area 502 includes a label 502a indicating the type of know-how information and input fields 502b for users to input these types of know-how information. Here, swimming depth, swimming speed, schooling patterns, appropriate water temperature, fishing season, and fishing location are given as examples of know-how information that can be entered. However, the types of know-how information are not limited to these.

[0087] Swimming depth is the range of depths at which the fish species in question (in this case, mackerel) swims, and swimming speed is the average swimming speed of the fish species. Schooling pattern is how the fish of this species school (how they form schools). Optimal water temperature is the water temperature suitable for the fish species, and fishing season and fishing location are the time and place where the fish of this species are caught. The user appropriately enters each type of know-how information for catching the fish of this species in their own fishing grounds into input field 502b. When input field 502b for schooling pattern is selected, a selection list will be displayed directly below this input field 502b in a pre-down list. The fishing location is set within a range of longitude and latitude.

[0088] The reliability input area 503 is an area for inputting the reliability (confidence level) of each piece of know-how information. The reliability input area 503 includes items for selecting whether the reliability of each piece of know-how information contained in the know-how input area 502 is high or average.

[0089] After setting the fish species in the fish species setting item 501, the user enters the information for the item they wish to set from the various types of know-how information displayed in the know-how input area 502 into the input item 502b. Furthermore, the user enters the confidence level (confidence level) for each piece of know-how information they have entered by selecting either "high" or "normal" in the confidence level input area 503.

[0090] Once the input to the fish species setting item 501, the know-how input area 502, and the reliability input area 503 is complete, the user operates the confirmation key 504. As a result, the control unit 101 of the underwater detection device 10 transmits the know-how information and reliability entered into the know-how input area 502 and the reliability input area 503, respectively, along with the fish species set in the fish species setting item 501, to the server 20. The server 20 stores this received information in the storage unit 202 as the individual data of the user in Figure 3, as shown in Figure 4(c).

[0091] Figure 7 illustrates an example of a process in which the predicted probability of each fish species, generated by a machine learning model, is corrected using a know-how model.

[0092] In this example, a know-how model is generated based on know-how information that the user has set to a high level of confidence in the input screen 500 in Figure 6. Since the user has set a high level of confidence in the swimming depth of mackerel and sardines and the timing of tuna catches, a know-how model is generated using this know-how information. Specifically, the know-how information used to generate the know-how model is as follows:

[0093] • Mackerel swimming depth: 40-120m • Swimming depth of sardines: 0-30m (shallower than 30m) • Tuna fishing season: March to July

[0094] The reliability of this know-how information was 100% during the initial user setup, but it has been changed as follows through adjustments based on subsequent feedback.

[0095] • Mackerel swimming depth: 70% • Swimming depth of sardines: 80% • Tuna fishing season… 90%

[0096] The know-how model corrects the predicted probability of each fish species by the machine learning model using, for example, the following calculation formula.

[0097] <When the school of fish to be identified meets the conditions of the know-how information> Corrected predicted probability = 100% × {1 - (1 - Rp) × (1 - Rn)} …(1)

[0098] <If the fish school to be identified does not meet the conditions of the know-how information> Corrected predicted probability = predicted probability × (1 - Rn) …(2)

[0099] In equations (1) and (2) above, Rp is a decimal value representing the predicted probability for the target fish species, and Rn is a decimal value representing the reliability of the know-how information for the target fish species. For example, in the example in Figure 7, if the target fish species is mackerel, Rp is 0.85 and Rn is 0.7. Also, if the target fish species is sardine, Rp is 0.7 and Rn is 0.8.

[0100] In the example in Figure 7, the depth range of the school of fish targeted for species identification was 10-20m. In contrast, the swimming depth of mackerel according to the know-how information is 40-120m, so the swimming depth of the school of fish does not meet the conditions for the swimming depth of mackerel according to the know-how information. For this reason, the predicted probability of mackerel by the machine learning model (85%) is corrected by equation (2) above. As a result, the corrected predicted probability of mackerel is calculated to be 25.5%, and this value is rounded to the first decimal place to become 26%.

[0101] On the other hand, since the swimming depth of sardines in the know-how information is 0-30m, the swimming depth of the fish school targeted for species identification satisfies the conditions for the swimming depth of sardines in the know-how information. For this reason, the prediction probability of sardines by the machine learning model (70%) is corrected by the above equation (1). As a result, the corrected prediction probability of sardines becomes 94%.

[0102] Furthermore, since the tuna fishing season in the know-how information is from March to July, the fishing season for the school of fish targeted for species identification (October 2nd) does not meet the conditions for the tuna fishing season in the know-how information. For this reason, the predicted probability of tuna by the machine learning model (10%) is corrected by the above equation (1). As a result, the corrected predicted probability of tuna becomes 1%.

[0103] Thus, in the corrected prediction probability, the sardine ranks first in prediction probability. Therefore, the classification result 402 in Figure 5 is obtained as sardine.

[0104] In the example shown in Figure 7, there was only one type of know-how information for each fish species. However, if there were multiple types of know-how information for each fish species, the revised prediction probability could be calculated using, for example, the following formula.

[0105] Corrected predicted probability = Σ(Result calculated using Equation 1 or Equation 2 above) / Number of types of know-how information …(3)

[0106] In other words, for each type of know-how information for the target fish species, the corrected predicted probability is calculated using either formula (1) or formula (2) above, and the average of all the calculated predicted probabilities is obtained as the corrected predicted probability for that target fish species.

[0107] Furthermore, the method for correcting the predicted probability in the know-how model is not limited to the methods shown in equations (1) to (3) above. Other correction methods may be used as long as the correction results reflect the user's know-how information.

[0108] Furthermore, in the method described above, only the know-how information set to high confidence in the input screen 500 of Figure 6 was used to generate the know-how model. However, know-how information set to normal confidence in the input screen 500 may also be used to generate the know-how model. In this case, for example, the normal confidence level is set to 50% at the time of setup and is changed based on subsequent feedback information. Then, from all the know-how information with high and normal confidence levels, the corrected prediction probability is calculated for each fish species using, for example, equations (1), (2), and (3) above.

[0109] Furthermore, in the input screen 500 of Figure 6, the confidence input area 503 may be omitted. In this case, the know-how information entered in the know-how input area 502 is initially set to a confidence level of 100%, and then modified based on subsequent feedback information. Then, from all the know-how information set by the user, the corrected prediction probability is calculated for each fish species, for example, using formula (3) above.

[0110] Figure 8 is a flowchart showing the fish species identification process.

[0111] When the control unit 201 of the server 20 begins receiving echo data from the underwater detection device 10 (S101:YES), it stores the received echo data in the storage unit 202 as individual data as shown in Figure 3 (S102). The control unit 201 also sequentially constructs an echo image from the received echo data and identifies the range (depth, time) of the fish school on the echo image. The control unit 201 then applies the echo data of the identified fish school range to a machine learning model to calculate a predicted probability for each fish species (S103). Furthermore, the control unit 201 modifies the calculated predicted probability using the know-how model of the underwater detection device 10 (user ID) as described above, and obtains the modified predicted probability (S104). The control unit 201 then applies the modified predicted probability to the output condition 401 in Figure 5 to determine the fish species identification result for the fish school (S105).

[0112] The control unit 201 transmits the acquired fish species identification result, along with the range (depth, time) of the fish school from which the identification result was obtained, to the underwater detection device 10, and further stores this information in the storage unit 202 (S106).

[0113] Subsequently, the control unit 201 repeatedly executes the processes in steps S102 to S106 until it finishes receiving echo data from the underwater detection device 10 (S107:NO). As a result, each time a new range (depth, time) of a school of fish is identified from the echo image, the species of fish in that school is determined by the machine learning model and the user's know-how model. The newly obtained fish school determination result and the range (depth, time) of that school are transmitted to the underwater detection device 10 as needed and stored in the storage unit 202 of the server 20.

[0114] Thus, when the reception of echo data from the underwater detection device 10 is completed (S107:YES), the control unit 201 terminates the process shown in Figure 8. As a result, one row of individual data shown in Figure 4(a) is stored in the storage unit 202. As described above, the echo data column in Figure 4(a) holds all the echo data received from the underwater detection device 10 during the process shown in Figure 8. In addition, the fish species identification result column in Figure 4(a) holds all the fish species identification results obtained through the process shown in Figure 8, along with the range (depth, time) of the fish school.

[0115] Figure 9 schematically shows an example of the display of an echo image P1 including the fish species identification result. For convenience, in Figure 9, depth lines are added only to the areas with high echo intensity.

[0116] When the control unit 201 of the underwater detection device 10 receives the discrimination result and the range (depth, time) of the fish school from the server 20, it displays a frame-shaped marker M0 indicating the range of the fish school in an area on the echo image P1 corresponding to the depth width and time width corresponding to the received range of the fish school. Furthermore, the control unit 201 displays a label L0 indicating the discrimination result of the received fish species around this marker M0. In the example in Figure 9, based on the discrimination result and the range (depth, time) of the fish school received from the server 20, markers M0 are displayed for fish schools F1 to F8, and furthermore, labels L0 indicating the discrimination result of the fish species are displayed around these markers M0. The current date and time are displayed near the upper left corner of the echo image P1.

[0117] In the example shown in Figure 9, the fish species classification result for fish school F9 was not output by the machine learning model 302, know-how model 312, and output condition 401 in Figure 5. Therefore, no markers or labels are displayed for fish school F9. This can occur, for example, if the corrected predicted probability of fish school F9 by the machine learning model 302 and know-how model 312 does not satisfy the output condition 401. For example, if the output condition is to output the fish species with the highest predicted probability that is above a predetermined lower limit, then if the corrected predicted probability of the first-ranked fish species by the machine learning model 302 and know-how model 312 is below this lower limit, no classification result will be output for this fish school. In such a case, the classification result for this fish school is not transmitted from the server 20 to the underwater detection device 10, and as shown for fish school F9 in Figure 9, the fish species classification result is not displayed.

[0118] Figure 10(a) is a flowchart showing the feedback information transmission process performed by the control unit 101 of the underwater detection device 10. Figure 10(b) is a flowchart showing the feedback information reception process performed by the control unit 201 of the server 20.

[0119] For example, if the user detects a discrepancy between the species identification result of a predetermined school of fish displayed in the echo image P1 and the species of that school of fish that they actually caught, or if they actually catch a school of fish for which no species identification result is displayed in the echo image P1 and determine the species of that school, the user performs an operation (feedback operation) to the underwater detection device 10 via the input unit 103 in Figure 2 to send feedback information to the server 20 to correct the species of these schools of fish. In this case, the user inputs the date and time range of the echo image containing the school of fish to be corrected via the input unit 103, and then performs an operation to obtain the species identification result and echo data (hereinafter referred to as "history information") for that range from the server 20.

[0120] Referring to Figure 10(a), when the control unit 101 of the underwater detection device 10 receives feedback input from the user via the input unit 103 (S201:YES), it sends a request to the server 20 to transmit history information including the date and time range entered by the user, and obtains the history information from the server 20 (S202). Referring to Figure 10(b), when the control unit 201 of the server 20 receives the request to transmit history information transmitted in step S202 of Figure 10(a) (S301:YES), it extracts the history information (echo data and fish species identification result) for the date and time range included in the transmission request from the individual data of the underwater detection device 10 that sent the transmission request, and transmits the extracted history information to the underwater detection device 10 that sent the request (S302).

[0121] Referring to Figure 10(a), when the control unit 101 of the underwater detection device 10 receives history information from the server 20 (S202), it displays an echo screen based on the received history information on the display unit 102 and accepts corrections of the fish species from the user (S203).

[0122] Figure 11 schematically shows the screen used to receive a modification request for the fish species from the user in step S203 of Figure 10(a).

[0123] The control unit 101 of the underwater detection device 10 displays the echo image and discrimination result for the first time period within the time range specified by the user on the display unit 102 when a request for transmission of history information is received. The user operates the scroll bar B0 via the input unit 103 to transition the echo images in the time direction and display a screen containing the echo image and discrimination result for the desired time period. This displays the time period screen shown in Figure 11.

[0124] On this screen, the user specifies the marker M0 of the fish school they wish to modify via the input unit 103. In the screen shown in Figure 11, the user has specified the marker M0 of fish school F5, which has been identified as mackerel. As a result, the marker M0 of fish school F5 is highlighted, and the candidate fish species C0 are displayed around this marker M0 along with a scroll bar. The user operates the scroll bar of the candidate C0 to display the desired fish species, and then selects the fish species they wish to change. In the example in Figure 9, sea bream is selected as the fish species of fish school F5.

[0125] Furthermore, on this screen, if the user wants to input a fish species for a school of fish for which no identification result has been assigned, they specify the range of the school of fish via the input unit 103. As shown in Figure 7, no fish species identification result has been obtained for school of fish F9. When the user wants to input a fish species for school of fish F9, they specify the range of school of fish F9 via the input unit 103. As a result, as shown in Figure 11, a new marker M1 is displayed in the specified range of school of fish F9, and around this marker M1, the candidate fish species C0 are displayed along with a scroll bar. The user operates the scroll bar of the candidate fish species C0 to display the desired fish species, and then selects the fish species they want to input. In the example in Figure 9, tuna is selected as the fish species for school of fish F9.

[0126] After performing operations to change or set the fish species, the user inputs an operation to confirm these operations via the input unit 103.

[0127] Referring to Figure 10(a), when the user inputs a confirmation operation (S204:YES), the control unit 101 sends feedback information to the server 20 in step S203, including the range of the fish school specified by the user and the fish species entered by the user for that fish school (S205). With this, the control unit 101 terminates the process shown in Figure 10(a).

[0128] Referring to Figure 10(b), when the control unit 201 of the server 20 receives feedback information from the control unit 101 of the underwater detection device 10 (S303: YES), it stores the received feedback information in the storage unit 202 as individual data for the underwater detection device 10 (S304). As a result, one line of feedback information as shown in Figure 4(b) is stored in the storage unit 202. With this, the control unit 201 terminates the process shown in Figure 10(b).

[0129] Figure 12(a) is a flowchart showing the know-how model modification process executed by the control unit 201 of the server 20.

[0130] When the control unit 201 receives feedback information from the underwater detection device 10 (S401:YES), it modifies the know-how model corresponding to the underwater detection device 10 based on the received know-how information (S402).

[0131] Figure 12(b) is a flowchart showing an example of the process in step S402 of Figure 12(a).

[0132] The control unit 201 compares the fish school information included in the feedback information with the conditions of the user's know-how information for each fish species, and extracts the know-how information that includes the fish school information according to those conditions from the user's individual data (S411). The fish school information includes the depth range, day and time width, location (longitude, latitude), and ocean condition data for that location. The know-how information used for comparison is the know-how information used to generate the know-how model, for example, in the example in Figure 6, the know-how information with a high reliability setting.

[0133] Next, the control unit 201 compares the fish species of each extracted know-how information with the corrected fish species that the user has modified for that fish group (S412). Then, the control unit 201 increases the confidence level of the know-how information for which the corrected fish species matches (S413), and decreases the confidence level of the know-how information for which the corrected fish species does not match (S414).

[0134] The increase and decrease in confidence in steps S413 and S414 are performed, for example, by changing the confidence by a predetermined value (e.g., 5%). However, the method of increasing and decreasing confidence is not limited to this; for example, it may be done by changing the confidence by a predetermined percentage. Also, the increase and decrease in confidence do not have to be the same; for example, the increase may be greater than the decrease. The upper limit of confidence is 100%. The lower limit of confidence may be set by default to a predetermined value (e.g., 50%), or it may be set by the user.

[0135] Figures 13(a) and (b) show examples of how the know-how model is modified by the process shown in Figure 12(b).

[0136] In this example, in step S411 of Figure 12(b), the swimming depth is extracted from the know-how information for mackerel and sea bream. Specifically, the depth of the school of fish for which the user modified the fish species was included in the swimming depth conditions of 40-120m and 60-150m for mackerel and sea bream, which are part of the user's know-how information. Therefore, this know-how information is extracted in step S411 of Figure 12(b).

[0137] In this example, for instance, the fish species of the fish group F5 in Figure 11 is changed from mackerel to sea bream. As a result, as shown in Figure 13(a), the confidence level of the know-how information regarding the swimming depth of the extracted mackerel is reduced by 5% in step S414 of Figure 12(b). Conversely, as shown in Figure 13(b), the confidence level of the know-how information regarding the swimming depth of the extracted sea bream is increased by 5% in step S413 of Figure 12(b).

[0138] The reliability of each piece of corrected know-how information is revised, thereby correcting the user's know-how model. In the next fish species identification process, the machine learning model's predicted probability for each fish species is revised using the revised know-how model, and the fish species identification result is obtained based on the revised predicted probabilities.

[0139] <Effects of the Embodiment> According to the embodiment, the following effects may be achieved.

[0140] As explained with reference to Figures 5 and 7, the prediction probability of each fish species by the machine learning model is modified based on a know-how model that uses know-how information from users. As a result, the modified prediction probability of each fish species is more likely to fit the user's region and fishing area. Therefore, the accuracy of the fish species identification results can be improved.

[0141] As shown in Figure 6, the know-how information includes information about the underwater characteristics of each fish species (in this case, swimming depth, swimming speed, and schooling behavior). The underwater characteristics of each fish species, such as swimming depth, swimming speed, and schooling behavior, can vary depending on the region. Therefore, by including the underwater characteristics of each fish species in the know-how information, the corrected results obtained by modifying the machine learning model's predicted probability for each fish species using the know-how model can be brought closer to the predicted probability that corresponds to the regional characteristics of each fish species. Thus, the accuracy of the fish species identification results can be improved.

[0142] As shown in Figure 6, the know-how information includes information on suitable ocean conditions (optimal water temperature) for each fish species. Suitable ocean conditions for each fish species, such as water temperature, salinity, and current speed, can vary depending on the region. Therefore, by including suitable ocean conditions for each fish species in the know-how information, the corrected results obtained by modifying the machine learning model's prediction probability for each fish species can be brought closer to the prediction probability that corresponds to the regional characteristics of each fish species. Thus, the accuracy of the fish species identification results can be improved.

[0143] As shown in Figure 6, the know-how information includes information on the timing and location of catch for each fish species. The timing and location of catch for each fish species may vary depending on the region. Therefore, by including at least one of the timing and location of catch for each fish species in the know-how information, the corrected result obtained by modifying the machine learning model's predicted probability for each fish species can be brought closer to the predicted probability that corresponds to the regional characteristics of each fish species. Thus, the accuracy of the fish species identification result can be improved.

[0144] As shown in Figures 12(a) and (b), the control unit 201 changes the degree of correction of the predicted probability in the know-how model (the reliability of the know-how information) based on feedback information indicating the user's modifications to the discrimination result. As a result, the degree of correction of the predicted probability in the know-how model (the reliability of the know-how information) is changed in real time according to the user's modifications to the discrimination result. For example, even if the user mistakenly sets inappropriate know-how information, the corrected predicted probability can be brought closer to the predicted probability corresponding to the actual fish species.

[0145] In the process shown in Figure 12(b), the degree of correction of the prediction probability in the know-how model was changed by changing the reliability of each piece of know-how information. However, the method for changing the degree of correction of the prediction probability in the know-how model is not limited to this. For example, if the prediction probability is corrected by the following equations (4) and (5) instead of the above equations (1) and (2), the adjustment rate Ra may be corrected by the feedback information.

[0146] <When the school of fish to be identified meets the conditions of the know-how information> Corrected predicted probability = predicted probability × (1 + Ra) …(4)

[0147] <If the fish school to be identified does not meet the conditions of the know-how information> Corrected predicted probability = predicted probability × (1-Ra) …(5)

[0148] Here, the initial value of the adjustment rate Ra is set to, for example, 0.5, and the adjustment rate Ra is changed within the range of 0.1 to 1 based on subsequent feedback information. In this case, for example, in step S413 of Figure 12(b), the adjustment rate Ra applied to the know-how information to be processed is increased by 0.05, and in step S414, the adjustment rate Ra applied to the know-how information to be processed is decreased by 0.05.

[0149] As shown in Figure 2, the fish species identification system 1 comprises an underwater detection device 10 for detecting schools of fish underwater and a server 20 capable of communicating with the underwater detection device 10. Here, the echo data acquisition unit 110 is located in the underwater detection device 10, while the storage unit 202 for storing machine learning models, know-how information, and know-how models, and the control unit 201 for identifying the fish species of the school using these are located in the server 20. With this configuration, the construction of machine learning models and know-how models necessary for fish species identification, and the processing of fish species identification using these are mainly performed in the server 20. Therefore, the fish species identification process can be performed efficiently while reducing the burden on the underwater detection device 10 installed on a ship or the like.

[0150] <Example of change 1> The present invention is not limited to the above embodiments, and various modifications are possible to the embodiments of the present invention other than the above configuration.

[0151] For example, the degree to which the prediction probability in the know-how model is corrected may be changed based on the learning progress of the machine learning model.

[0152] Figure 14(a) is a flowchart showing the process of changing the degree of correction of the prediction probability in the know-how model based on the learning progress of machine learning.

[0153] Each time the control unit 201 updates the machine learning model with new training data (S501:YES), it sets a correction rate according to the learning depth of the machine learning (S502). In this case, equations (1) and (2) above are modified as follows, for example.

[0154] <When the fish school to be discriminated satisfies the know-how information conditions> Corrected prediction probability = 100% × {1 - (1 - Rp) × (1 - Rn × Rm)} …(6)

[0155] <When the fish school to be discriminated does not satisfy the know-how information conditions> Corrected prediction probability = Prediction probability × (1 - Rn × Rm) …(7)

[0156] Rm in the above formulas (6) and (7) is the correction rate in step S502 of FIG. 14(a). The initial value of the correction rate Rm is 1, and it is reduced from 1 as the learning progress of the machine learning model increases. Therefore, as the learning progress of the machine learning model increases, the influence of the reliability Rn (minority representation) in formulas (6) and (7) is weakened, and the degree of correction of the prediction probability in the know-how model is reduced.

[0157] The control unit 201 sets the correction rate Rm, for example, according to the table in FIG. 14(b). The learning progress is defined, for example, by the total number of teacher data used for the learning of the machine learning model. When the learning progress reaches P1, the correction rate Rm is set from 1 to R1 (R1 < 1), and then when the learning progress reaches P2, the correction rate Rm is set from R1 to R2 (R2 < R1). Similarly, hereinafter, each time the next learning progress is reached, the correction rate Rm is reduced. The difference in the correction rate Rm between the learning progresses does not necessarily have to be constant.

[0158] Also, when the above formulas (4) and (5) are used, they may be corrected as follows.

[0159] <When the fish school to be discriminated satisfies the know-how information conditions> Corrected prediction probability = Prediction probability × (1 + Ra × Rm) …(8)

[0160] <When the fish school to be discriminated does not satisfy the know-how information conditions> Corrected prediction probability = Prediction probability × (1 - Ra × Rm) …(9)

[0161] When formulas (6), (7), or (8), (9) are used, formula (3) is used as appropriate.

[0162] According to the process shown in Figure 14(a), for example, if the learning progress of the machine learning model is low and the accuracy of the predicted probability for each fish species is low, the degree of correction of the predicted probability in the know-how model is increased. Subsequently, as the learning progress of the machine learning model increases and the accuracy of the predicted probability for each fish species improves, the degree of correction of the predicted probability in the know-how model is reduced. In this way, the machine learning model and the know-how model work complementaryly to each other, making it possible to efficiently improve the accuracy of the corrected predicted probability and thus improve the accuracy of the fish species classification result.

[0163] Furthermore, the correction rate Rm set in Figure 14(a) does not necessarily have to be applied to all fish species. For example, for fish species where the discrimination accuracy remains low even as the learning progress of the machine learning model increases, i.e., fish species where the user frequently corrects the discrimination result, the correction rate Rm may be set to 1, and the prediction probability by the machine learning model may be corrected by the know-how model in the same way as in equations (1) and (2) above.

[0164] <Example of change 2> In the above embodiment, the know-how information for each fish species set by the user was used directly to generate the know-how model. However, if, for example, the know-how information set by the user overlaps between fish species, the control unit 201 may modify the know-how information so that no overlap occurs between fish species.

[0165] Figure 15 shows an example of how to modify know-how information.

[0166] In this example, the know-how information (in this case, swimming depth) for each fish species on the left side overlaps with each other. In this case, the control unit 201 modifies the know-how information for each fish species so that there is no overlap between fish species. The right side of Figure 15 shows the modified know-how information. In this case, the control unit 201 uses the modified know-how information to perform the same fish species discrimination process as in the above embodiment.

[0167] It should be noted that the method for modifying the know-how information is not limited to the method shown in Figure 15. For example, the swimming depth of mackerel may be modified to 55m-120m, and the swimming depth of sardines may be modified to 20-55m. Also, in the example in Figure 15, the know-how information for each fish species was modified to completely eliminate duplication, but the know-how information for each fish species may also be modified to reduce the range of duplication. For example, the swimming depth of mackerel may be modified to 60m-120m, and the swimming depth of sardines may be modified to 20-80m.

[0168] Furthermore, instead of, or in conjunction with, the process of correcting the range of overlapping know-how information between fish species as described above, the initial value of the confidence level of overlapping know-how information between fish species may be corrected. For example, in the example in Figure 15, the swimming depths of mackerel and sardines have a wide range, so it is conceivable that they overlap not only with the swimming depths of these fish species but also with those of tuna and many other fish species. In this way, the confidence level of know-how information that overlaps with a predetermined number of fish species may be initially set to a value less than 100% (for example, 60%).

[0169] As described above, by correcting the scope of overlapping know-how information or by correcting the reliability of this know-how information, it is conceivable that the know-how model can be made more appropriate and more accurate fish species identification results can be obtained.

[0170] <Other examples of changes> In the above embodiment, a process was performed to automatically correct the know-how model based on feedback information, but this process may be omitted. For example, if the frequency or number of corrections to know-how information that caused the user to correct the fish species exceeds a predetermined threshold, a process may be performed to prompt the user to reset this know-how information. In this case, the control unit 201 of the server 20 sends a notification to the underwater detection device 10 to reset this know-how information, and the control unit 201 of the underwater detection device 10 displays a screen for resetting this know-how information on the display unit 102 based on the receipt of this notification. In this case, the control unit 201 may further display on the display unit 102 the reason why resetting is necessary, for example, that the fish school discrimination result based on this know-how information has been frequently corrected by the user.

[0171] Furthermore, in the above embodiment, an example of the type of know-how information set by the user is shown in the know-how input area 502 of the input screen 500 in Figure 6, but the types of know-how information set by the user are not limited to this. For example, other types of know-how information, such as salinity and current speed, may be included in the know-how information that can be set by the user.

[0172] Furthermore, while the input screen 500 in Figure 6 allowed the user to set the reliability level for each piece of know-how information, it is not necessary for the reliability level of each piece of know-how information to be configurable. In this case, for example, all the know-how information entered by the user in the know-how input area 502 of the input screen 500 would be set to high reliability. Therefore, the input screen 500 may also display a message encouraging the user to enter only the information they are confident in.

[0173] Furthermore, in the above embodiment, as shown in Figure 5, the machine learning model 302 and the know-how model 312 were configured separately, but the know-how model 312 may be incorporated into the machine learning model 302.

[0174] Furthermore, in the above embodiment, the storage of the machine learning model, know-how information, and know-how model, as well as the fish species discrimination process using these, were performed on the server 20 side. However, these storage and discrimination processes may also be performed on the underwater detection device 10 side.

[0175] In this case, the server 20 continuously transmits the latest machine learning model to the underwater detection device 10, and the control unit 101 of the underwater detection device 10 stores the latest machine learning model received from the server 20 in its memory. The control unit 101 of the underwater detection device 10 also stores know-how information for each fish species set by the user in its memory, and further stores a know-how model generated using this know-how information in its memory. Then, based on the echo data acquired by the echo data acquisition unit 110, the control unit 101 performs fish species discrimination using the machine learning model and the know-how model, similar to the control unit 201 of the server 20. The control unit 101 also stores information similar to that in Figure 4(a) in its memory, and modifies the know-how model, similar to the above, based on the feedback information input via the echo image P1 in Figure 11.

[0176] Furthermore, the storage of machine learning models, know-how information, and know-how models, as well as the processing of fish species identification using these, may be shared between the underwater detection device 10 and the server 20.

[0177] Furthermore, in the above embodiment, user feedback information was input via the screen shown in Figure 11, but the method of inputting feedback information is not limited to this.

[0178] Furthermore, in the above embodiment, the machine learning model was generated by machine learning using training data created by experts, but the training data used for machine learning of the machine learning model is not limited to this. For example, each user may input a school of fish and its species from an echo image based on their own fishing results, and the echo data of this school of fish and the species of that school of fish may be used as training data for the machine learning model. In this case, the server 20 stores the school of fish, its echo data, and the species of that school of fish input by each user as training data. The server 20 may, for example, aggregate this training data by region and generate a machine learning model for each region. In this case, the control unit 201 of the server 20 should perform machine learning on the machine learning model for each region using the training data aggregated by region.

[0179] Furthermore, in the above embodiment, the underwater detection device 10 was a fish finder, but the underwater detection device 10 may be a device other than a fish finder, such as a sonar.

[0180] In addition, embodiments of the present invention can be modified in various ways as appropriate within the scope of the claims. [Explanation of Symbols]

[0181] 1. Fish Species Identification System 10 Underwater detection equipment 20 servers 201 Control Unit 202 Storage section 302 Machine Learning Models 312, 322 Know-how Models

Claims

1. An echo data acquisition unit that acquires underwater echo data, Memory unit and, It comprises a control unit and, The aforementioned storage unit is A machine learning model that has been trained using training data combining echo data of the range of a fish school and the fish species of that fish school, and which outputs a predicted probability for each fish species of a fish school based on the echo data of the range of the fish school acquired by the echo data acquisition unit, Know-how information regarding the characteristics of fish schools set by the user for each of the aforementioned fish species, The system stores a know-how model that modifies the predicted probability for each fish species by calculation based on the characteristics of the fish school defined in the know-how information for each fish species and the characteristics of the fish school within the range of the fish school from which the predicted probability was obtained. The control unit, Based on the corrected results obtained by modifying the predicted probabilities for each fish species acquired by the machine learning model using the know-how model, the fish species of the fish school from which the predicted probabilities were obtained by the machine learning model are identified. A fish species identification system characterized by the following features.

2. In the fish species discrimination system described in claim 1, The aforementioned know-how information includes information regarding the characteristics of the aforementioned fish species in water, A fish species identification system characterized by the following features.

3. In the fish species discrimination system described in claim 1, The aforementioned know-how information includes information on sea conditions suitable for the aforementioned fish species. A fish species identification system characterized by the following features.

4. Claim 1, in a fish species discrimination system, The aforementioned know-how information includes information regarding at least one of the timing and location of capture of the fish of the aforementioned fish species. A fish species identification system characterized by the following features.

5. In the fish species discrimination system described in claim 1, The control unit modifies the degree of modification of the prediction probability in the know-how model based on feedback information indicating the user's modifications to the discrimination result. A fish species identification system characterized by the following features.

6. In the fish species discrimination system described in claim 1, The control unit modifies the degree of correction of the prediction probability in the know-how model based on the learning progress of the machine learning. A fish species identification system characterized by the following features.

7. In the fish species discrimination system described in claim 1, An underwater detection device for detecting schools of fish underwater, The system comprises a server capable of communicating with the aforementioned underwater detection device, The echo data acquisition unit is located in the underwater detection device. The storage unit and the control unit are located in the server. A fish species identification system characterized by the following features.

8. In the fish species discrimination system described in claim 7, The aforementioned underwater detection device is Display unit and Input section, The system includes a control unit that displays the discrimination result on the display unit, accepts a correction of the discrimination result via the input unit, and transmits the correction as feedback information to the server. A fish species identification system characterized by the following features.

9. A server capable of communicating with an underwater detection device that detects schools of fish underwater, Memory unit and, It comprises a control unit and, The aforementioned storage unit is A machine learning model that has been trained using training data combining echo data of the range of a fish school and the fish species of that school, and which outputs a predicted probability for each fish species of a fish school based on the echo data of the range of the fish school from the echo data received from the underwater detection device, The user has set the following know-how information for each fish species regarding the characteristics of the fish school, The system stores a know-how model that modifies the predicted probability for each fish species obtained by the machine learning model based on the characteristics of the fish school defined in the know-how information for each fish species and the characteristics of the fish school within the range of the fish school from which the predicted probability was obtained. The control unit, Based on the corrected results obtained by modifying the predicted probabilities for each fish species acquired by the machine learning model using the know-how model, the fish species of the fish school from which the predicted probabilities were obtained by the machine learning model are identified. A server characterized by the following features.

10. We acquire underwater echo data, Using a machine learning model trained on training data combining echo data of the fish school's range and the fish species within that school, the machine learning model calculates the predicted probability for each fish species in the fish school based on the echo data of the fish school's range. User-submitted know-how information regarding the characteristics of fish schools is stored for each of the aforementioned fish species. The predicted probability for each fish species is modified by a calculation based on the characteristics of the fish school defined in the know-how information for each fish species and the characteristics of the fish school within the range of the fish school from which the predicted probability was obtained. Based on the corrected predicted probabilities for each fish species, the machine learning model identifies the fish species of the fish school from which the predicted probabilities were obtained. A method for identifying fish species characterized by the following features.

11. A function to calculate the predicted probability for each fish species in a fish school based on the echo data of the fish school's range acquired from underwater, using a machine learning model trained with training data combining echo data of the fish school's range and the fish species in the fish school. A function to store user-submitted know-how information regarding the characteristics of fish schools for each of the aforementioned fish species, A function to modify the predicted probability for each fish species by calculation based on the characteristics of the fish school defined in the know-how information for each fish species and the characteristics of the fish school within the range of the fish school from which the predicted probability was obtained. A program that causes a computer to perform the following functions: determine the fish species of the fish school from which the machine learning model has obtained the predicted probability, based on the corrected predicted probability for each fish species.

Images