Fish species learning device, fish species learning method, and program

Abstract

To provide a fish species learning device, a fish species learning method, and a program that can more accurately perform machine learning for fish species discrimination.SOLUTION: A server 20 (fish species learning device) comprises: a storage unit 202 that stores annotation data for every unit echo image; and a control unit 201. The control unit 201 extracts, from annotation data of a first unit echo image and a second unit echo image, which are temporally continuous, first annotation data and second annotation data of a fish school present between the first unit echo image and second unit echo image, and integrates the extracted first annotation data and second annotation data to generate annotation data of the entire fish school present between the first unit echo image and second unit echo image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a fish species learning device, a fish species learning system, a fish species learning method, and a program that perform machine learning for fish species discrimination using teacher data (annotation data).

Background Art

[0002] Conventionally, a fish school detection device for detecting a fish school in water has been known. In this type of fish school detection device, ultrasonic waves are transmitted into the water, and the reflected waves are received. Echo data corresponding to the intensity of the received reflected waves is generated, and an echo image is displayed based on the generated echo data. A user can confirm a fish school from the echo image and smoothly proceed with the capture of the fish school.

[0003] In this case, it is preferable that the fish species of the fish school on the echo image is further discriminated and displayed. Thereby, the user can efficiently capture the fish of the fish species desired by the user.

[0004] Such discrimination of fish species can be performed, for example, using a machine learning model (machine learning algorithm). Learning is performed on the machine learning model using a large number of teacher data (annotation data). Each annotation data includes echo data of a fish school, the range (water depth, time) of the fish school, and the fish species of the fish school. The following Patent Document 1 describes a configuration when such annotation data is generated by a user.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Generally, annotation data is generated for each unit echo image. That is, a series of echo images for a predetermined time is divided into multiple segments corresponding to one frame, generating multiple unit echo images that will be used to generate annotation data. For each unit echo image, an expert operator specifies the range of a school of fish and then assigns a fish species label to the specified school. This generates annotation data that associates the range of a school of fish with the fish species within that range and the image data (echo data) contained within that range.

[0007] In this generation method, the entire school of fish is not necessarily included in a single unit echo image. That is, a single school of fish may span two temporally consecutive unit echo images. In such cases, the two fragmented portions of the school of fish will be included in the two separate echo images. Since the above generation method processes data on a unit echo basis, annotation data is generated individually for each fragmented portion of the school of fish.

[0008] However, in the above machine learning method, features arising from the entire fish school, such as the trailing and distribution of echoes specific to each fish species, can affect the accuracy of the machine learning. Therefore, as described above, if annotation data generated for only a portion of the fish school is used in machine learning, the accuracy of the machine learning may decrease.

[0009] In view of these challenges, the present invention aims to provide a fish species learning device, a fish species learning method, and a program that can perform machine learning for fish species discrimination with greater accuracy. [Means for solving the problem]

[0010] A first aspect of the present invention relates to a fish species learning device. The fish species learning device according to this aspect is: A series of echo images were divided into multiple sections. It comprises a storage unit that stores annotation data for each unit echo image, and a control unit. The control unit is time-sequential These are two unit echo images.From the annotation data of the first unit echo image and the second unit echo image, the first annotation data and the second annotation data of the fish school spanning between the first unit echo image and the second unit echo image are extracted, and the extracted first annotation data and the second annotation data are integrated to generate annotation data for the entire fish school.

[0011] According to the fish species learning device of this embodiment, the first annotation data and the second annotation data of a fish school spanning between the first unit echo image and the second unit echo image are integrated to generate annotation data for the entire fish school. Therefore, since the annotation data for the entire fish school can be used for machine learning, machine learning for fish species discrimination can be performed with greater accuracy.

[0012] In the fish species learning device according to this embodiment, the control unit may be configured to identify a first fish group that extends to the vicinity of the first boundary on the second unit echo image side on the first unit echo image, and a second fish group that extends to the vicinity of the second boundary on the first unit echo image side on the second unit echo image, and to extract the annotation data of the first fish group and the second fish group as the first annotation data and the second annotation data, respectively, when the depth range of the first fish group and the depth range of the second fish group substantially coincide.

[0013] This configuration allows for the smooth identification of the first and second fish schools, which have a high probability of constituting a single fish school, in the first and second unit echo images, respectively. Therefore, by integrating the annotation data of these fish schools, accurate annotation data for the entire fish school can be generated.

[0014] In this configuration, the control unit may be configured to determine whether the depth range of the first fish school and the depth range of the second fish school substantially coincide, based on the degree of agreement between the depth range of the first fish school near the first boundary and the depth range of the second fish school near the second boundary.

[0015] With this configuration, it is determined whether the depth ranges of the first and second fish schools approximately coincide near the first and second boundaries, which are the boundaries between the first and second unit echo images. Therefore, it is possible to accurately determine whether the first and second fish schools are continuous with each other. Thus, annotation data for an entire fish school can be generated with high accuracy from the annotation data of the first and second fish schools.

[0016] Furthermore, in this configuration, the control unit may be configured to extract a group of fish that is at least partially included within a predetermined time range from the first boundary as the first group of fish, and to extract a group of fish that is at least partially included within a predetermined time range from the second boundary as the second group of fish.

[0017] With this configuration, even when the distribution of fish is sparse near the boundary, the first and second fish groups that constitute the fish school spanning the first and second unit echo images can be identified without any omissions in the first and second unit echo images. Therefore, annotation data for the entire fish school can be generated appropriately.

[0018] In the fish species learning device according to this embodiment, the control unit may be configured to set the fish species of the fish school annotation data based on predetermined setting conditions if the fish species included in the first annotation data and the fish species included in the second annotation data are different.

[0019] In this case, the setting conditions may include, for example, setting the fish species to the more recent annotation data among the first annotation data and the second annotation data.

[0020] With these configurations, if the fish species included in the first and second annotation data sets to be integrated differ, the most reliable fish species can be set in the integrated annotation data. Therefore, the accuracy of machine learning can be improved.

[0021] The second aspect of the present invention relates to a fish species learning method. The fish species learning method according to this aspect is Annotation data is stored for each unit echo image, which is divided into multiple sections from a series of echo images. successive in time These are two unit echo images. From the annotation data of the first unit echo image and the second unit echo image, extract the first annotation data and the second annotation data of the fish school straddling between the first unit echo image and the second unit echo image respectively, and integrate the extracted first annotation data and the second annotation data to generate the annotation data of the entire fish school.

[0022] The third aspect of the present invention relates to a program for causing a computer to execute a predetermined function. The program according to this aspect is A function that stores annotation data for each unit echo image, which is divided into multiple sections from a series of echo images, successive in time These are two unit echo images. From the annotation data of the first unit echo image and the second unit echo image, a function of extracting the first annotation data and the second annotation data of the fish school straddling between the first unit echo image and the second unit echo image respectively, and a function of integrating the extracted first annotation data and the second annotation data to generate the annotation data of the entire fish school.

[0023] According to the above second and third aspects, the same effects as the first aspect are achieved.

Effect of the Invention

[0024] As described above, according to the present invention, it is possible to provide a fish species learning device, a fish species learning method, and a program capable of performing machine learning for fish species discrimination with higher accuracy.

[0025] The effects or significance of the present invention will become clearer from the description of the embodiments shown below. However, the embodiments shown below are merely examples when implementing the present invention, and the present invention is not limited to those described in the following embodiments.

Brief Description of the Drawings

[0026] [Figure 1] Figure 1 is a diagram showing the configuration of a fish species identification system according to an embodiment. [Figure 2] Figure 2 is a block diagram showing the configuration of a fish species identification system according to an embodiment. [Figure 3] Figure 3 is a schematic diagram illustrating a fish species discrimination process using a neural network according to an embodiment. [Figure 4] Figure 4 is a schematic diagram showing an example of displaying an echo image including the fish species identification result according to the embodiment. [Figure 5] Figure 5 is a schematic diagram illustrating a method for generating annotation data in a terminal device according to an embodiment. [Figure 6] Figure 6 shows the storage configuration of annotation data in the server's storage unit according to an embodiment. [Figure 7] Figure 7 is a flowchart showing the annotation data integration process performed by the server's control unit according to the embodiment. [Figure 8] Figure 8 is a flowchart illustrating the process of extracting annotation data for a school of fish spanning two unit echo images according to an embodiment. [Figure 9] Figures 9(a) and 9(b) schematically show the state of a school of fish near the boundary between the first unit echo image and the second unit echo image according to the embodiment. [Figure 10] Figures 10(a) and 10(b) illustrate methods for integrating annotation data for the first and second fish schools according to the embodiment, respectively. [Figure 11] Figure 11(a) schematically shows the state of the fish school near the boundary between the first unit echo image and the second unit echo image in Modification Example 1. Figure 11(b) shows the method for integrating the annotation data of the first and second fish schools in Modification Example 1. [Figure 12] Figure 12 is a flowchart showing the fish species setting process performed in the annotation data integration process related to Change Example 2. [Modes for carrying out the invention]

[0027] Embodiments of the present invention will be described below with reference to the drawings.

[0028] In the following embodiments, the server 20 corresponds to the "fish species learning device" described in the claims. However, the "fish species learning device" according to the present invention is not necessarily limited to the server 20, and other devices such as the terminal device 50 may perform the functions of the "fish species learning device" according to the present invention.

[0029] Figure 1 shows the configuration of the fish species identification system 1.

[0030] The fish species identification system 1 comprises an underwater detection device 10 and a server 20. The underwater detection device 10 is a fish finder installed on the ship 2. The underwater detection device 10 can communicate with the server 20 via an external communication network 30 (for example, the internet) and a base station 40. The underwater detection device 10 and the server 20 each hold address information for communicating with each other. This address information is set in the underwater detection device 10 and the server 20 during initial setup.

[0031] The underwater detection device 10 comprises a transducer 11 and a control unit 12. The transducer 11 is installed on the bottom of the ship 2, and the control unit 12 is installed in the ship's wheelhouse or elsewhere. The transducer 11 and the control unit 12 are connected by a signal cable (not shown). The transducer 11 is equipped with an ultrasonic transducer for transmitting and receiving waves. In response to control from the control unit 12, the transducer 11 transmits ultrasonic waves 3 (transmitted waves) toward the seabed 4 using the ultrasonic transducer and receives the reflected waves. The transducer 11 transmits a received signal based on the received reflected waves to the control unit 12.

[0032] The control unit 12 processes the received signal and generates echo data indicating the echo intensity at each depth. The control unit 12 arranges the echo intensity at each depth based on the echo data in chronological order to generate an echo image for one screen. The control unit 12 displays the generated echo image on the display unit. The control unit 12 updates the echo image for each ultrasonic wave transmission and reception. By referring to the echo image, the user can determine the presence and location of the school of fish 5.

[0033] Furthermore, the control unit 12 transmits the generated echo data to the server 20 as needed. The server 20 stores the received echo data and generates echo images similar to those of the control unit 12. The server 20 uses a machine learning model (machine learning algorithm) to calculate the predicted probability (the probability that the fish is of that species) for each fish species included in the school of fish in the echo image.

[0034] Server 20 obtains the species identification result for the school of fish based on the predicted probability for each fish species calculated by the machine learning model. Server 20 then transmits the obtained species identification result, along with the range (depth, time) of the school of fish to be identified, to the underwater detection device 10 that receives the echo data.

[0035] The underwater detection device 10 overlays the fish species identification results onto the corresponding area of ​​the echo image based on the received identification results and the range (depth, time) of the fish school. This allows the user to confirm the fish species of each fish school on the echo image, enabling them to smoothly proceed with catching the desired fish.

[0036] Furthermore, the server 20 acquires annotation data (training data) for training the machine learning model from multiple terminal devices 50 via the external communication network 30. In other words, the server 20 distributes the echo data received from the underwater detection device 10 to one of the multiple terminal devices 50. The terminal devices 50 are used to generate annotation data from the received echo data.

[0037] In other words, the terminal device 50 is held by an operator, such as a specialist, who generates annotation data. The operator displays echo images based on the received echo data on the terminal device 50 and sets the fish species for each school of fish included in these echo images. The set fish species is associated with the range (depth, time) of the school of fish and the echo data of that range. The annotation data for the school of fish is then composed of the associated fish species, the range of the school of fish, and the echo data. The annotation data is transmitted to the server 20 along with the identification information of the echo image from which the annotation data was generated.

[0038] Server 20 uses the received annotation data to train a machine learning model. This improves the accuracy of the machine learning model in identifying fish species.

[0039] In this example, annotation data was generated from the echo data provided by server 20 to terminal device 50, and the generated annotation data was returned to server 20. However, the method of providing annotation data to server 20 is not limited to this. For example, echo data may be provided to terminal device 50 from a device other than server 20, annotation data may be generated, and the generated annotation data may be provided to server 20.

[0040] Furthermore, although only one underwater detection device 10 is shown in Figure 1, in reality, numerous underwater detection devices 10 can communicate with the server 20 via the external communication network 30 and the nearest base station. In addition to the underwater detection device 10 installed on the ship 2 as shown in Figure 1, the underwater detection devices 10 that communicate with the server 20 may include several types of underwater detection devices used in different fishing methods, such as underwater detection devices installed on fixed nets.

[0041] Figure 2 is a block diagram showing the configuration of the fish species identification system 1.

[0042] The underwater detection device 10 comprises a control unit 101, a display unit 102, an input unit 103, a transmitter / receiver unit 104, a signal processing unit 105, a communication unit 106, and a position detection unit 107.

[0043] The control unit 101 consists of a microcomputer and memory, etc. The control unit 101 controls each part of the underwater detection device 10 according to the program stored in the memory.

[0044] The display unit 102 includes a monitor and displays a predetermined image under control from the control unit 101. The input unit 103 includes a trackball for moving a cursor on the image displayed on the display unit 102, operation keys, etc., and outputs signals to the control unit 101 in response to user operations. The display unit 102 and the input unit 103 may be integrated using a liquid crystal touch panel or the like.

[0045] The transmitter / receiver unit 104 includes a transducer 11 as shown in Figure 1, a transmitting circuit for supplying a transmission signal to the transducer 11, and a receiving circuit for processing the received signal output from the transducer 11 and outputting it to the signal processing unit 105. The transmitting circuit and the receiving circuit are included in the control unit 12 shown in Figure 1.

[0046] The transmitter / receiver unit 104 transmits a transmission wave (ultrasound) of a predetermined frequency according to the control unit 101. The transmitter / receiver unit 104 receives the reflected wave of the transmitted wave and outputs a received signal. The receiving circuit extracts the received signal of the frequency of the transmitted wave and outputs it to the signal processing unit 105.

[0047] The signal processing unit 105 generates echo data indicating the intensity of the reflected wave according to the depth from the received signal input from the transmitter / receiver unit 104, and outputs the generated echo data to the control unit 101. The elapsed time from the time the transmitted wave was sent corresponds to the depth. Here, the intensity of the reflected wave attenuates as the depth increases. Therefore, in order to handle the echo data quantitatively regardless of the difference in depth, the signal processing unit 105 corrects the intensity of the reflected wave that attenuates according to the elapsed time, and outputs the echo data with the corrected intensity to the control unit 101.

[0048] The control unit 101 generates an echo image based on the received echo data and displays it on the display unit 102. The control unit 101 generates a single column of images in the depth direction from the echo data, representing the echo intensity at each depth using a color scale. The control unit 101 integrates the images in each column from the present time up to a predetermined time ago in the time direction to generate a single screen echo image.

[0049] The communication unit 106 is a communication module capable of wireless communication with the base station 40. The position detection unit 107 is equipped with GPS and detects the position of the underwater detection device 10. The position detection unit 107 outputs the detected position information to the control unit 101.

[0050] As explained with reference to Figure 1, the control unit 101 transmits echo data to the server 20 via the communication unit 106 as needed. The control unit 101 also receives the fish species identification result from the server 20 via the communication unit 106. Furthermore, the control unit 101 transmits the location information detected by the location detection unit 107 to the server 20.

[0051] As shown in Figure 2, in addition to the underwater detection device 10, numerous other underwater detection devices 10a, 10b, ... can communicate with the server 20 via the external communication network 30 and the nearest base stations 40a, 40b, .... As described above, the underwater detection devices 10 that communicate with the server 20 include not only those installed on the ship 2 as shown in Figure 1, but also several types of underwater detection devices used for different fishing methods, such as underwater detection devices installed on fixed nets. The basic configuration of the other underwater detection devices is the same as that of the underwater detection device 10 in Figure 2.

[0052] The server 20 comprises a control unit 201, a storage unit 202, and a communication unit 203. The control unit 201 is composed of a CPU, etc. The storage unit 202 is composed of ROM, RAM, hard disk, etc. The storage unit 202 stores programs for fish species identification and programs for machine learning. The control unit 201 controls each part based on the programs stored in the storage unit 202. The communication unit 203 communicates with the underwater detection device 10 via the external communication network 30 and base station 40 under control from the control unit 201. The server 20 also communicates with multiple terminal devices 50 via the external communication network 30.

[0053] The terminal device 50 is a device capable of inputting and outputting information, such as a personal computer or a tablet computer. The terminal device 50 comprises a control unit 501, a storage unit 502, a display unit 503, an input unit 504, and a communication unit 505.

[0054] The control unit 501 is composed of a CPU and the like. The storage unit 502 is composed of ROM, RAM, hard disk, and the like. The storage unit 502 stores a program for generating annotation data. The control unit 501 controls each part using the program stored in the storage unit 502. The display unit 503 is composed of an LCD monitor and the like, and displays a predetermined image under control from the control unit 501. The input unit 504 is equipped with input means such as a mouse and keyboard. The communication unit 505 communicates with the server 20 under control from the control unit 501.

[0055] Figure 3 schematically illustrates the fish species discrimination process using a neural network.

[0056] In this embodiment, machine learning using a neural network is applied. For example, a neural network using deep learning, which combines neurons in multiple stages, is applied. However, the machine learning applied is not limited to this, and other machine learning methods such as support vector machines and decision trees may also be applied.

[0057] The control unit 201 of server 20 extracts the range (depth, time) of a school of fish from the echo data for one screen to be processed. On the echo image, areas where the echo intensity is above a predetermined threshold and where there is a connection between the echo intensities are extracted as a school of fish, and further, the rectangular area formed by the maximum time width and maximum depth width of this school of fish is extracted as the range of the school of fish. The method for extracting schools of fish may be incorporated by reference from the description in International Publication No. 2019 / 003759, which was previously filed by the applicant.

[0058] The control unit 201 applies the echo data of the extracted fish school range to the input 301a of the machine learning model (machine learning algorithm using a neural network) 301 shown in Figure 3.

[0059] The output 301b of the machine learning model 301 is assigned items for fish species such as sardines, horse mackerel, and mackerel. When echo data of the range of the fish school is applied to the input 301a of the machine learning model 301, the probability (prediction probability) that the fish species in the fish school is the species of the item is output from each item in the output 301b of the machine learning model 301. In the example in Figure 3, the sardine item outputs a prediction probability of 85%, the mackerel item outputs a prediction probability of 70%, and the horse mackerel item outputs a prediction probability of 10%.

[0060] The predicted probability for each item is compared with output condition 302. Output condition 302 may include, for example, a condition that outputs the fish species of the item whose predicted probability is above a predetermined lower limit and is the highest ranked (highest) as the classification result 303. The lower limit is set to prevent fish species with low accuracy from being output as classification results. In the example in Figure 3, sardines, which have a predicted probability of 85%, are output as the fish species classification result 303.

[0061] Machine learning for the machine learning model 301 is performed by sequentially applying a series of annotation data (training data) to the input 301a and output 301b of the machine learning model 301. Specifically, the input 301a of the machine learning model 301 is the echo data of a school of fish contained in one annotation data, and the output 301b of the machine learning model 301 is set to 100% for the item corresponding to the fish species contained in this annotation data, and to 0% for the other items, and machine learning is performed.

[0062] In addition to the echo data of the fish school, the input 301a of the machine learning model 301 may also contain other information that can be used for fish species identification, such as the location where the echo data was obtained and oceanographic data for that location. The annotation data may further include this other information.

[0063] Figure 4 schematically shows an example of the display of an echo image P1 including the fish species identification result. For convenience, in Figure 4, depth lines are added only to the areas with high echo intensity.

[0064] When the control unit 201 of the underwater detection device 10 receives the discrimination result and the range (depth, time) of the fish school from the server 20, it displays a frame-shaped marker M0 indicating the range of the fish school in an area on the echo image P1 corresponding to the depth width and time width corresponding to the received range of the fish school. Furthermore, the control unit 201 displays a label L0 indicating the discrimination result of the received fish species around this marker M0.

[0065] In the example shown in Figure 4, based on the discrimination results and the range (depth, time) of the fish school received from server 20, markers M0 are displayed for fish schools F1 to F8, and labels L0 indicating the fish species discrimination results are displayed around these markers M0. The current date and time are displayed near the upper left corner of the echo image P1.

[0066] In the example in Figure 4, the machine learning model 301 and output conditions 302 in Figure 3 did not output a fish species classification result for fish school F9, and therefore no markers or labels are displayed for fish school F9. This can occur, for example, if the predicted probability of fish school F9 by the machine learning model 301 does not satisfy the output conditions 302. In such a case, the classification result for this fish school is not transmitted from the server 20 to the underwater detection device 10, and as shown for fish school F9 in Figure 4, the fish species classification result is not displayed.

[0067] Figure 5 is a schematic diagram illustrating the method of generating annotation data performed in the terminal device 50.

[0068] Annotation data is generated in the terminal device 50 by displaying an echo image based on the echo data to be processed on the display unit 503.

[0069] An expert operator specifies the range of a school of fish on the displayed echo image. For example, the range of a school of fish is specified by a rectangle. The operator specifies the range of the school of fish by operating the input unit 504 to specify the diagonal vertices of the rectangle. Furthermore, the operator operates the input unit 504 to set the fish species for each school of fish. For example, depending on the specified range of the school of fish, a list of candidate fish species is displayed on the display unit 503, and the operator selects the target fish species from this list. This sets the fish species for that range of school of fish.

[0070] Here, annotation data is generated for each unit echo image, as shown in Figure 5. That is, a series of echo images for a predetermined time is divided into multiple segments with a time width corresponding to one frame, and multiple unit echo images are generated to be used for annotation data generation. Of the unit echo image boundaries B1 and B2 in the time direction, boundary B1 on the more recent side coincides with boundary B2 on the older side of the next unit echo image.

[0071] Expert operators specify the range of a fish school for each unit echo image and then assign a fish species to the specified school. This generates annotation data that associates the range of the fish school with the fish species within that range and the image data (echo data) included within that range.

[0072] In the example in Figure 5, annotation processing An on the unit echo image Pn specifies the ranges of six fish schools (ranges indicated by dashed rectangles), and assigns a fish species to each range. Similarly, annotation processing An+1 on the unit echo image Pn+1 specifies the ranges of five fish schools (ranges indicated by dashed rectangles), and assigns a fish species to each range. Therefore, annotation data is generated from the unit echo image Pn for each of the six fish schools, associating the range of the fish school, the fish species within that range, and the image data (echo data) contained within that range. Furthermore, annotation data is generated from the unit echo image Pn+1 for each of the five fish schools, associating the range of the fish school, the fish species within that range, and the image data (echo data) contained within that range.

[0073] The annotation data generated in this way is transmitted from the terminal device 50 to the server 20 and stored in the server 20's storage unit 202.

[0074] Figure 6 shows the storage configuration of annotation data in the storage unit 202 of the server 20.

[0075] The memory unit 202 stores unit echo data and annotation data, associated with the echo ID and image ID.

[0076] The Echo ID is identification information used to identify a series of echo images (echo data) for a predetermined time period as described above. The Echo ID is uniquely set by the server 20. The Image ID is identification information used to identify a unit echo image. The Image ID is, for example, a number assigned to each unit echo image in chronological order from oldest to newest. The unit echo data is one frame of echo data corresponding to each unit echo image. The annotation data is annotation data generated from each unit echo image.

[0077] When the control unit 201 of the server 20 receives new echo data for a predetermined time period that is to be annotated from the underwater detection device 10 or the like, it assigns an echo ID to this echo data. Furthermore, the control unit 201 divides this echo data into time widths of one frame each, starting from the oldest part, and generates unit echo data corresponding to each unit echo image. Then, the control unit 201 assigns an image ID to each unit echo data and stores the unit echo data in the storage unit 202, associating it with the image ID. This constitutes the echo ID, image ID, and unit echo data parts of the data structure in Figure 6.

[0078] Subsequently, the control unit 201 transmits each unit echo data, along with the echo ID and image ID, to the terminal device 50. The control unit 501 of the terminal device 50 stores this received data in the storage unit 502 and provides it for annotation processing by the operator. When the operator performs the annotation processing, the control unit 501 displays the unit echo image based on the unit echo data to be processed on the display unit 503. The operator performs the annotation processing on the unit echo image to be processed according to the process shown in Figure 5. This generates annotation data for the unit echo image.

[0079] Once annotation processing is complete for all unit echo data received from server 20, the control unit 501 associates the annotation data generated for each unit echo image with the image ID of that unit echo image and sends it to server 20 along with the echo ID. The control unit 201 of server 20 associates the received annotation data with the image IDs in Figure 6 and stores it in the storage unit 202. As shown in Figure 5, each image ID is associated with annotation data corresponding to the number of fish in the unit echo image. With the annotation data stored in the storage unit 202 in this way, the data structure in Figure 6 is completed.

[0080] Incidentally, the unit echo image shown in Figure 5 does not necessarily contain the entire school of fish; it is possible for a single school of fish to span across two temporally consecutive unit echo images.

[0081] For example, in the example in Figure 5, the school of sardines in the upper right of unit echo image Pn is thought to be connected to the school of sardines in the upper left of the next unit echo image Pn+1. However, in the above generation method, since a school of fish is specified for each unit echo image, the school of sardines that spans unit echo image Pn and unit echo image Pn+1 is specified as being divided into the parts of unit echo image Pn and unit echo image Pn+1. Then, annotation data is generated individually for each divided part of the school of fish.

[0082] However, in the machine learning described above, features arising from the entire school of fish, such as the trailing and distribution of echoes specific to each fish species, can affect the accuracy of the machine learning. Therefore, as mentioned above, if annotation data generated for only a portion of the school of fish is used in machine learning, the accuracy of the machine learning will decrease.

[0083] To resolve these issues, in this embodiment, annotation data for fish schools spanning between two temporally consecutive unit echo images is extracted from the annotation data of these unit echo images, and the two extracted annotation data sets are integrated to generate annotation data for the fish school. That is, in the example shown in Figure 5, the annotation data for the sardine school in the upper right of unit echo image Pn and the annotation data for the sardine school in the upper left of unit echo image Pn+1 are integrated to generate annotation data for the entire sardine school.

[0084] Figure 7 is a flowchart showing the annotation data integration process performed by the control unit 201 of the server 20.

[0085] The control unit 201 references annotation data for two unit echo images (unit echo data) that are consecutive in time from among the unit echo images (unit echo data) with the same echo ID (S11). Next, the control unit 201 extracts annotation data for the fish school that spans these two unit echo images from the referenced annotation data (S12). Then, the control unit 201 integrates the extracted annotation data to generate annotation data for the entire fish school that spans the two unit echo images, and stores the generated annotation data in the storage unit 202 (S13). The control unit 201 performs the process shown in Figure 7 for all unit echo images (unit echo data) with the same echo ID.

[0086] The machine learning model 301 in Figure 3 uses the annotation data in Figure 6 generated by the terminal device 50 and the annotation data generated by the processing in Figure 7. In this case, the two annotation data integrated by the processing in Figure 7 may be excluded from the annotation data used for machine learning. For example, the two annotation data integrated by the processing in Figure 7 may be deleted from the data structure in Figure 6 constructed in the storage unit 202.

[0087] Figure 8 is a flowchart showing an example of the process in step S12 of Figure 7.

[0088] For convenience, Figure 8 includes steps S11 and S13 of Figure 7. Steps S101 to S105 in Figure 8 correspond to step S12 in Figure 7.

[0089] In step S11, the control unit 201 refers to the annotation data of two temporally consecutive unit echo images and determines the extent of the fish school in each annotation data. Next, the control unit 201 determines whether or not a fish school exists near the right boundary, i.e., near boundary B1 in Figure 5, in the first unit echo image, which is the older of the two unit echo images (S101). If no fish school exists near the right boundary B1 of the first unit echo (S101: NO), the control unit 201 terminates the process shown in Figure 8. If a fish school exists near the right boundary B1 of the first unit echo (S101: YES), the control unit 201 identifies this fish school as the first fish school to be integrated (S102).

[0090] Next, the control unit 201 determines whether a school of fish exists near the left boundary, i.e., near boundary B2 in Figure 5, in the second unit echo image, which is the more recent of the two unit echo images (S103). If no school of fish exists near the left boundary B2 of the second unit echo (S103: NO), the control unit 201 terminates the process shown in Figure 8. If a school of fish exists near the left boundary B2 of the second unit echo (S103: YES), the control unit 201 identifies this school of fish as the second school of fish to be integrated (S104).

[0091] Next, the control unit 201 determines whether the depth range of the first fish school and the depth range of the second fish school substantially coincide (S105). More specifically, the control unit 201 determines whether the agreement rate between the depth range of the first fish school and the depth range of the second fish school is greater than or equal to a predetermined threshold Th1. If the determination in step S105 is NO, the control unit 201 terminates the process shown in Figure 8 without performing the process in step S13.

[0092] If the determination in step S105 is YES, the control unit 201 determines that the first and second fish groups constitute fish groups spanning the first unit echo image and the second unit echo image, integrates the annotation data of the first and second fish groups, and generates annotation data for the entire fish group (S13). With this, the control unit 201 finishes processing the annotation data of these two unit echo images.

[0093] The control unit 201 performs the processing shown in Figure 8 on the annotation data of the next two consecutive unit echo images. In this way, the control unit 201 performs the processing shown in Figure 8 on the annotation data of all unit echo images that have the same echo ID.

[0094] Figures 9(a) and 9(b) schematically show the state of a school of fish near the boundary between the first unit echo image and the second unit echo image.

[0095] Figure 9(a) shows the upper right portion of the unit echo image Pn (first unit echo image) and the upper left portion of the unit echo image Pn+1 (second unit echo image) from Figure 5. Fa and Fb represent schools of sardines, and Ra and Rb represent the range (time range, depth range) of the fish schools included in the annotation data. As described above, the fish school ranges Ra and Rb are set to rectangles.

[0096] In step S101 of Figure 8, it is determined whether a school of fish exists within a predetermined time range Ta from the boundary B1 of the first unit echo image Pn, and in step S103, it is determined whether a school of fish exists within a predetermined time range Tb from the boundary B2 of the second unit echo image Pn+1. The time ranges Ta and Tb are set to approximately the maximum value of the time-axis gap on the echo image that may occur in a single school of fish. The time ranges Ta and Tb may be set to the same time width.

[0097] In the example in Figure 9(a), since the fish school Fa (fish school range Ra) is within the time range Ta, the determination in step S101 in Figure 8 is YES, and the fish school Fa is identified as the first fish school. Also in the example in Figure 9(a), since the fish school Fb (fish school range Rb) is within the time range Tb, the determination in step S103 in Figure 8 is YES, and the fish school Fb is identified as the second fish school.

[0098] In contrast, in the example of Figure 9(b), since the fish school Fa (fish school range Ra) does not exist within the time range Ta, the determination in step S101 in Figure 8 is NO, and the fish school Fa is not identified as the first fish school. Also, in the example of Figure 9(b), since the fish school Fb (fish school range Rb) does not exist within the time range Tb, the determination in step S103 in Figure 8 is NO, and the fish school Fb is not identified as the second fish school.

[0099] Furthermore, in step S105 of Figure 8, the determination is made using the following formula (1).

[0100]

number

[0101] As shown in Figure 9(a), Amax and Amin are the maximum and minimum depths of the first fish school Fa (fish school range Ra) in the first unit echo image Pn, respectively, and Bmax and Bmin are the maximum and minimum depths of the second fish school Fb (fish school range Rb) in the second unit echo image Pn+1, respectively.

[0102] In other words, the denominator of the left side of equation (1) is the depth range from the smaller of Amin and Bmin (shallower) to the larger of Amax and Bmax (deeper), and in the example of Figure 9(a), it is the depth range from Amin to Bmax. Also, the numerator of the left side of equation (1) is the depth range from the larger of Amin and Bmin (deeper) to the smaller of Amax and Bmax (shallower), and in the example of Figure 9(a), it is the depth range from Bmin to Amax. The left side of equation (1) is used to calculate the matching rate between the depth range of the first fish group Fa and the depth range of the second fish group Fb. In step S105 of Figure 8, it is determined whether the depth ranges of the first fish group Fa and the second fish group Fb approximate match based on whether this matching rate is greater than or equal to the threshold Th1. The threshold Th1 may be a statistically set default value, or it may be arbitrarily set by the administrator of server 20.

[0103] Figures 10(a) and (b) show the method for integrating the annotation data of the first and second fish groups in step S13 of Figure 8.

[0104] In step S13 of Figure 8, the control unit 201 aligns the depth ranges of the fish school range Ra of the first fish school Fa and the fish school range Rb of the second fish school Fb. That is, as shown in Figure 10(a), the control unit 201 corrects the depth ranges of the fish school ranges Ra and Rb so that the smaller of the maximum depths Amax and Bmax (Amax) is aligned with the larger of the maximum depths Amax and Bmax (Amax), and the larger of the minimum depths Amin and Bmin (Bmin) is aligned with the smaller of the minimum depths Amin and Bmin (Amin). Then, as shown in Figure 10(b), the control unit 201 integrates the annotation data of the corrected fish school ranges Ra and Rb to generate annotation data for a single fish school composed of fish schools Fa and Fb.

[0105] The generated annotation data consists of a fish school range Rab, which is an integrated range of the fish school Ra and Rb, echo data included in this fish school range Rab, and the fish species of the fish school (in this case, sardines). Like the fish school ranges Ra and Rb, the fish school range Rab is also rectangular, and the fish schools Fa and Fb are contained within this rectangular range. As described above, the control unit 201 of the server 20 uses the integrated annotation data to train the machine learning model 301.

[0106] Figures 9(a) to 10(b) illustrate the fish schools in the upper right portion of the first unit echo image Pn and the upper left portion of the second unit echo image Pn+1. However, if there are multiple fish schools that span both the first and second unit echo images, the same processing as described above is performed on each fish school, generating multiple integrated annotation data sets.

[0107] Furthermore, if a single school of fish spans three or more unit echo images, the above processing is performed sequentially on adjacent unit echo images. This integrates the annotation data for the parts of the fish school that are divided across three or more unit echo images, generating annotation data for a single school of fish.

[0108] <Effects of the Embodiment> According to the embodiment, the following effects may be achieved.

[0109] As shown in Figures 7 to 10(b), annotation data of the fish school spanning between the first unit echo image Pn and the second unit echo image Pn+1 (first annotation data and second annotation data) are integrated to generate annotation data for the entire fish school. Therefore, since annotation data for the entire fish school can be used in machine learning, the accuracy of machine learning for fish species discrimination can be improved.

[0110] As shown in Figure 9(a), the control unit 201 identifies a fish school Fa that extends to the vicinity of boundary B1 (first boundary) on the second unit echo image Pn+1 side on the first unit echo image Pn as the first fish school, and identifies a fish school Fb that extends to the vicinity of boundary B2 (second boundary) on the first unit echo image Pn+1 side on the second unit echo image Pn as the second fish school. When the range Ra of fish school Fa (first fish school) and the range Rb of fish school Fb (second fish school) substantially coincide, the control unit 201 extracts the annotation data of these fish schools Fa and Fb (first fish school and second fish school) as annotation data to be integrated (first annotation data and second annotation data), respectively. This makes it possible to smoothly identify fish schools Fa and Fb (first fish school and second fish school), which have a high probability of constituting a single fish school, in the first unit echo image Pn and the second unit echo image Pn+1. Therefore, by integrating the annotation data of these fish schools, it is possible to generate accurate annotation data for an entire fish school.

[0111] As shown in Figure 9(a), the control unit 201 extracts the fish group Fa, which is at least partially included in a predetermined time range Ta from boundary B1 (first boundary), as the fish group to be integrated (first fish group), and extracts the fish group Fb, which is at least partially included in a predetermined time range Tb from boundary B2 (second boundary), as the fish group to be integrated (second fish group). As a result, even when the distribution of fish is sparse near boundaries B1 and B2, the first and second fish groups that constitute a fish group spanning the first unit echo image Pn and the second unit echo image Pn+1 can be identified without any omissions in the first unit echo image Pn and the second unit echo image Pn+1. Therefore, annotation data for the entire fish group can be generated appropriately.

[0112] <Example of change 1> The present invention is not limited to the above embodiments, and various modifications are possible to the embodiments of the present invention other than the above configuration.

[0113] For example, in the above embodiment, the range of the fish school constituting the annotation data (time range, depth range) was rectangular, but the range of the fish school constituting the annotation data may be other shapes besides rectangular.

[0114] For example, as shown in Figure 11(a), the fish school ranges Ra and Rb may have a shape that follows the outer edges of the fish schools Fa and Fb. In this case, as described above, on the echo image, regions where the echo intensity is above a predetermined threshold and which are connected to the echo intensity are extracted as fish schools Fa and Fb, and the ranges along the outer edges of the extracted fish schools Fa and Fb are extracted as fish school ranges Ra and Rb. When the operator indicates the location of the fish school on the echo image, the control unit 501 of the terminal device 50 displays the fish school range extracted by the above process overlaid on the echo image for the fish school that includes the specified location.

[0115] In this case as well, in step S101 of Figure 8, it is determined whether a school of fish exists in the time range Ta from boundary B1, and in step S103, it is determined whether a school of fish exists in the time range Tb from boundary B2. As a result, as shown in Figure 11(a), the school of fish Fa and Fb are identified as the first school of fish and the second school of fish, respectively.

[0116] In this case, in step S105 of Figure 11(a), whether the depth range of fish school Fa (first fish school) and the depth range of fish school Fb (second fish school) are nearly identical is determined based on the degree of agreement between the depth range of fish school Fa (first fish school) near boundary B1 (first boundary) and the depth range of fish school Fb (second fish school) near boundary B2 (second boundary).

[0117] Specifically, for fish school Fa, the maximum depth Amax and minimum depth Amin are obtained for the portion of the fish school range Ra that falls within the time range Ta, and for fish school Fb, the maximum depth Bmax and minimum depth Bmin are obtained for the portion of the fish school range Rb that falls within the time range Tb.

[0118] The control unit 201 applies the maximum depths Amax and Bmax and minimum depths Amin and Bmin obtained in this way to equation (1) and makes the determination in step S105. In this way, by determining whether the depth range of fish school Fa (first fish school) and the depth range of fish school Fb (second fish school) substantially coincide near boundary B1 (first boundary) and boundary B2 (second boundary), it is possible to accurately determine whether these fish schools Fa and Fb (first fish school and second fish school) are continuous with each other. Therefore, annotation data for the entire fish school can be generated with high accuracy from the annotation data of fish schools Fa and Fb (first fish school and second fish school).

[0119] In this case, in step S13 of Figure 11(a), as shown in Figure 11(b), the range Ra and Rb of the fish school from which the maximum depths Amax and Bmax were obtained are connected to each other, and the range Ra and Rb of the fish school from which the minimum depths Amin and Bmin were obtained are connected to each other, thereby setting the range Rab of the entire fish school. This allows the range Rab of the fish school that constitutes the annotation data for the entire fish school to be set smoothly and appropriately.

[0120] However, this is not the only way to set the fish school range Rab. For example, the fish school range Rab may be set by connecting the locations of the fish school range Ra and Rb slightly closer to the boundaries B1 and B2 than the locations of the fish school range Ra and Rb from which the maximum depths Amax and Bmax were obtained, respectively, and by connecting the locations of the fish school range Ra and Rb slightly closer to the boundaries B1 and B2 than the locations of the fish school range Ra and Rb from which the minimum depths Amin and Bmin were obtained, respectively. In this case as well, the fish school range Rab that constitutes the annotation data for the entire fish school can be set smoothly and appropriately.

[0121] <Example of change 2> In the above embodiment, it was assumed that the fish species constituting the annotation data for the first fish school and the fish species constituting the annotation data for the second fish school were the same. However, determining the fish species based on the fish school images can be difficult for operators. Therefore, it is possible that the fish species in the annotation data set for the first and second fish schools, which are identified as constituting a single fish school, may differ from each other.

[0122] In example 2 of the modification, in such cases, the fish species is set for the integrated annotation data based on predetermined conditions.

[0123] Figure 12 is a flowchart showing the fish species setting process performed in the annotation data integration process in step S13 of Figure 8.

[0124] The control unit 201 determines whether the fish species in the annotation data of the first and second fish groups to be integrated match (S201). If they match (S201: YES), the control unit 201 sets the matched fish species as the fish species of the integrated fish group (S202). On the other hand, if they do not match, the control unit 201 sets the fish species of the integrated fish group according to predetermined setting conditions (S203).

[0125] The setting conditions in step S203 may include, for example, setting the fish species to the more recent annotation data among the annotation data of the first and second fish groups (first annotation data, the second annotation data).

[0126] For example, in the example shown in Figure 11(a), if the annotation data for fish school Fa (first fish school) and fish school Fb (second fish school) consists of different fish species, the control unit 201 sets the fish species of the annotation data for fish school Fb obtained from the newer unit echo image Pn+1 as the fish species for the integrated fish school range Rab.

[0127] Furthermore, the setting conditions in step S203 are not limited to the condition (first condition) of setting the fish species in the newer annotation data, as described above. Other conditions may be used as long as it is possible to set a highly reliable fish species in the annotation data after integration.

[0128] For example, a condition (second condition) that sets the fish species of the annotation data of the integrated fish group to be set to the fish species of the annotation data of the fish group after integration (first annotation data, second annotation data) of the first and second fish groups to be integrated (first annotation data, second annotation data).

[0129] Alternatively, a condition (third condition) that the fish species obtained by processing the integrated annotation data with the fish species discrimination machine learning model 301 is set as the fish species in the annotation data of the integrated fish school may be included in the setting conditions of step S203.

[0130] In this case, for example, if the fish species cannot be set by the second condition (for example, if the range of the first fish group and the range of the second fish group are approximately the same), the first or third condition may be applied to set the fish species. Alternatively, the fish species with the largest number of matching species among the fish species obtained by the first to third conditions may be set as the fish species in the annotation data of the merged fish group. The setting conditions are not limited to the first to third conditions, but may include other conditions as well.

[0131] Thus, when the annotation data for fish school Fa (first fish school) and fish school Fb (second fish school) consist of different fish species, performing the process in step S203 allows us to set the most reliable fish species in the integrated annotation data. Therefore, the accuracy of machine learning can be improved.

[0132] If the determination in step S201 is NO, the process of integrating the annotation data of the first and second fish groups may be canceled. In this case, the control unit 201 may exclude both the annotation data of the first and second fish groups from the annotation data used for machine learning of the machine learning model 301, or it may exclude one of these annotation data (for example, the older one or the one with a smaller fish group range) from the annotation data used for machine learning of the machine learning model 301.

[0133] <Other examples of changes> In the above embodiments and modified examples 1 and 2, the processes shown in Figures 7 and 8 were performed in the control unit 201 of the server 20. However, these processes may be performed in a device other than the server 20, and the integrated annotation data may be provided to the server 20. For example, the processes shown in Figures 7 and 8 may be performed by the control unit 501 of the terminal device 50 after the operator has generated the annotation data, and the control unit 501 may transmit the annotation data generated by the operator and the annotation data generated by the processes shown in Figures 7 and 8 to the server 20. In this case, the terminal device 50 corresponds to the fish species learning device described in the claims.

[0134] Furthermore, in the above embodiments and modified examples 1 and 2, the depth ranges of the first and second fish schools were determined by formula (1) above, but this method of determination is not limited to this. For example, in Figure 9(a), if the difference between the first depth range between the maximum depth Amax and the minimum depth Amin and the second depth range between the maximum depth Bmax and the minimum depth Bmin is smaller than a predetermined threshold, and the difference between the intermediate depth of the first depth range and the intermediate depth of the second depth range is smaller than a predetermined threshold, then the depth ranges of the first and second fish schools may be determined to be roughly the same, and if these conditions are not met, then the depth ranges of these fish schools may be determined to be not the same. In the case of Figure 11(a), the depth ranges of the first and second fish schools may also be determined by a similar determination method.

[0135] Furthermore, the method for extracting annotation data for a fish school spanning the first unit echo image and the second unit echo image is not limited to the method shown in steps S101 to S105 of Figure 8. For example, it may be determined whether the first and second fish schools constitute a single fish school based on the continuity of the distribution of the echo data for the first fish school and the echo data for the second fish school.

[0136] Furthermore, in the above embodiments and modified examples 1 and 2, it was assumed that the frequency of the transmitted wave sent by the transmitter / receiver unit 104 was one, but two different frequencies of transmitted waves may be sent from the transmitter / receiver unit 104. In this case, echo data based on the received signals of each frequency may be applied to the machine learning model 301 for each school of fish to determine the species of fish in each school. By transmitting and receiving at two different frequencies in this way, the machine learning model 301 can determine the species of fish with higher accuracy. For example, the presence or absence of a swim bladder will cause a difference in the echo intensity of each frequency. Therefore, by referring to the difference in the echo intensity from the school of fish, the species of fish in that school can be determined with high accuracy.

[0137] In this case, annotation data generated from unit echo images can be generated for each fish school at each frequency. Furthermore, the integration process of annotation data shown in Figures 7 and 8 can be performed individually for each frequency of echo data (unit echo image).

[0138] Furthermore, in the above embodiment, the fish species discrimination process using the machine learning model 301 was performed on the server 20 side, but this discrimination process may also be performed on the underwater detection device 10 side. In this case, the machine learning model 301 updated with annotation data is transmitted to the underwater detection device 10 as needed and stored in the underwater detection device 10. The control unit 101 of the underwater detection device 10 performs fish species discrimination using the machine learning model 301 based on the echo data acquired by the transmitter / receiver unit 104 and the signal processing unit 105, similar to the control unit 201 of the server 20, and displays the discrimination result in the echo image.

[0139] Furthermore, although annotation data was generated in the terminal device 50 in the above embodiment, the device that generates annotation data is not limited to this. For example, a user of the underwater detection device 10 may input a school of fish and its species from the echo image based on their own fishing results, and the echo data of this school of fish and the species of fish in that school may be sent to the server 20 as annotation data.

[0140] Furthermore, in the above embodiment, the underwater detection device 10 was a fish finder, but the underwater detection device 10 may be a device other than a fish finder, such as a sonar.

[0141] In addition, embodiments of the present invention can be modified in various ways as appropriate within the scope of the claims. [Explanation of symbols]

[0142] 20 Servers (Machine Learning Devices) 201 Control Unit 202 Storage section 301 Machine Learning Models Pn, Pn+1 unit echo images (1st unit echo image, 2nd unit echo image) Fa, Fb Fish school (1st fish school, 2nd fish school) B1, B2 boundary (1st boundary, 2nd boundary) Ta, Tb time range

Claims

1. A storage unit that stores annotation data for each unit echo image in which a series of echo images are divided into multiple parts, It comprises a control unit and, The control unit, From the annotation data of the first and second unit echo images, which are two temporally consecutive unit echo images, the first and second annotation data of the fish school spanning between the first and second unit echo images are extracted, respectively. The extracted first annotation data and the second annotation data are integrated to generate annotation data for the entire fish school. A fish species learning device characterized by the following features.

2. In the fish species learning device described in claim 1, The control unit, Identify a first group of fish that extends to the vicinity of the first boundary on the second unit echo image side in the first unit echo image, and identify a second group of fish that extends to the vicinity of the second boundary on the first unit echo image side in the second unit echo image. When the depth range of the first fish school and the depth range of the second fish school substantially coincide, the annotation data of the first fish school and the second fish school are extracted as the first annotation data and the second annotation data, respectively. A fish species learning device characterized by the following features.

3. In the fish species learning device according to claim 2, The control unit determines whether the depth range of the first fish school and the depth range of the second fish school substantially coincide based on the degree of agreement between the depth range of the first fish school near the first boundary and the depth range of the second fish school near the second boundary. A fish species learning device characterized by the following features.

4. In the fish species learning device according to claim 2, The control unit extracts a group of fish whose fish are at least partially included within a predetermined time range from the first boundary as the first group of fish, and extracts a group of fish whose fish are at least partially included within a predetermined time range from the second boundary as the second group of fish. A fish species learning device characterized by the following features.

5. In the fish species learning device described in claim 1, If the fish species included in the first annotation data differ from the fish species included in the second annotation data, the control unit sets the fish species in the annotation data of the fish group based on predetermined setting conditions. A fish species learning device characterized by the following features.

6. In the fish species learning device described in claim 5, The aforementioned setting conditions are: This includes setting the fish species to the more recent annotation data among the first and second annotation data. A fish species learning device characterized by the following features.

7. Annotation data is stored for each unit echo image into which a series of echo images are divided into multiple parts. From the annotation data of the first and second unit echo images, which are two temporally consecutive unit echo images, the first and second annotation data of the fish school spanning between the first and second unit echo images are extracted, respectively. The extracted first annotation data and the second annotation data are integrated to generate annotation data for the entire fish school. Our fish species learning method.

8. On the computer, A function that stores annotation data for each unit echo image, which is divided into multiple sections from a series of echo images, A function to extract first and second annotation data of a school of fish spanning between the first and second unit echo images, respectively, from the annotation data of two temporally consecutive unit echo images, namely the first unit echo image and the second unit echo image. A program that performs the function of integrating the extracted first annotation data and the second annotation data to generate annotation data for the entire school of fish.

Images