Fish weight measurement method and fish weight measurement device

Abstract

To determine the body length of a fish with high accuracy and calculate the weight of the fish from the body length.SOLUTION: A fish weight measurement method has an imaging device and an ultrasonic transmitter / receiver in which the imaging direction and the ultrasonic transmission direction are the same and the ultrasonic transmission timing and the imaging timing are synchronized, and uses the echo obtained by the ultrasonic transmitter / receiver to determine the distance between the imaging device and a reference point of the fish area in image data of the imaging device, determines a body length of the fish from the image of the fish in the image data of the imaging device and the distance, and uses the body length to determine the fish weight.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a fish weight measuring method and a fish weight measuring device for measuring the weight of fish (fish weight) in, for example, an aquaculture cage. [Background technology]

[0002] In aquaculture management, it is necessary to know the total weight (biomass weight) of the fish in the pen. By knowing the total weight of the fish in the pen, appropriate feeding can be carried out. The total weight of the fish in the pen is expressed as (number of fish x weight of fish per fish). The present invention measures the weight of fish per fish, and by multiplying this by the number of fish determined by other techniques, biomass weight can be determined.

[0003] Known conventional methods for counting the number of fish are described in Patent Documents 1 and 2. The method described in Patent Document 1 uses a camera. Patent Document 2 describes a measurement system that includes means for detecting characteristic parts, such as the head and tail, from a captured image, means for calculating their length, and means for displaying a frame or the like surrounding the fish body. The methods described in Patent Documents 1 and 2 use a stereo camera. However, with a configuration that uses an underwater stereo camera, measurement becomes difficult when the image is unclear due to low light or turbidity.

[0004] Patent document 3 describes a method for tracking fish to estimate their body length with high accuracy (calculating their body length in successive images). Patent document 4 describes a system that includes a camera-based fish size calculation unit and an echo-based fish counting unit.

[0005] Furthermore, Patent Document 5 describes a method of applying ultrasound to a fish from above and obtaining information about the fish's body height from the resulting echoes, and Patent Document 6 describes a method of applying ultrasound to a fish from the side and obtaining information about the fish's body width from the resulting echoes. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 6694039 [Patent Document 2] Patent Publication No. 2021-157490 [Patent Document 3] Japanese Patent Publication No. 2022-032975 [Patent Document 4] Japanese Patent Publication No. 2023-035996 [Patent Document 5] Japanese Patent Application Laid-Open No. 2008-267834 [Patent Document 6] Patent Publication No. 2021-045102 Summary of the Invention [Problem to be solved by the invention]

[0007] The system described in Patent Document 3 requires multiple consecutive images, which makes the processing more complex than processing a single image. The system described in Patent Document 4 uses a fish size calculation unit based on camera images, which can lead to blurred images due to low light or turbidity. Furthermore, the systems described in Patent Documents 5 and 6 calculate fish size information from ultrasonic echoes, but this method is less accurate than analyzing camera images.

[0008] An object of the present invention is to provide a method and device for measuring fish weight that can determine body length with high accuracy and measure fish weight using the body length. [Means for solving the problem]

[0009] The present invention has an imaging device and an ultrasonic transmitter / receiver in which the imaging direction and the ultrasonic wave transmission direction are the same direction, and the ultrasonic wave transmission timing and the imaging timing are synchronized, Using the echoes obtained by the ultrasonic transducer, fish Calculate the distance between the reference point in the area and the imaging device, The length of the fish is calculated based on the image of the fish in the image data of the imaging device and the distance. Fish weight measurement method using body length to determine fish weight And, Calculate the center of gravity of the fish area in the image data; The pixel at the center of gravity, the pixel at the head side of the fish, and the pixel at the tail side of the fish are set as pixels at positions that define the body length of the fish, A method for measuring fish weight, which calculates the distance from the pixel on the head side to the pixel on the tail side passing through the pixel at the center of gravity from coordinates, and regards this distance as the body length of the fish is. [Effects of the Invention]

[0010] According to the present invention, an ultrasonic transmitter / receiver is provided, and the body length of a fish can be calculated from image data of a single imaging device using echoes obtained by the ultrasonic transmitter / receiver, and the body length can be used to measure the weight of the fish. Note that the effects described herein are not necessarily limited to those described herein, and any of the effects described in this specification may be used. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of an embodiment of a fish weight measuring device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram for explaining the downward placement of the transducer and camera. [Figure 3] FIG. 3 is a schematic diagram for explaining the horizontal arrangement of the transducer and the camera. [Figure 4] FIG. 4 is a schematic diagram for explaining one embodiment of the present invention. [Figure 5] FIG. 5 is a flowchart illustrating the echo analysis process in one embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart illustrating the image analysis process according to one embodiment of the present invention. [Figure 7] FIG. 7 is a flowchart illustrating an example of a fish weight analysis process according to one embodiment of the present invention. [Figure 8] FIG. 8 is a flowchart illustrating another example of the fish weight analysis process according to one embodiment of the present invention. [Figure 9]FIG. 9 is a block diagram for explaining the fish weight prediction process in one embodiment of the present invention. [Figure 10] FIG. 10 is a graph showing an example of echo intensity. [Figure 11] FIG. 11 is a schematic diagram for explaining the process of connecting echo peaks. [Figure 12] FIG. 12 is an echogram corresponding to FIG. [Figure 13] 13A, 13B, and 13C are schematic diagrams for explaining the change in distance for each ping of a fish. [Figure 14] 14A, 14B, 14C, 14D, 14E, and 14F are schematic diagrams used to explain region calculation using a camera image. [Figure 15] 15A, 15B, 15C, and 15D are schematic diagrams used to explain calculation of the movement direction of a fish. [Figure 16] 16A and 16B are schematic diagrams used to explain the calculation of fish size. [Figure 17] FIG. 17 is a schematic diagram illustrating an example of a size calculation method. [Figure 18] FIG. 18 is a schematic diagram illustrating an example of a size calculation method. [Figure 19] 19A, 19B, 19C, 19D, 19E, and 19F are schematic diagrams used to explain region calculation using a camera image. [Figure 20] 20A, 20B, 20C, and 20D are schematic diagrams used to explain calculation of the movement direction of a fish. [Figure 21] 21A and 21B are schematic diagrams used to explain the calculation of fish size. [Figure 22] FIG. 22 is a schematic diagram illustrating an example of a size calculation method. [Figure 23] FIG. 23 is a schematic diagram illustrating an example of a size calculation method. [Figure 24] FIG. 24 is a graph showing the relationship between body height and body height on the body axis. [Figure 25] FIG. 25 is a schematic diagram showing a cross section of a fish at a position a predetermined distance from the head. [Figure 26] FIG. 26 is a graph showing an example of a 3D model of a fish. [Figure 27] FIG. 27 is a graph showing an example and another example of the cross-sectional area along the body axis. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described. Note that the embodiments described below are preferred specific examples of the present invention, and various technically preferable limitations are attached, but the scope of the present invention is not limited to these embodiments unless otherwise stated in the following description to the effect that the present invention is limited.

[0013] 1 is a block diagram showing a schematic configuration of a fish weight measurement system according to one embodiment of the present invention. The embodiment is an example in which the present invention is applied to measuring the weight of fish in a fish cage in the sea.

[0014] In one embodiment, for example, one camera (underwater camera) V and one ultrasonic transmitter / receiver S are used. The camera V and the transmitter / receiver S are installed relative to the fish pen as shown in Figure 2 or Figure 3. In Figures 2 and 3, the dashed lines represent the net that forms the fish pen. In the installation method shown in Figure 2, the imaging direction of the camera V and the ultrasonic transmission direction of the transmitter / receiver S are set from the sea near the surface toward the bottom (seabed). The camera V and the transmitter / receiver S are positioned relatively close to each other, and the imaging direction of the camera V and the ultrasonic transmission direction of the transmitter / receiver S are set downward. This type of arrangement is called a downward-facing arrangement.

[0015] The ultrasonic beam generated by the transducer S is a conical beam that spreads out in a cone shape with a beam angle of, for example, 5 degrees. The cone shape is when the transducer of the transducer S is circular, and when the transducer is square, it becomes a square pyramid. A fan beam may also be used. For example, the frequency of the ultrasonic waves is set to about 240 kHz.

[0016] In measurements, it is important to detect each individual fish. Because fish pass by in an instant, the ultrasonic waves are transmitted at least 10 times per second, for example 20 times. As a result, the echoes of individual fish can be separated and received even if they are swimming fast.

[0017] Figure 3 shows another installation method. In this installation method, the imaging direction of the camera V and the ultrasonic transmission direction of the transducer S are set in the horizontal direction of the fish pen. The camera V and the transducer S are positioned relatively close to each other. This type of installation is called a horizontal installation. In the horizontal installation, the ultrasonic beam is set to hit the side of the fish as much as possible, and the camera V is set to image the side of the fish.

[0018] In one embodiment of the present invention, either a downward-facing arrangement (Fig. 2) or a horizontal arrangement (Fig. 3) is used depending on the shape of the fish pen. For example, in the case of a fish pen for sea bream or the like, there is a bridge in the center of the pen to allow for the use of a feeder, and the transducer S and camera V can be installed on this bridge, so either a downward-facing arrangement or a horizontal arrangement is applicable. On the other hand, in the case of a fish pen for tuna or the like, there is no bridge, so a downward-facing arrangement cannot be applied and a horizontal arrangement is used.

[0019] Note that an upward-facing arrangement, in which the imaging direction of the camera V and the ultrasonic transmission direction of the transducer S are set from underwater toward the sea surface, may be used instead of a downward-facing arrangement. Furthermore, the imaging direction of the camera V and the ultrasonic transmission direction of the transducer S can also be installed in an oblique direction. In the case of an oblique orientation, if the direction is close to downward, it can be considered a downward-facing arrangement, and if it is close to horizontal, it can be considered a horizontal arrangement. Alternatively, calculations can be performed as an oblique orientation. In this case, the pixel-to-pixel distance corresponding to the body width in the direction perpendicular to the direction of ultrasonic propagation can be calculated. In other words, even with an oblique orientation, if the body width can be calculated from the echo, it can be arranged and processed in the same way as a horizontal orientation. An oblique orientation that is slightly shifted from horizontal can be treated as a horizontal orientation.

[0020] Information about the size of the fish includes the length from the tip of the fish's mouth to the base of the tail fin (called body length), the maximum width (thickness) of the fish (called body width), and the maximum length from the dorsal to the ventral side of the fish (called body height). Note that information about the fork length may be used instead of the body length. In the downward-facing configuration, the body length and body height of the target fish can be measured by processing the ultrasonic reception signal (called an echo). In the downward-facing configuration, the body length and body width can be measured by processing the image signal from the camera V.

[0021] In the case of a horizontal position, the body length and body width can be measured by processing the echoes. In the case of a horizontal position, the body length and body height can be measured by processing the image signal of the camera V.

[0022] The timing of ultrasonic transmission from the transducer S and the timing of image capture by the camera V are synchronized. For example, the transducer S transmits ultrasonic waves about 20 times per second, and the camera V captures images synchronized with the transmission timing. For example, an image is extracted at the time (ping) when a fish is detected from the echo. In other words, the frame rate of the camera V is synchronized with the ultrasonic transmission cycle. An object close to directly in front is detected from the digital image data (simply referred to as image data) captured by the camera at the time the echo detects the object, and the area of ​​the fish is calculated. Note that in Figure 1, the circuit for generating the transmission signal for the transducer S is omitted.

[0023] The reference point for the fish region is, for example, the center of gravity of the region. The reference point is not limited to the center of gravity, and the tip of the fish's head or the base of the caudal fin may also be used. The movement direction of the fish is calculated based on the image signal of camera V around the time of the echo and the change in distance (depth in the case of downward placement) calculated from the echo. In other words, the movement direction is the direction of movement that causes a change in distance to camera V.

[0024] FIG. 4 shows the relationship between the distance between the camera V and the fish F when the camera is positioned facing downward. As will be described later, in one embodiment of the present invention, the body length of the fish F is measured to determine the fish's weight. The body length is defined as the length from the tip of the head P of the fish F to the base Q of the caudal fin. Pv represents a plane perpendicular to the camera V at a certain distance. If each part of the body is on this plane, that is, if the fish is at a certain distance from the camera V, the actual body length can be determined by knowing this distance and the number of pixels between P and Q. A fish at a certain distance is a fish that does not swim in the distance direction.

[0025] However, there are many cases where the distance is not constant. Figure 4 shows an example of a fish F swimming downward. In this case, the distance varies depending on the part of the fish, so calculating the actual size from the number of pixels using a representative distance will result in errors. To reduce errors, it is necessary to calculate the distance for each part of the fish, which requires the swimming direction of the fish. In Figure 4, the movement direction indicated by the downward arrow can be used to calculate the distance for each part of the fish, making it possible to calculate accurate actual size values. The movement direction refers to the direction of movement that causes a change in distance, but will be simply referred to as the movement direction below. The method for calculating the movement direction will be described later. The movement direction is also taken into consideration when the fish swims upward. Furthermore, in a horizontal orientation, the direction of movement toward and away from the camera V is detected as the movement direction.

[0026] In the image signal from camera V, the fish area is converted into size based on the distance between camera V and the fish. A more accurate size can be determined based on the direction of movement of the fish. The center of gravity of the fish area in the image data is determined, and the body length and width (if camera V is positioned facing downward) or body length and height (if camera V is positioned facing sideways) are calculated using this center of gravity.

[0027] Camera V can capture images of fish at relatively close range, but cannot capture images of fish that have hidden behind the fish's shadow. Transducer S allows ultrasonic waves to pass through fish, so it can also receive signals (echoes) reflected by later fish. When an image of the fish cannot be obtained, the fish's weight is predicted from the echoes based on the results of learning the measurement data.

[0028] An embodiment of the present invention will be further described. The received signal (echo) of the transducer S is supplied to the receiving unit 1. The output of the receiving unit 1 is supplied to the fish detection unit 2. Fish information is obtained through processing by the fish detection unit 2. The fish information output from the fish detection unit 2 is supplied to the prediction unit 3 and the fish weight calculation unit 8. The output of the memory unit 4 is supplied to the prediction unit 3. The prediction unit 3 performs a fish weight prediction process. The fish weight data predicted by the prediction unit 3 is supplied to the display unit 5 and displayed on the display unit 5.

[0029] The image signal from the camera V is supplied to the receiving unit 6, where it is digitized and processed. The output of the receiving unit 6 is supplied to the fish detection unit 7. Information about the size of the fish is obtained through processing by the fish detection unit 7. The fish size information output from the fish detection unit 7 is supplied to the fish weight calculation unit 8. The fish weight obtained by the fish weight calculation unit 8 is stored in the memory unit 4 and displayed on the display unit 5.

[0030] In aquaculture management, it is necessary to know the total weight of the fish in the pen (biomass weight). By knowing the total weight of the fish in the pen, appropriate feeding can be carried out. In order to know the total weight of the fish in the pen, information on the number of fish and their weight is important. Total fish weight in the fish pen = number of fish x weight of each fish The present invention measures fish weight, and the number of fish is counted using a separate fish counting device. The measured fish weight data can be statistically processed to determine the weight of each fish.

[0031] The fish weight calculation unit 8 measures the fish weight using the first, second, and third fish weight measurement methods using the fish information from the fish detection units 2 and 7. The measured fish weight data is supplied to the display unit 5 and displayed. The measured fish weight data may also be transmitted to a display device in a remote location or to a smartphone. Furthermore, the display unit 5 may display the number of fish measured by a fish counting device (not shown), or the weight of the entire fish in the fish pen.

[0032] The outline of the processing flow according to one embodiment of the present invention is as follows. Process 1: Measure the echo and detect the fish. If the sensor is positioned facing downwards, measure the distance and body height. If the sensor is positioned facing sideways, measure the distance and body width. Calculate the difference in distance between successive transmissions (called pings).

[0033] Process 2: Detect an object close to directly in front from the image data of camera V at the time of detection in process 1, and calculate the fish area and center of gravity. If the fish is positioned facing downwards, calculate the number of pixels related to the fish's length and width, and if the fish is positioned facing sideways, calculate the number of pixels related to the fish's length and height.

[0034] Process 3: Calculate the direction and speed of the fish's movement based on the change in the direction of the center of gravity from the image data of camera V around the time detected in process 1 and the change in depth calculated from the echo.

[0035] Process 4: The fish area (number of pixels) is converted into size based on the distance between camera V and the fish. While distance alone can be used, a more accurate size can be obtained by using the direction of movement. Furthermore, the center of gravity is used to calculate the body length and width (in downward orientation) or the body length and height (in sideways orientation) from the image data of camera V.

[0036] Process 5-1: The first fish weight measurement method is performed. The direction of movement is determined from the echo, and the body length and body width (if positioned facing downwards) or body height (if positioned facing sideways) are calculated from the image data of camera V. Fish weight is calculated directly from the body length. This calculation method uses a body length-fish weight conversion formula (referred to as Almetri's formula, calculation formula, or formula as appropriate) to calculate fish weight from body length, and the parameters of this formula are automatically corrected based on the degree of obesity. In the case of a downward-facing arrangement, the degree of obesity is calculated based on the body width obtained from the image data of camera V, and the parameters of the formula are automatically adjusted based on the degree of obesity, and then converted into fish weight. In the case of measurements when the camera is positioned facing sideways, the degree of obesity is calculated based on the body height obtained from the image data of camera V, and the parameters of the formula are automatically adjusted based on the degree of obesity. The Almetri's formula is determined for each fish species.

[0037] Process 5-2: The second fish weight measurement method is performed. Not only the direction of movement but also the body width is calculated from the echo, and the body length and body height are calculated from the image data of camera V. The fish weight is calculated based on Almetri's formula, which adjusts the degree of obesity based on the body width. The parameters of Almetri's formula are automatically corrected based on the degree of obesity. When measuring from the side, the degree of obesity is evaluated using the body width calculated from the echo, the parameters of the formula are automatically adjusted, and the fish weight is calculated.

[0038] Process 5-3: The third fish weight measurement method (a measurement method based on a 3D (three-dimensional) model) is performed. The fish weight is calculated based on the image data from Camera V. In the case of downward placement: The volume is calculated by combining the size (body length and body width), body height information, and fish species information, and the fish weight is calculated from the density determined for each fish species. When measuring horizontally: The volume is calculated by combining the size (body length and height), body width information, and fish species information, and the fish weight is calculated from the density determined for each fish species.

[0039] Process 6: This is a process for determining fish weight by prediction (inference). The third fish weight measurement method is performed for a predetermined time, for example, about 30 minutes, and the relationship between echo shape and fish weight is learned. Measurements over a certain period of time provide information on hundreds to thousands of fish, and learning is performed based on this information. Machine learning methods, including deep learning, are used for learning. In the downward-facing example, body width is calculated by analyzing the camera's image data, and the body width information is used to determine the degree of obesity for that fish species. The relationship between this degree of obesity and the echo is learned based on the measurement data. Based on this learning result, the degree of obesity is estimated from the echo, and the fish weight is calculated. In other words, fish weight prediction is performed for echoes (fish) that do not correspond to the camera.

[0040] One of the first, second, and third fish weight measurement methods described above is selected and used. The method considered best is selected depending on the shape of the net cage and the fish species. For example, the fish weight measurement method is used in the following order of priority: first, second, and third fish weight measurement methods. However, for fish species for which there is no Almetri's formula, the third fish weight measurement method (a measurement method based on a 3D model) is used in the initial stage. Based on the fish weight calculated from the volume, parameters of the Almetri's formula are adjusted depending on the degree of obesity. Once data can be collected at a level that allows the relationship between body length, body width, and body height and parameters to be shown, the first fish weight measurement method, which does not involve a 3D model, is used.

[0041] The first fish weight measurement method will now be described in further detail. When face-down placement is used, the following process is performed: (1) Calculate the distance and direction of movement from the echo. (2) Calculate the size related to body length and body width from the image of camera V. (3) Calculate the fish weight using Almetri's equation for each fish species. The Almetri's equation used is the one that has already been proposed. (4) The automatic adjustment of body width and obesity level can be created as learning data, and improved automatic adjustment can be performed for the next measurement. In other words, the accuracy of the adjustment parameters can be improved through learning.

[0042] Here, the relationship between body length and weight of fish (Almetri's equation) is expressed as follows: W=KL^p In the above equation, W is the fish's weight, L is its body length, and K and p are adjustment parameters.

[0043] These K and p are adjusted. Basically, the value of K is changed. In other words, the fish weight W and body length L are input, and the relationship between K and p is found based on the body height and body width. You can either calculate an approximate formula using body width and body height, or learn the relationship.

[0044] However, since this method cannot be applied to the first and second fish weight measurement methods described above, the following method is used: In the case of the third fish weight measurement method, three-dimensional information of the fish can be obtained, so it is possible to manipulate the adjustment parameters. This section explains how to adjust K and p for the first and second fish weight measurement methods. Since factors such as the fish species and on-site feeding may have an effect, adjustments are made based on samples. A sample involves bringing multiple fish from the fish pen onto land and measuring their body length, height, and width (all or part of them) as well as their weight. This allows adjustments to the formula.

[0045] When using landscape orientation, the following process occurs: (1) Calculate the distance and direction of movement from the echo. (2) Calculate the size related to body length and body height from the image of camera V. (3) The weight of the fish is calculated using the Almetry equation for each fish species. In this case, the parameters of the Almetry equation are automatically adjusted, as in the first fish weight measurement method. (4) The automatic adjustment of withers height and obesity level can be created as learning data, and improved automatic adjustment can be performed for the next measurement. In other words, the accuracy of the adjustment parameters can be improved through learning.

[0046] The second fish weight measurement method will now be described in further detail. When using landscape orientation (1) Calculate distance, body width, and direction of movement from the echo. (2) Calculate the size related to body length and height from the camera image. (3) Fish weight is calculated using the Almetri equation for each fish species. In this case, the fatness level is calculated from the body height and body width, and the parameters of the Almetri equation are automatically adjusted based on the body width. (4) The automatic adjustment of body height, body width, and obesity level can be created as learning data, and improved automatic adjustment can be performed for the next measurement. In other words, the accuracy of the adjustment parameters can be improved through learning.

[0047] The third fish weight measurement method (measurement method based on 3D models) The third method for measuring fish weight uses images from camera V. When to use face-down placement (1) Calculate distance, height, and direction of movement from the echo. (2) Calculate the size related to body length and width from the camera image. (3) Calculate the volume using a 3D model for each fish species.

[0048] When using landscape orientation (1) Calculate distance, body width, and direction of movement from the echo. (2) Calculate the size related to body length and height from the camera image. (3) Calculate the volume using a 3D model for each fish species.

[0049] Common processing that is not dependent on the orientation of the layout (4) However, the swim bladder size is evaluated by evaluating the reflection level of the swim bladder from the echo, and the swim bladder volume is subtracted from the total volume.

[0050] An example of how to create a 3D shape model for each fish species is as follows. (1) Form a region in two dimensions along the body axis. (2) The cross section of the fish is represented by an ellipsoid (the shape of the ellipsoid itself is determined by the location of the body axis and the fish species). (3) Calculate the elliptical shape (2D shape) based on the body width (you can apply different formulas to the upper and lower halves). (4) Determine the two-dimensional shape along the body axis and create a 3D model. (5) Calculate the volume from the 3D model and calculate the fish weight based on the density of each fish species.

[0051] The first, second, and third fish weight measurement methods described above are selected, for example, by the user. However, the third fish weight measurement method, which creates a 3D shape model, requires calculating all of the body length, body height, and body width, which likely limits the number of fish weights that can be calculated, but provides a more accurate evaluation. Therefore, the third fish weight measurement method may be used for learning. Furthermore, if the fish weight cannot be measured using any of the above-mentioned fish weight measurement methods, the fish weight is predicted.

[0052] An example of the fish detection process using echoes performed by the fish detection unit 2 in the embodiment of the present invention will be described with reference to the flowchart of FIG. Step S1: The process starts. Step S2: A transmission signal is generated at a predetermined time interval, echoes are received, and peaks are detected from the measured echoes. Figure 10 shows an example of echo intensity (relative values ​​are shown on the vertical axis of the graph; the same applies below). In the example of Figure 10, the echo intensity is at its maximum when the distance m (horizontal axis of the graph) is around 0.8 m. The dotted line indicates the threshold for peak detection, and echo intensities above the threshold are detected as peaks. The threshold is set to an absolute value or a constant multiple of the overall average value. Furthermore, the distance at which the peak exists is determined.

[0053] Step S3: Echo continuity evaluation is performed. If a peak exists at approximately the same distance in the previous transmission signal, the peaks are linked as shown in Figure 11. In the graph in Figure 11, the vertical axis represents pings and the horizontal axis represents distance (m). Pings are transmitted at intervals of, for example, 1 / 20 (seconds) based on the transmission timing. If the estimated distance and the distance of the peak for the previous transmission are greater than a threshold, linking is terminated.

[0054] Figure 12 shows the echogram corresponding to the graph in Figure 11. The horizontal axis represents pings, and the vertical axis represents distance (m). The interval between pings is 0.05 seconds. In Figure 12, the echo intensity is displayed as a monochrome image, with darker colors indicating greater echo intensity.

[0055] Step S4: When the connection is complete, it is determined whether the detection criteria are met. That is, if the number of consecutive transmissions detected as a peak in the processing of step S2 is within a certain range (for example, 5 to 20 times) and the TS calculated at the maximum echo intensity is within a certain range, the fish is detected.

[0056] The ratio of the transmission strength of the ultrasound waves sent from the transducer S to the strength of the echo reflected by the fish (TS (Target Strength): echo strength 1 m from the transducer) is roughly expressed by the following formula: W is the body length of the fish, and A is a coefficient determined by the signal frequency and the fish species. TS=20logW+20logA

[0057] If the detection criteria are not met in step S4, the process returns to step S2. This process can remove noise because, while echoes from fish are measured continuously, noise is likely to be detected only once.

[0058] Step S5: If it is determined in step S4 that the detection criteria are met, the time of maximum echo intensity (echo time) in the link and the distance at that time are calculated.

[0059] Step S6: The change in distance at the time of the echo is calculated. The change in distance of the fish before and after the time when the strongest echo is emitted is detected. As shown in Figure 13A, when the fish swims downward, the fish moves away from the transducer S. As shown in Figure 13B, when the fish swims parallel to the transducer S, the distance between the fish and the transducer S does not change. Furthermore, as shown in Figure 13C, when the fish swims upward, the fish moves closer to the transducer S. In this way, the movement direction of the fish is detected from the change in distance. While obtaining the fish's position information from the camera V, the movement direction of the fish is calculated from the change in distance.

[0060] Step S7: Calculate body height or body width from the echo waveform. If the sensor is positioned face down, body height is calculated, and if it is positioned sideways, body width is calculated. In the case of a face down position, body height information is obtained from the duration for which the echo level is maintained above the threshold. In the case of a face down position, when an ultrasound wave is incident from the side of a fish, such as a tuna, two echoes are generated due to reflections from the body surface and swim bladder. Body width can be obtained by doubling the difference in distance between multiple echoes for which multiple reflections have occurred. For fish without swim bladder, body width can be obtained from the difference in distance between multiple echo reflections.

[0061] Step S8: Determine whether the image data from camera V and the echo can be matched. That is, if the image of the fish that generated the analyzed echo can be captured by camera V, it is determined that they can be matched. If it is determined that they can be matched, image analysis processing is performed. If it is determined that they cannot be matched, the process moves on to fish weight prediction processing.

[0062] If it is determined in step S8 that the image and the echo can be associated, the image analysis process shown in the flowchart of FIG. 6 is performed. Step S11: A fish in the vicinity of the direct front is detected from the image data at the echo time. Step S12: The center of gravity of the fish region is calculated.

[0063] Step S13: It is determined whether or not the fish region has been extracted. If it is determined that the fish region has not been extracted, the process returns to step S11 (detection of a fish in the vicinity of the front). Step S14: If it is determined in step S13 that the fish region has been extracted, the number of pixels in the region, as well as the number of pixels for the body length and body width (or body height) are calculated.

[0064] Step S15: The moving direction is calculated from the change in direction of the center of gravity and the change in distance around the echo time. Step S16: The area of ​​the area, body length and body height (or body width) are calculated (corrected) from the distance and movement direction, and then the process moves to fish weight analysis processing.

[0065] The processing of steps S11 to S14 in the image analysis process will be described. Figs. 14A to 14F show schematic images of a single fish F selected as a target in an image captured by camera V. The image of fish F was captured by camera V positioned facing downward. As shown in Fig. 14A, the size is calculated from the area of ​​fish F (the range from the tip of the head P indicated by a circle to the base of the caudal fin Q). For example, the image of fish F is approximated by a rectangle and the size is calculated as follows:

[0066] In Figure 14B, the dashed line represents the body length, and the maximum width of the fish indicated by the arrow is the body width W0. When the degree of curvature of the body axis indicated by the dashed line is large, it is difficult to accurately calculate the body length, so the following processing is performed.

[0067] As shown in Figure 14C, the position of the center of gravity C0 is calculated as a representative pixel from the region. The number of pixels from the tip of the head P to the position of the center of gravity C0 and from the position of the center of gravity C0 to the base of the caudal fin Q are calculated. An example of image processing to determine the coordinates of the center of gravity in the fish region is shown below. x coordinate of center of gravity = sum of (brightness value x x coordinate) / sum of brightness value Y coordinate of the center of gravity = (brightness value x y coordinate) sum / brightness value sum

[0068] As shown in FIG. 14D, the length of the shortest line segment as the width of the area at the position C0 of the center of gravity is calculated as the number of pixels of the body width W0 at that position of the center of gravity.

[0069] As shown in FIG. 14E, the positions of the centers of gravity C1 and C2 are calculated in each of the two regions separated by the line of the body width W0, and the body widths W1 and W2 at those positions are calculated.

[0070] As shown in FIG. 14F, the center of gravity positions C3, C4, C5, and C6 are calculated in the four regions divided by the lines of body width W0, body width W1, and body width W6 at those positions, and the body widths W3, W4, W5, and W6 are calculated.

[0071] As a result of the image analysis described above, the body length L and body width W of fish F are calculated as follows: Body length L of fish F: The sum of the line segments from the tip of the head P through the center of gravity to the base of the caudal fin Q Body width W of fish F: The maximum body width among the centers of gravity C0 to C6 (Body width among the centers of gravity C0 to C6: The number of pixels of the shortest line segment that fills the area passing through the centers of gravity C0 to C6) This method allows the body length and width of a fish to be determined with minimal error.

[0072] If the distance to the fish is known, the actual body length and width can be calculated from the number of pixels. In this way, in one embodiment of the present invention, accurate size (body length and width in the case of downward orientation, body length and height in the case of sideways orientation) can be calculated from the distance and movement direction from the image of a single camera.

[0073] With reference to Figure 15, we will explain how to calculate the movement direction of a fish when it is placed facing downwards. For example, if ultrasound is transmitted 20 times per second, echoes are measured at multiple times and an image is also taken by camera V. The time when the maximum value of the continuously measured echoes is reached is used as the representative value. Five ping echoes are detected for a given fish, and the distance for each ping is calculated. As shown in Figure 15A, the round-trip distance is found from the echo at a certain ping (time), and the distance r0 between the transducer S and the fish is measured.

[0074] As shown in Figure 15B, the fish area is calculated from the image data of camera V for each ping, and the pixel at the center of gravity is calculated. The pixel at the center of gravity at each time can be converted into an angle (θ, φ) in a polar coordinate system (3D coordinate system) from the camera based on the angle of view. The origin of this polar coordinate system is the position of camera V. Distance r can be found from distance r0.

[0075] As shown in Figure 15C, the three-dimensional coordinate position is calculated based on the distance d (meters) between the camera V and the transducer S. That is, the three-dimensional coordinate position is calculated to satisfy the distance r0 obtained from the echo and the direction of the camera V. If the angle from the camera V is (θ, φ) and the distance from the camera V to the fish is r, the three-dimensional position coordinate is expressed as follows:

[0076] x=r0sinθcosφ y=r0sinθsinφ z=r0cosθ

[0077] Calculate r that satisfies the distance r0 from the transducer S. If the distance between the camera V and the transducer S is d, then the distance r0 can be calculated by setting (x1=xd, y1=y, z1=z). If the distance is r0, then the distance r from the camera V to the center of gravity is determined.

[0078] As shown in Figure 15D, the movement direction is calculated from the change in three-dimensional position at different times. In Figure 15D, rt1 indicates the distance at time t1, and rt2 indicates the distance at time t2. The movement direction is calculated from the change in three-dimensional position at different times.

[0079] Accurate size measurements cannot be made using only the distance obtained from the echo. As mentioned above, the direction of movement can be accurately calculated using the distance between camera V and the fish and the distances between each part corrected for the direction of movement.

[0080] By knowing the distance to the object (fish), the size can be calculated from the area of ​​the image captured by camera V. When the fish is swimming downward as shown in FIG. 16A, the calculated size will differ compared to when the fish is swimming horizontally as shown in FIG. 16B. As described above, the direction of movement of the fish is calculated from the change in the calculated three-dimensional position, and correction is made. Note that when the fish is positioned facing downward, the body length and width are calculated from the image captured by camera V. For example, when a fish swims downward, the distance from camera V will vary depending on the part of the body. To calculate accurate actual size, it is necessary to calculate the actual size based on the distance of each part.

[0081] An example of size calculation will be further explained with reference to Figure 17. In Figure 17, the rectangular plane is a plane that passes through the fish's center of gravity and the direction of movement of the fish indicated by the arrow. It is assumed that the fish's direction of movement and its body axis coincide. A normal vector that intersects perpendicularly with the direction of movement is calculated, and a plane that satisfies this normal vector is created.

[0082] The intersections of the plane with the pixel positions of the center of gravity (including the divided centers of gravity), the tip of the head, and the base of the caudal fin are calculated. The distance from the tip of the head to the base of the caudal fin, passing through each individual center of gravity, is calculated from the coordinates to calculate body length. In the downward orientation, the body width is calculated from the distance at the position of the center of gravity of the fish and the pixel-to-pixel distance corresponding to the body width.

[0083] The seven points in Figure 17 correspond to the five center of gravity points C0, C3, C4, C5, and C6 shown in Figure 14F, point P at the tip of the head, and point Q at the base of the caudal fin. As shown in Figure 18, the three-dimensional position of each point is calculated based on the distance r from camera V and the inclination α based on the three-dimensional position and movement direction of the fish, and the distance between the points is calculated, and the fish length is calculated by adding these up.

[0084] Next, the case of horizontal placement will be described. In horizontal placement, the body height is calculated from the distance at the position of the fish's center of gravity and the inter-pixel distance corresponding to the body height. Figures 19A to 19F show schematic images of a single fish F narrowed down as the target in an image taken by camera V. The image of fish F was taken with camera V placed horizontally. As shown in Figure 19A, the size is calculated from the area of ​​fish F (the range from the tip of the head P shown by a circle to the base of the caudal fin Q). For example, the image of fish F is approximated by a rectangle and the size is calculated as follows:

[0085] In Figure 19B, the dashed line represents the body length, and the maximum body height of the fish indicated by the arrow is the body height H0. When the degree of curvature of the body axis indicated by the dashed line is large, it is difficult to accurately calculate the body height, so the following processing is performed.

[0086] As shown in Figure 19C, the position of the center of gravity D0 is calculated as a representative pixel from the region. The number of pixels from the tip of the head P to the position of the center of gravity D0 and from the position of the center of gravity D0 to the base of the caudal fin Q are calculated. The image processing for finding the coordinates of the center of gravity in the fish region can be the same as that for the downward placement described above.

[0087] As shown in FIG. 19D, the length of the shortest line segment as the width of the area at the position D0 of the center of gravity is calculated as the number of pixels of the body height H0 at that position of the center of gravity.

[0088] As shown in FIG. 19E, the positions of the centers of gravity D1 and D2 are calculated in each of the two regions separated by the line of the withers height H0, and the withers heights H1 and H2 at those positions are calculated.

[0089] As shown in FIG. 19F, the center of gravity positions D3, D4, D5, and D6 are calculated in the four regions divided by the lines of withers height H0, withers height H1, and withers height H6 at those positions are calculated.

[0090] As a result of the image analysis described above, the body length L and body height H of fish F are calculated as follows: Body length L of fish F: The sum of the line segments from the tip of the head P through the center of gravity to the base of the caudal fin Q Body height H of fish F: The maximum value of the body heights at the center of gravity positions D0 to D6 (body height at the center of gravity positions D0 to D6: the number of pixels of the shortest line segment that fills the area passing through the center of gravity positions D0 to D6) This method allows the body length and body height of a fish to be determined with minimal error.

[0091] With reference to Figures 20A, 20B, 20C, and 20D, we will explain how to calculate the movement direction of a fish when it is placed horizontally. Figures 20A to 20D and 21 are views looking downward from the sea surface. For example, if ultrasonic waves are transmitted 20 times per second, echoes are measured at multiple times and images are also taken by camera V. The time at which the maximum value of the continuously measured echoes is reached is used as the representative value. Five ping echoes are detected for a single fish, and the distance for each ping is calculated. As shown in Figure 20A, distance r0 is calculated from the round-trip distance from the echo at a certain ping (time). Distances are also measured from echoes at different times.

[0092] As shown in Figure 20B, the fish area is calculated from the image data of camera V at these times, and the center of gravity is calculated in the same way as in the downward-facing position. The pixel at the center of gravity position at each time can be converted into an angle (θ, φ) in a three-dimensional coordinate system with camera V as the origin based on the angle of view.

[0093] As shown in Figure 20C, the three-dimensional coordinate position is calculated based on the distance d (meters) between the camera V and the transducer S. That is, the three-dimensional coordinate position is calculated to satisfy the distance r0 obtained from the echo and the direction of the camera V. If the angle from the camera V is (θ, φ) and the distance from the camera V to the fish is r0, the three-dimensional position coordinate is expressed as follows:

[0094] x=r0sinθcosφ y=r0sinθsinφ z=r0cosθ

[0095] Calculate r that satisfies the distance r0 from the transducer S. If the distance between the camera V and the transducer S is d, then the distance r0 can be calculated by setting (x1=xd, y1=y, z1=z). If the distance is r0, then the distance r from the camera V to the center of gravity is determined.

[0096] As shown in Figure 20D, the direction of movement is calculated based on the change in the distances rt1 and rt2 between the centers of gravity for successive different pings. The direction of movement is either an approaching direction (closer to the camera V) or an away direction (farther away from the camera V).

[0097] Accurate size measurements cannot be made using only the distance obtained from the echo. As mentioned above, the direction of movement can be accurately calculated using the distance between camera V and the fish and the distances between each part corrected for the direction of movement.

[0098] By knowing the distance to the object (fish), the size is calculated from the area of ​​the image captured by camera V. When the fish is swimming away from the camera, as shown in FIG. 21A, the calculated size differs from when the fish is swimming parallel to the camera, as shown in FIG. 21B. As with the downward placement described above, the direction of movement of the fish is calculated from the calculated change in distance, and correction is made. When the camera is placed sideways, the body length and height are calculated.

[0099] The size calculation will be further explained with reference to Figure 22. In Figure 22, the rectangular plane is a plane that passes through the fish's center of gravity and the direction of movement of the fish indicated by the arrow. It is assumed that the fish's direction of movement and its body axis coincide. A normal vector that intersects perpendicularly with the direction of movement is calculated, and a plane that satisfies this normal vector is created.

[0100] The intersections of the plane with the pixel positions of the center of gravity (including the divided centers of gravity), the tip of the head, and the base of the caudal fin are calculated. The distance from the tip of the head to the base of the caudal fin, passing through each individual center of gravity, is calculated from the coordinates to calculate body length. When the fish is placed horizontally, its body height is calculated from the distance at the center of gravity and the pixel-to-pixel distance corresponding to its body height.

[0101] The seven points in Figure 21 correspond to the five center of gravity points D0, D3, D4, D5, and D6 shown in Figure 19F, point P at the tip of the head, and point Q at the base of the caudal fin. As shown in Figure 23, the three-dimensional position of each point is calculated based on the distance r from camera V and the inclination α based on the three-dimensional position and movement direction of the fish, and the distance between the points is calculated, and the fish length is calculated by adding these up.

[0102] Next, the fish weight analysis process is performed. Figure 7 is a flowchart showing a fish weight analysis method according to one embodiment of the present invention. It shows the processing flow of the first fish weight measurement method or the second fish weight measurement method using the body length-to-fish weight conversion formula described above.

[0103] Step S21: Fish weight is calculated based on Almetry's formula for each fish species. Almetry's formula is a formula for converting body length into fish weight, and the Almetry formula is calculated in advance for each fish species. Furthermore, the parameters of Almetry's formula are adjusted based on the calculated body width or body height. Step S22: The fish weight is calculated based on the Almetri's formula with adjusted parameters.

[0104] FIG. 8 shows the process flow of the third method for measuring fish weight described above. Step S31: The volume of the fish is calculated from the cross-sectional area, body height, and body width at each location. Step S32: The weight of the fish is calculated from the volume and density of the fish. The density is calculated using a value determined in advance for each fish species.

[0105] 9 is a block diagram for explaining an example of fish weight prediction performed in the prediction unit 3. The learning data creation unit 11 learns fish weight based on echoes as follows. (1) Fish detection using echoes is performed over the entire distance range. (2) The echo from the nearest object is used to calculate the fish weight in relation to the camera. (3) Based on all the fish weight data within the measured time, the relationship between the echo information and the measured fish weight is learned. The created learning data is stored in the learning data storage unit 12.

[0106] The echo information is the following information: It is not necessary to use all of this information, and it is sufficient to use only a part of it. TS value calculated from the peak value of the echo (echo level corrected for distance attenuation) Echo time pattern Echo ping-time pattern (2D values ​​of the time pattern per ping) Frequency characteristics obtained by Fourier transforming the time waveform of the echo Echo ping - frequency response

[0107] The echo information at the echo time is supplied to the fish weight prediction unit 13. The fish weight prediction unit 13 is supplied with learning data from the learning data storage unit 12, and outputs a predicted fish weight value. The fish weight prediction unit 13 predicts the fish weight for echo information that cannot be matched with the image from the camera V.

[0108] This fish weight prediction makes it possible to evaluate the weight of distant fish, which was previously difficult to measure from images taken by Camera V. With a camera, distant fish cannot be evaluated due to obstructions from the fish in front, and there was also the problem that the camera could not evaluate fish several meters away depending on the turbidity of the seawater.

[0109] An example of a method for creating a 3D shape model for each fish species will be described with reference to FIGS. 24, 25, 26 and 27. FIG. First, the region is represented two-dimensionally along the body axis as shown in Figure 24. Figure 24 is a graph showing an example of the change in body width (solid line) L1 and the change in body height (dashed line) L2 relative to the body axis.

[0110] Next, the cross section of the fish is represented by an ellipsoid. The shape of the ellipsoid itself is determined by the location of the body axis and the fish species. Figure 25 shows an example of a cross section of a fish at a position, for example, 4 cm from the tip of the head. Furthermore, the elliptical shape (2D shape) is calculated based on the body width. Separate formulas can be applied to the upper and lower halves.

[0111] The two-dimensional shape is determined along the body axis, and a 3D model is created. Figure 26 shows a 3D model of a fish expressed on the three-dimensional coordinates of body axis, body width, and body height. Calculate the volume from the 3D model and calculate the fish weight based on density.

[0112] FIG. 27 is a graph showing an example of the change in cross-sectional area calculated along the body axis. The volume is calculated by accumulating the cross-sectional areas. In the example of FIG. 27, volume information (e.g., 580 cm 2 The fish weight can be calculated by multiplying the volume by the density of the fish species in question.

[0113] By having the 3D shape of each fish species, it is possible to accurately estimate the fish's weight from the area (body length and body width) and body height (or area (body length and body height) and body width in the case of a sideways position) calculated by the camera.

[0114] The fish weight measurement method in the above-described embodiment can be recorded as a program on a semiconductor memory device or a recording medium such as a magnetic disk, a magneto-optical disk, a ROM, etc. Therefore, the functions of the fish weight measurement device can be realized by reading this semiconductor memory device or recording medium into a computer and executing it on an MPU (Micro Processing Unit), DSP (Digital Signal Processor), etc.

[0115] Although the embodiments of the present invention have been specifically described above, the present invention is not limited to the above-described embodiments, and various modifications based on the technical concept of the present invention are possible. The configurations, methods, processes, shapes, materials, and numerical values ​​given in the above-described embodiments are merely examples, and different configurations, methods, processes, shapes, materials, and numerical values ​​may be used as necessary. [Explanation of symbols]

[0116] S: ultrasonic transducer, V: camera, 3: prediction unit, 8: fish weight calculation unit

Claims

1. The imaging device and the ultrasonic transmitter / receiver have an imaging direction and an ultrasonic wave transmission direction that are the same direction, and the ultrasonic wave transmission timing and the imaging timing are synchronized, Using the echo obtained by the ultrasonic transducer, a distance between the imaging device and a reference point in a fish region in the image data of the imaging device is calculated; determining the body length of the fish based on the image of the fish in the image data of the imaging device and the distance; A fish weight measurement method for determining the weight of the fish using the body length, Calculating the center of gravity of a fish region in the image data; The pixel at the center of gravity, the pixel on the head side of the fish, and the pixel on the tail side of the fish are set as pixels at positions that define the body length of the fish, A fish weight measurement method in which the distance from the pixel on the head side to the pixel on the tail side, which passes through the pixel at the center of gravity position, is calculated from coordinates, and this distance is taken as the body length of the fish.

2. converting the reference point to a three-dimensional coordinate position with the imaging device as the origin; calculating a movement direction of the fish from the change in the three-dimensional coordinate position over time; correcting the reference point using the direction of movement of the fish, and calculating the length of the fish using the corrected reference point; Calculate fish weight using the body length The method for measuring fish weight according to claim 1.

3. Assuming that the movement direction of the fish coincides with the body axis of the fish, a plane is created that satisfies the movement direction passing through the three-dimensional coordinates of the reference point; The body length is calculated by calculating the intersection between the plane and the pixel at the position that defines the body length of the fish. The method for measuring fish weight according to claim 2.

4. The imaging direction and the transmission direction are downward, The shortest width of the widths of the fish region at the position of the center of gravity is calculated, and the shortest width is calculated as the body width at the position of the center of gravity of the fish region. The method for measuring fish weight according to claim 1.

5. repeating the process of dividing the fish region into two by the shortest width, calculating the center of gravity of each of the two divided regions, calculating the shortest width among the widths of the fish region at each of the center of gravity positions, and calculating the shortest width as the body width at each of the center of gravity positions; Calculate the maximum body width among the body widths at multiple center of gravity positions. The method for measuring fish weight according to claim 4.

6. The imaging direction and the transmission direction are set to landscape orientation, The shortest width of the widths of the fish region at the center of gravity position is calculated, and the shortest width is calculated as the body height at the center of gravity position of the fish region. The method for measuring fish weight according to claim 1.

7. repeating the process of dividing the fish area into two by the shortest line segment, calculating the center of gravity of each of the two divided areas, calculating the shortest width among the widths of the fish area at each of the center of gravity positions, and calculating the shortest width as the body height at each of the center of gravity positions; The maximum body height among the body heights at multiple center of gravity positions is calculated as the body height. The method for measuring fish weight according to claim 6.

8. An imaging device and an ultrasonic transmitter / receiver in which the imaging direction and the ultrasonic wave transmission direction are the same direction, and the ultrasonic wave transmission timing and the imaging timing are synchronized, Using the image data obtained by the imaging device and the echoes obtained by the ultrasonic transducer, the body length, body width, and body height of the fish are determined; determining the volume of the fish using the body length, the body width, and the body height; A fish weight measurement method for measuring fish weight from the volume and density of each fish species.

9. assessing the size of the swim bladder by assessing the level of swim bladder reflection from the echo; Subtract the volume of the swim bladder from the volume The method for measuring fish weight according to claim 8.

10. The imaging device and the ultrasonic transmitter / receiver have an imaging direction and an ultrasonic wave transmission direction that are the same direction, and the ultrasonic wave transmission timing and the imaging timing are synchronized, Using the echo obtained by the ultrasonic transducer, a distance between the imaging device and a reference point in a fish region in the image data of the imaging device is calculated; A fish weight measuring device comprising a processing device that calculates the body length of the fish based on the image of the fish in the image data of the imaging device and the distance, and calculates the weight of the fish using the body length, Calculating the center of gravity of a fish region in the image data; The pixel at the center of gravity, the pixel on the head side of the fish, and the pixel on the tail side of the fish are set as pixels at positions that define the body length of the fish, A fish weight measuring device that calculates the distance from the pixel on the head side to the pixel on the tail side, which passes through the pixel at the center of gravity position, from coordinates, and determines this distance as the body length of the fish.

Images