Method, program and system for determining actual fish size

Abstract

To provide an improved method for determining the actual size of a fish. [Solution] The method includes a fish shape recognition step in which a computer recognizes a fish body shape, which is approximated as a polygon with pentagons or more sides, in each of right and left images captured by a stereo camera, thereby recognizing the fish by recognizing the fish body shape; a gravity center determination step in which the computer determines the center of gravity of each of one or more recognized fish based on the fish body shape in the recognized image of the fish; a correspondence determination step in which the computer determines a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish based on the gravity center; and an actual size determination step in which the computer determines the actual size of the same fish based on the size of the same fish in at least one of the right and left images.

Description

[Technical field]

[0001] The present disclosure relates to determining actual size of fish. [Background technology]

[0002] In order to calculate the value of aquaculture pens, techniques have been developed to determine the actual size of the fish in the pens.

[0003] In relation to this, a fish size calculation device is known that can calculate the actual size of migrating fish without contacting the fish (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6694039 Summary of the Invention [Problem to be solved by the invention]

[0005] It is an object of the present disclosure to provide an improved method, etc., for determining the actual size of fish. [Means for solving the problem]

[0006] According to an embodiment of the present disclosure, there is provided a method including: a fish shape recognition step in which a computer recognizes a fish body shape that is approximated as a polygon with pentagons or more sides in each of a right image and a left image captured by a stereo camera, whereby a fish is recognized by recognizing the fish body shape; a gravity center determination step in which the computer determines a gravity center for each of one or more recognized fish in the recognized image of the fish based on the fish body shape; a correspondence determination step in which the computer determines a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish based on the gravity center; and an actual size determination step in which the computer determines an actual size of the same fish based on the size of the same fish in at least one of the right image and the left image.

[0007] In one embodiment, the method may include a step in which a computer determines, for at least one of the one or more recognized fish, a coordinate corresponding to the mouth among the coordinates constituting the fish body shape in a recognized image of the fish; a step in which a computer determines, in the recognized image of the fish, the longest line segment connecting the coordinate corresponding to the mouth and another coordinate among the coordinates constituting the fish body shape as a first line segment; and an on-image size determination step in which the computer determines the size of the fish in the recognized image of the fish based on the first line segment.

[0008] In one embodiment, the on-image size determination step may include a step in which the computer determines, for the fish, a second line segment in the recognized image of the fish that is perpendicular to the first line segment and connects two of the coordinates that constitute the fish body shape, and a step in which the computer determines the size of the fish in the recognized image of the fish based on the second line segment.

[0009] In one embodiment, the size of the fish in the recognized image of the fish may be the length of the first line segment.

[0010] In one embodiment, the size of the fish in the recognized image of the fish may be the length of the second line segment.

[0011] In one embodiment, the fish includes a fish having convex portions at both ends of a tail fin, and the fish body shape may be a fish body shape excluding the convex portions at both ends of the tail fin.

[0012] In one embodiment, the fish shape recognition step includes a step in which a computer recognizes the fish body shape in each of the right image and the left image, which is approximated as a polygon, excluding the convex portions at both ends of the tail fin, based on a machine learning model, and the machine learning model is trained based on a plurality of learning data, and the plurality of learning data may include, as input, images of a fish, and, as output, learning data including the fish body shape approximated as a polygon, excluding the convex portions at both ends of the tail fin.

[0013] In one embodiment, the plurality of learning data may include learning data including images of a first type of fish and learning data including images of a second type of fish different from the first type.

[0014] In one embodiment, all of the images of fish included in the plurality of learning data may be images of fish captured from a predetermined direction.

[0015] In one embodiment, the right image and the left image have a common coordinate system, and the conditions for determining that a fish recognized in the right image and a fish recognized in the left image correspond to the same fish may include a condition that the distance between the coordinates of the center of gravity in the right image for the fish recognized in the right image and the coordinates of the center of gravity in the left image for the fish recognized in the left image is less than or equal to a predetermined distance.

[0016] In one embodiment, the right image and the left image have a common coordinate system, and the conditions for determining that a fish recognized in the right image and a fish recognized in the left image correspond to the same fish may include a condition that the distance in a first direction between the coordinates of the center of gravity in the right image for the fish recognized in the right image and the coordinates of the center of gravity in the left image for the fish recognized in the left image is less than or equal to a first predetermined distance, and the distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance.

[0017] In one embodiment, the method further includes a step in which the computer determines, for each of the one or more recognized fish, an orientation of the fish in the recognized image, and the condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish may further include a condition that the orientation of the fish recognized in the right image in the right image and the orientation of the fish recognized in the left image in the left image are consistent.

[0018] In one embodiment, the size of the same fish in the at least one image may be the larger of the size of the same fish in the right image and the size of the same fish in the left image.

[0019] In one embodiment, the method may further include excluding fish whose centroids in the right image or the left image are outside a predetermined range from the fish recognized in that image.

[0020] In one embodiment, in the actual size determination step, the actual size of the same fish may be determined based on a disparity determined from the center of gravity in the right image and the left image of the same fish and also based on characteristics of the stereo camera.

[0021] According to an embodiment of the present disclosure, there is provided a program for causing a computer to execute a method including a fish shape recognition step of recognizing a fish body shape approximated as a polygon with pentagons or more sides in each of a right image and a left image captured by a stereo camera, whereby a fish is recognized by recognizing the fish body shape; a gravity center determination step of determining, for each of one or more recognized fish, a center of gravity in the recognized image of the fish based on the fish body shape; a correspondence determination step of determining, based on the gravity center, a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish; and an actual size determination step of determining an actual size of the same fish based on the size of the same fish in at least one of the right image and the left image.

[0022] According to an embodiment of the present disclosure, there is provided a system configured to execute the following steps: a fish body shape recognition step of recognizing a fish body shape approximated as a polygon with pentagons or more sides in each of a right image and a left image captured by a stereo camera, whereby a fish is recognized by recognizing the fish body shape; a gravity center determination step of determining, for each of one or more recognized fish, a center of gravity in the recognized image of the fish based on the fish body shape; a correspondence determination step of determining, based on the gravity center, a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish; and an actual size determination step of determining the actual size of the same fish based on the size of the same fish in at least one of the right image and the left image. Effect of the Invention

[0023] According to one embodiment of the present disclosure, the actual size of a fish can be determined based on images captured by a stereo camera. [Brief description of the drawings]

[0024] [Figure 1A] FIG. 1 is a block diagram of an example system 100A. [Figure 1B] FIG. 1 is a block diagram of an example system 100B. [Figure 1C] FIG. 1 is a block diagram of an example system 100C. [Diagram 2] 2 is a flow chart of an example process 200. [Diagram 3] 3 is a flow chart of an example process 300. [Figure 4] 4 is a flow chart of an example process 400. [Diagram 5] 5 is a flow chart of an example process 500. [Figure 6] 6 is a flow chart of an example process 600. [Figure 7A] FIG. 11 is a diagram for explaining a polygonal approximation of a fish body shape. [Figure 7B] FIG. 11 is a diagram for explaining a polygonal approximation of a fish body shape. [Figure 7C] FIG. 11 is a diagram for explaining a polygonal approximation of a fish body shape. [Figure 8] FIG. 13 is a diagram for explaining the distance between centers of gravity. [Figure 9A] FIG. 13 is a diagram for explaining a first line segment. [Figure 9B] FIG. 13 is a diagram for explaining a first line segment. [Figure 9C] FIG. 13 is a diagram for explaining a first line segment. [Figure 10] FIG. 13 is a diagram for explaining a second line segment. [Figure 11A] 1 is a diagram for explaining an image of a fish captured from a predetermined direction. FIG. [Figure 11B] 1 is a diagram for explaining an image of a fish captured from a predetermined direction. FIG. [Figure 12A] FIG. 13 is a diagram for explaining the shape of a fish body excluding the convex portions at both ends of the tail fin. [Figure 12B] FIG. 13 is a diagram for explaining the shape of a fish body excluding the convex portions at both ends of the tail fin. [Figure 13] FIG. 2 is a diagram for explaining the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] 1 System according to an embodiment of the present disclosure FIG. 1A is a block diagram of an example system 100A according to one embodiment of the present disclosure.

[0026] 110 denotes a fish. The system 100A is intended to at least measure the actual size of the fish 110. In this disclosure, measuring the actual size of the fish 110 includes determining the actual size of the fish 110.

[0027] Reference numeral 120 denotes a fish pen in which fish 110 swim. In Fig. 1A, two fish 110 are shown in the fish pen 120, but it should be noted that in reality, two or more fish may be present in the fish pen 120. Note that the fish pen 120 may be installed outdoors, such as in the sea or a lake, or may be installed indoors.

[0028] 130 indicates a stereo camera for capturing an image of the fish 110. The stereo camera 130 includes two cameras, which are arranged so that when capturing an image of an object such as the fish 110, a parallax occurs in the horizontal direction (x-axis direction) of the image between an image captured by one of the cameras and an image captured by the other. Hereinafter, an image captured by one of the two cameras is referred to as a right image, and an image captured by the other is referred to as a left image. The coordinate system of the right image and the left image may be common, but is not limited to this. The stereo camera 130 may capture the right image and the left image as still images, or may capture the right image and the left image as one frame of a moving image. The stereo camera 130 has certain known characteristics, which may include, for example, the focal length of the camera (or lens or optical system), the distance between the two cameras (or lenses or optical systems), the pixel pitch (mm / pixel) of the image sensor, or the size (mm) of one pixel of the image sensor, but are not limited to these.

[0029] Reference numeral 140 denotes a first device configured to acquire right and left images captured by the stereo camera 130 and upload them to a cloud 160 via a network 170. The first device 140 may be a computer configured with one or more computers.

[0030] 1A, the first device 140 is illustrated as being connected to the stereo camera 130 via a separate wire (which may be, but is not limited to, a USB cable), but the first device 140 and the stereo camera 130 may be connected via a network 170. The connection between the first device 140 and the stereo camera 130 may be wired or wireless.

[0031] The connection between the first device 140 and the network 170 may be wired or wireless.

[0032] Reference numeral 150 denotes a second device configured to obtain processing results from a cloud 160 via a network 170. The second device 150 may be a computer configured with one or more computers.

[0033] The processing results obtained from the cloud 160 may include the actual size of each of the one or more fish 110. Alternatively, the processing results obtained from the cloud 160 may include information derived from the actual size of each of the one or more fish 110 (for example, but not limited to, information derived from statistical processing of the actual size of each of the one or more fish 110).

[0034] The connection between the second device 150 and the network 170 may be wired or wireless.

[0035] Reference numeral 160 denotes a so-called cloud that processes the right and left images captured by the stereo camera 130. The cloud 160 may be a computer configured by one or more computers.

[0036] The connection between the cloud 160 and the network 170 may be wired or wireless.

[0037] Reference numeral 162 denotes an arithmetic execution unit. The arithmetic execution unit 162 is an abstraction of the arithmetic function that the cloud 160 has.

[0038] Reference numeral 164 denotes a database. The database 164 is an abstraction of the storage function provided in the cloud 160.

[0039] 166 indicates a trained model. The trained model 166 is an abstraction of a machine learning model trained based on multiple training data used by the cloud 160 for calculation. The machine learning model may be any machine learning model, for example, a model based on deep learning, but is not limited thereto. The substance of the trained model may be configured by a program that implements the machine learning model and is stored in the database 164, and one or more trained parameters that are used in the program.

[0040] A network is shown at 170. The network 170 may be composed of one or more networks including the Internet.

[0041] 1B is a block diagram of an exemplary system 100B according to an embodiment of the present disclosure. Differences from the exemplary system 100A will be described below.

[0042] 142 is The right and left images captured by the stereo camera 130 are acquired and uploaded to the cloud 160 via the network 170, and Acquires processing results from the cloud 160 via the network 170 The device 142 may be a computer configured with one or more computers. The device 142 corresponds to a combination of the first device 140 and the second device 150 in the exemplary system 100A.

[0043] FIG. 1C is a block diagram of an example system 100C according to one embodiment of the present disclosure. Differences from example system 100A will be described below.

[0044] A device 144 is configured to acquire and process the right and left images captured by the stereo camera 130 and obtain the results. The device 144 may be a computer configured with one or more computers. The device 144 corresponds to the integration of the first device 140, the second device 150, and the cloud 160 in the exemplary system 100A.

[0045] The system according to an embodiment of the present disclosure is not limited to the exemplary systems 100A to 100C.

[0046] 2. Processing according to an embodiment of the present disclosure 2 is a flowchart of an exemplary process 200 according to an embodiment of the present disclosure. Note that each step included in the exemplary process 200 may be executed by a computer. This computer may correspond to, but is not limited to, the first device 140, the second device 150, or the cloud 160 of the exemplary system 100A, the device 142 or the cloud 160 of the exemplary system 100B, or the device 144 of the exemplary system 100C.

[0047] Thus, one embodiment of the present disclosure may be a method including the exemplary process 200. Another embodiment of the present disclosure may be a program (a computer program, including a program stored in a computer-readable storage medium or a non-transitory computer-readable medium) that causes a computer to execute the method including the exemplary process 200, or a computer-readable storage medium or a non-transitory computer-readable medium storing the program. Yet another embodiment of the present disclosure may be a system (including one or more computers) configured to execute the method including the exemplary process 200. The same applies to the exemplary processes 300 to 600 described below.

[0048] Reference numeral 210 denotes a fish body shape recognition step for recognizing a fish body shape approximated to a polygon having pentagons or more sides in each of a right image and a left image captured by a stereo camera (e.g., stereo camera 130; the same applies below). In the present disclosure, a fish is recognized by recognizing the fish body shape.

[0049] It will be appreciated that, according to step 210, more than one fish may be recognized from an image.

[0050] The method of recognizing the fish body shape in the image is arbitrary, and for example, the fish body shape in the image may be recognized based on a machine learning model, but is not limited thereto. The machine learning model is trained based on a plurality of learning data, and the plurality of learning data may include, as an input, an image of a fish, and as an output, learning data including a polygonal approximation of the fish body shape, but is not limited thereto. Such a machine learning model may take an image as an input, and when a fish is captured in the image, output the fish body shape of the fish. In other words, such a machine learning model may not output the fish body shape of the fish when a fish is not captured in the input image. Note that the polygonal approximation of the fish body shape may be expressed as the coordinates of the vertices of a polygon in the image, but is not limited thereto.

[0051] The polygonal approximation of the fish body shape will be described below with reference to FIGS. 7A to 7C.

[0052] 710 shows an example image which may be a right or left image.

[0053] 720A-720C show the fish captured in the example image 710.

[0054] 730A to 730C show the shape of a fish approximated by a polygon having pentagons or more sides. The points in the fish shape 730 represent the coordinates of the vertices of the polygon, and the dashed lines connect the coordinates of the vertices.

[0055] The fish body shapes may not include a part of the fish. For example, the fish body shapes 730A to 730C do not include the dorsal fins or pelvic fins of the fish 720A to 720C. However, the fish body shapes 730A to 730C may include the dorsal fins or pelvic fins of the fish 720A to 720C. In addition, the fish body shape 730B includes the entire tail fin of the fish 720B, whereas the fish body shape 730C does not include a part of the tail fin of the fish 720C. In detail, the fish 720B and 720C have convex parts 740 at both ends of the tail fin, whereas the fish body shape 730C is a fish body shape that excludes the convex parts 740 at both ends of the tail fin.

[0056] Returning to FIG. 2, 220 illustrates a centroid determination step of determining, for each of one or more recognized fish, a centroid in the recognized image of the fish based on the fish body shape.

[0057] The center of gravity may be determined as coordinates in the image. The method of determining the center of gravity based on the fish body shape is arbitrary, and may be, for example, by averaging the coordinates of pixels in the image included inside the fish body shape, but is not limited thereto.

[0058] 230 shows a correspondence determination step for determining which fish recognized in the right image and which fish recognized in the left image correspond to the same fish, based on the centroids.

[0059] The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is arbitrary. The condition may include, for example, a condition that the distance between the coordinates of the center of gravity in the right image of the fish recognized in the right image and the coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a predetermined distance. The condition may also include, for example, a condition that the distance in a first direction between the coordinates of the center of gravity in the right image of the fish recognized in the right image and the coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a first predetermined distance, and the distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance. In any case, the condition is not limited to this. The first direction and the second direction may be one and the other of the horizontal direction (x-axis direction) and the vertical direction (y-axis direction) of the image, respectively, but are not limited to this.

[0060] The distance between the centers of gravity will be described below with reference to FIG.

[0061] Reference numeral 810 denotes one of the example right and left images, and reference numeral 815 denotes the other of the example right and left images. Note that the images 810 and 815 share a common coordinate system (for example, the coordinates of the four corners are the same in the images 810 and 815).

[0062] 820 shows the fish recognized in image 810 and 825 shows the fish recognized in image 815.

[0063] Reference numeral 830 indicates the coordinates of the center of gravity of the fish 820 in the image 810 , and reference numeral 835 indicates the coordinates of the center of gravity of the fish 825 in the image 815 .

[0064] Reference numeral 840 indicates the coordinates of the center of gravity of the fish 820 in the image 810 in the image 815 .

[0065] 850 indicates the distance between the coordinates 830 (840) of the center of gravity in image 810 for a fish 820 recognized in image 810 and the coordinates 835 of the center of gravity in image 815 for a fish 825 recognized in image 815.

[0066] 860 indicates an example first direction and 865 indicates an example second direction that is perpendicular to the first direction 860 .

[0067] 870 is the distance in a first direction between the coordinates 830 (840) of the center of gravity in image 810 for fish 820 recognized in image 810 and the coordinates 835 of the center of gravity in image 815 for fish 825 recognized in image 815, and 875 indicates the distance in a second direction 865 perpendicular to the first direction 860 between the coordinates 830 (840) of the center of gravity in image 810 for fish 820 recognized in image 810 and the coordinates 835 of the center of gravity in image 815 for fish 825 recognized in image 815.

[0068] Furthermore, the method of determining the predetermined distance (including the first predetermined distance and the second predetermined distance; the same applies hereinafter in this paragraph) used in this condition is arbitrary. For example, the predetermined distance may be determined based on the characteristics of the stereo camera, the expected range of the distance from the stereo camera to the fish, the expected size of the fish, the expected range of the magnitude of the parallax, and the like. In particular, when the first direction (or the second direction) is the horizontal direction of the image, it is preferable that the first predetermined distance (or the second predetermined distance) is determined based at least on the expected range of the magnitude of the parallax (the possible range of the parallax), and the like. Furthermore, the optimum value of the predetermined distance may be determined by experiment. In any case, the method of determining the predetermined distance is not limited to this.

[0069] The reason why distance 850 or both distances 870 and 875 are used to determine whether they correspond to the same fish is as follows.

[0070] As described above, the two cameras of the stereo camera 130 are arranged so that when an object such as the fish 110 is captured, a parallax basically occurs in the horizontal direction (x-axis direction) of the right and left images. This means that ideally, only a shift in the horizontal direction (x-axis direction) of the image occurs in the field of view of the two cameras. However, as a unique finding of the inventor of the present disclosure, it has been found that in practice, particularly for a stereo camera that captures images underwater, a shift in the vertical direction (y-axis direction) of the image may also occur in the field of view of the two cameras. In such a case, it is difficult to determine whether the right and left images correspond to the same fish using only the vertical positional relationship of the fish 110 in the right and left images.

[0071] Even under the assumption that there is no misalignment in the vertical direction (y-axis direction) of the images in the fields of view of the two cameras, multiple fish 110 may be recognized in the right or left image, and the vertical positions of these multiple fish 110 may match in the images. Even in such cases, it is difficult to determine whether the fish 110 in the right and left images correspond to the same fish using only the vertical positional relationship between the fish 110 in the right and left images.

[0072] Returning to FIG. 2, 240 illustrates an actual size determination step of determining the actual size of the same fish based on the size in the right and / or left image of the same fish.

[0073] The size of the fish in the image may be, but is not limited to, the total length or fork length of the fish on the image.

[0074] The size of the same fish in at least one of the right and left images may be, for example, the size of the same fish in one of the right and left images. The size may be the larger of the size of the same fish in the right image and the size of the same fish in the left image. The size may be the smaller of the size of the same fish in the right image and the size of the same fish in the left image. The size may be the average of the size of the same fish in the right image and the size of the same fish in the left image. In any case, the size is not limited to this.

[0075] The method of determining the real size of the same fish is arbitrary, and for example, the real size of the same fish may be determined using the principle of a known stereo camera, but is not limited to this. When the principle of a known stereo camera is used, the real size of the same fish will be determined based on the parallax determined from the center of gravity in the right image and the left image of the same fish, and also on the characteristics of the stereo camera. That is, according to the principle of a known stereo camera, The size of the same fish in at least one of the right and left images; Disparity determined from the centroids in the right and left images of the same fish; Characteristics of stereo cameras and The real size of the same fish can be determined based on the above. For example, the distance between the same fish and the stereo camera is determined based on the parallax determined from the center of gravity in the right and left images of the same fish and the characteristics of the stereo camera (particularly, the focal length of the camera (, lens or optical system) and the distance between the two cameras (, lens or optical system)), the correspondence (mm / pixel) between one pixel of the imaging element and the real size at the distance is determined based on the distance and the characteristics of the stereo camera (particularly, the focal length of the camera (, lens or optical system) and the size of one pixel of the imaging element), and the real size of the same fish can be determined based on the correspondence and the size (number of pixels in the image) of the same fish in at least one of the right and left images, but the method of determining the real size of the same fish is not limited to this. Note that the parallax determined from the center of gravity in the right and left images of the same fish may be the difference in the horizontal direction (x-axis direction) of the coordinates of the center of gravity in the right and left images (corresponding to the distance 875 in FIG. 8), but is not limited to this.

[0076] The exemplary process 200 may include, for example, a step (not shown) after step 220 and before step 230, of excluding fish whose centers of gravity in the right or left image are outside a predetermined range from the fish recognized in that image. Fish recognized near the edge of an image are undesirable for purposes of determining the actual size of the fish due to various reasons such as image distortion near the edge. By such a step, such undesirable fish can be excluded from the recognized fish.

[0077] 3 is a flow chart of an example process 300 that may be further included in the example process 200 (and in the example process 600 described below). The example process 300 may be performed after step 210 and before step 240 of the example process 200, but is not limited to this.

[0078] 310 shows a step of determining, for at least one of the one or more fish recognized in step 210 (hereinafter referred to as “the fish”), coordinates that correspond to the mouth of the fish among the coordinates that constitute the fish body shape in the recognized image of the fish.

[0079] The coordinates constituting the fish body shape may be, for example, when the fish body shape is represented as the coordinates of the vertices of a polygon in the image, or may include the coordinates of the vertices and the coordinates of pixels in the image through which the line segments connecting adjacent vertices pass. In any case, the coordinates constituting the polygon shape are not limited to this.

[0080] Furthermore, the method of determining the coordinates corresponding to the mouth among the coordinates constituting the fish body shape is arbitrary. For example, one of the coordinates of the part where the fish body shape is convex may be determined as the coordinates corresponding to the mouth. Furthermore, for example, the coordinates corresponding to the mouth may be determined based on a machine learning model. Such a machine learning model may be trained based on a plurality of training data, and the training data may include training data including the coordinates constituting the fish body shape as input and the coordinates corresponding to the mouth as output. In any case, the method of determining the coordinates corresponding to the mouth among the coordinates constituting the fish body shape is not limited to this.

[0081] Numeral 320 indicates a step of determining, as a first line segment, a line segment that connects a coordinate corresponding to the mouth and another coordinate among the coordinates that configure the shape of the fish body in the recognized image of the fish, and that is the longest line segment.

[0082] The first line segment will be described below with reference to FIGS. 9A to 9C.

[0083] 910 shows an example image which may be a right or left image.

[0084] 920A-920C show the fish captured in the example image 910.

[0085] Reference numeral 930 denotes a fish body shape approximated by a polygon having pentagons or more.

[0086] 940 shows the coordinates corresponding to the mouth.

[0087] 950A to 950C indicate the longest line segments (first line segments) connecting the coordinates corresponding to the mouth and other coordinates among the coordinates constituting the fish body shape. The lengths of the first line segments 950A and 950C correspond to the total lengths of the fish 920A and 920C, and the first line segment 950B corresponds to the fork length of the fish 920B.

[0088] Reference numeral 960 denotes a line segment that connects the coordinates corresponding to the mouth and other coordinates among those constituting the shape of the fish body, but is not the longest line segment.

[0089] As is apparent, the first line segment 950A and the first line segment 950C are not parallel, i.e., the example process 300 allows the determination of the first line segment (whose length, as described above, may be the total length or fork length of the fish) regardless of the inclination of the fish in a plane parallel to the recognized image of the fish.

[0090] 3, step 330 may include determining the length of the first line segment as the size of the fish in the recognized image of the fish. Thus, the total length of fish 920A and 920C according to first line segments 950A and 950C, and the fork length of fish 920B according to first line segment 940B may be determined as the size of the fish in the recognized image of the fish.

[0091] Step 330 may also include determining another line segment based on the first line segment to determine a size of the fish in the recognized image.

[0092] FIG. 4 is a flow chart of an example process 400 that the on-image size determination step 330 may include.

[0093] Numeral 410 indicates a step of determining, as a second line segment for the fish, the longest line segment that is perpendicular to the first line segment determined in step 320 in the recognized image of the fish and connects two of the coordinates that constitute the fish body shape.

[0094] The second line segment will be described below with reference to FIG.

[0095] 1010 shows an example image which may be a right or left image.

[0096] 1020 shows the fish imaged in example image 1010 .

[0097] Reference numeral 1030 denotes a fish body shape approximated by a polygon having pentagons or more sides.

[0098] 1040 indicates the first line segment.

[0099] Reference numeral 1050 denotes the longest line segment (second line segment) that is perpendicular to the first line segment 1040 and connects two coordinates among those that form the shape of the fish body. The length of the second line segment 1050 corresponds to the body height of the fish 920.

[0100] Reference numeral 1060 denotes a line segment that is perpendicular to the first line segment 1040 and connects two coordinates among those that form the shape of the fish, but is not the longest line segment.

[0101] Returning to FIG. 4, 420 shows determining the size of the same fish in at least one image based on the second line segment.

[0102] Step 420 may include, for example, determining the length of the second line segment as the size of the fish in at least one image. Thus, the height of the fish 1020 according to the second line segment 1050 may be determined as the size of the fish in at least one image.

[0103] FIG. 5 is a flow chart of an example process 500 that may be included in the fish shape recognition step 210.

[0104] 510 shows a step of recognizing a polygonal approximation of a fish body shape including the entire tail fin in each of the right image and the left image based on a first machine learning model.

[0105] The first machine learning model is trained based on a first plurality of training data. The first plurality of training data may include, for example, training data including an image of a fish as an input and a polygonal approximation of a fish body shape including the entire tail fin as an output, but is not limited thereto.

[0106] The plurality of first learning data (and the plurality of third learning data described later) may include learning data including images of a first type of fish and learning data including image data of a second type of fish different from the first type. With such learning data, it will be possible to recognize the fish body shapes from images of a plurality of types of fish.

[0107] Furthermore, all of the images of fish contained in the plurality of first learning data (and the plurality of third learning data described below) may be images of fish captured from a predetermined direction.

[0108] This will be described below with reference to Figures 11A and 11B. 1110 shows an example image, which is a right or left image. 1120A and 1120B show fishes captured in the example image 1110. The fishes 1120A and 1120B are captured from different directions. It can be seen that the fish 1120B is tilted towards the depth of the image compared to the fish 1120A.

[0109] By making all images of fish included in the multiple learning data images of the fish from a predetermined direction, it is possible to prevent fish 1120B from being recognized by setting the predetermined direction to the direction in which fish 1120A is imaged. While the total length and fork length of the fish can be used to indicate the size of the fish, it will be understood that the total length and fork length of fish 1120A can be determined more accurately than those of fish 1120B. In other words, the predetermined direction in which the fish is imaged may be a direction in which an image can be captured in which the size of the fish in the image can be determined more accurately.

[0110] Returning to FIG. 5, 520 indicates a step of recognizing a fish body shape, in which the convex portions at both ends of the tail fin are excluded from the fish body shape including the entire tail fin, based on a second machine learning model.

[0111] This will now be described with reference to Figures 12A and 12B.

[0112] 1210 shows an example image which may be a right or left image.

[0113] Reference numeral 1220 denotes a fish captured in the sample image 1210. The fish 1220 has convex portions 1225 on both ends of its tail fin.

[0114] Reference numeral 1230 denotes a fish body shape approximated by a polygon having pentagons or more sides. However, Fig. 12A shows a fish body shape 1230 including the entire tail fin, while Fig. 12B shows a fish body shape 1230 excluding the convex parts at both ends of the tail fin.

[0115] 1240 indicates the coordinates corresponding to the mouth.

[0116] Reference numeral 1250 denotes the longest line segment that connects the coordinates corresponding to the mouth and other coordinates among those constituting the shape of the fish body.

[0117] 1260 indicates a line segment corresponding to the fork length of the fish 1220.

[0118] As is clear, in Fig. 12B, the line segment 1250 and the line segment 1260 are the same, whereas in Fig. 12A, the line segment 1250 and the line segment 1260 are different. This is because, in the case of a fish 1220 having convex portions 1225 at both ends of the tail fin, the distance from the mouth to both ends of the tail fin may be longer than the fork length of the fish 1220.

[0119] That is, the exemplary process 500 enables the fork length of a fish having convex portions 1225 on both ends of its tail fin to be derived as the size in the image by the exemplary process 300.

[0120] The second machine learning model is trained based on a second plurality of training data. The second plurality of training data may include, for example, a polygonal approximation of a fish body shape including the entire tail fin as an input, and a polygonal approximation of a fish body shape excluding the convex parts at both ends of the tail fin as an output, but is not limited thereto.

[0121] Needless to say, in the fish shape recognition step 210, the polygonal approximation of the fish shape, excluding the convex parts at both ends of the tail fin, may be directly recognized based on a machine learning model in each of the right and left images captured by the stereo camera, instead of using the exemplary process 500. Such a machine learning model (hereinafter referred to as a "third machine learning model") is trained based on a plurality of third learning data, and the plurality of third learning data may include, for example, an image of a fish as an input, and learning data including a polygonal approximation of the fish shape, excluding the convex parts at both ends of the tail fin, as an output, but is not limited thereto.

[0122] In addition, both the first machine learning model (which recognizes the shape of the fish body including the entire tail fin) and the third machine learning model (which recognizes the shape of the fish body excluding the convex parts at both ends of the tail fin) may be prepared and used according to the type of fish assumed. For example, the third machine learning model may be used for fish types that have convex parts at both ends of the tail fin (for example, fish types whose tail fins are bifurcated, crescent-shaped, or indented, but are not limited thereto), and the first machine learning model may be used for fish types that do not have such convex parts.

[0123] The computer may determine the type of fish captured in the image, and selectively use the first machine learning model and the third machine learning model based on the determined type of fish. The method of determining the type of fish captured in the image is arbitrary, and for example, the type of fish captured in the image may be determined based on a machine learning model. Such a machine learning model is trained based on multiple learning data, and the learning data may include an image of a fish as an input, and learning data including the type of fish as an output. In any case, the method of determining the type of fish captured in the image is not limited to this.

[0124] Alternatively, the computer may determine whether the fish captured in the image has convex parts at both ends of the tail fin, and selectively use the first machine learning model when it determines that the fish does not have convex parts at both ends of the tail fin, and the third machine learning model when it determines that the fish has convex parts at both ends of the tail fin. The method of determining whether the fish captured in the image has convex parts at both ends of the tail fin is arbitrary, and for example, the method of determining whether the fish captured in the image has convex parts at both ends of the tail fin based on a machine learning model may be used. Such a machine learning model is trained based on multiple learning data, and the learning data may include, as input, an image of the fish captured, and as output, learning data including whether the fish has convex parts at both ends of the tail fin. In any case, the method of determining whether the fish captured in the image has convex parts at both ends of the tail fin is not limited to this.

[0125] The fish targeted by an embodiment of the present disclosure may include fish having convex portions at both ends of the tail fin. It will be understood that the fish body shape recognized in step 210 may be a fish body shape excluding the convex portions at both ends of the tail fin according to the exemplary process 500 and the above-described process.

[0126] 6 is a flowchart of an example process 600 according to an embodiment of the present disclosure. Each step included in the example process 600 may be executed by a computer. The example process 600 can be used as a substitute for the example process 200, and the same steps as those in the example process 200 are denoted by the same reference numerals.

[0127] 610 illustrates the step of determining, for each of one or more fish recognized in step 210, an orientation of the fish in the recognized image.

[0128] The method for determining the orientation of the fish is arbitrary.

[0129] For example, the exemplary process 300 may be further executed to determine the orientation of the fish from the positional relationship between the coordinates of the center of gravity of the fish determined in step 220 and the coordinates corresponding to the mouth of the fish determined in step 310. For example, a vector pointing from the coordinates of the center of gravity of the fish to the coordinates corresponding to the mouth of the fish may be regarded as the orientation of the fish. Also, for example, the lateral (x-axis) component of the vector of the recognized image of the fish may be regarded as the orientation of the fish.

[0130] Also, for example, example process 300 may be performed to determine an orientation for the fish from a relationship between the center of gravity for the fish determined in step 220 and the first line segment for the fish determined in step 330 .

[0131] In any case, the method for determining the orientation of a fish is not limited to this.

[0132] 620 illustrates a correspondence determination step for determining which fish recognized in the right image and which fish recognized in the left image correspond to the same fish, based on the centroids.

[0133] The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is arbitrary. The condition can be, for example, The distance between the coordinates of the center of gravity in the right image of the fish recognized in the right image and the coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a predetermined distance, and The orientation of the fish recognized in the right image in the right image coincides with the orientation of the fish recognized in the left image in the left image. The condition may include, for example, A distance in a first direction between the coordinates of the center of gravity in the right image of a fish recognized in the right image and the coordinates of the center of gravity in the left image of a fish recognized in the left image is less than or equal to a first predetermined direction, and a distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance, and The orientation of the fish recognized in the right image in the right image coincides with the orientation of the fish recognized in the left image in the left image. In any case, the condition is not limited to this. The first direction and the second direction may be the horizontal direction (or x-axis direction) and the vertical direction (or y-axis direction) of the image, but are not limited to this.

[0134] 3. Computer An example of a hardware configuration of a computer that can be used to implement an embodiment of the present disclosure will be described below.

[0135] 13 shows an example of a computer hardware configuration. As shown in the figure, the computer 1300 mainly includes, as hardware resources, a processor 1310, a main memory device 1320, an auxiliary memory device 1330, an input / output interface 1340, and a communication interface 1350, which are connected to each other via a bus line 1360 including an address bus, a data bus, a control bus, and the like. Note that an interface circuit (not shown) may be interposed between the bus line 1360 and each hardware resource as appropriate. Note that some of these components (e.g., the communication interface 1350) may not be included in the computer 1300.

[0136] The processor 1310 controls the entire computer or at least a part of the computer, such as a CPU or a microprocessor. Note that one computer may include multiple processors 1310. In such a case, the "processor" in the above description may be a general term for the multiple processors 1310.

[0137] The main memory 1320 is a volatile memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM) that provides a working area for the processor 1310.

[0138] The auxiliary storage device 1330 is a non-volatile memory such as an HDD, SSD, or flash memory that stores software such as programs and data. The programs and data are loaded from the auxiliary storage device 1330 to the main storage device 1320 via a bus line 1360 at any time. The auxiliary storage device 1330 may be called a computer-readable storage medium, a non-transitory computer-readable medium, or a computer-readable storage medium. The programs include instructions that cause the processor to execute desired processes.

[0139] The input / output interface 1340 presents information and / or receives information input, and may be a digital camera, a LiDAR sensor, a keyboard, a mouse, a display, a touch panel display, a microphone, a speaker, various sensors, etc.

[0140] The communication interface 1350 is connected to a network 1370 consisting of one or more of the Internet, a local area network (LAN), etc., and transmits and receives data via the network 1370. The communication interface 1350 and the network 1370 may be connected by wire or wirelessly. The communication interface 1350 may also obtain information related to the network, such as information related to Wi-Fi access points and information related to the base stations of a communication carrier.

[0141] It will be apparent to those skilled in the art that the above-exemplified hardware resources and software can cooperate to cause the computer 1300 to function as desired means, execute desired steps, and achieve desired functions.

[0142] 4. Conclusion Although the embodiments of the present disclosure have been described above, it goes without saying that the present disclosure is not limited to the above-described embodiments and may be embodied in various different forms within the scope of the technical concept thereof.

[0143] Moreover, the scope of the present disclosure is not limited to the exemplary embodiments shown and described, but includes all embodiments that achieve equivalent effects to those intended by the present disclosure. Furthermore, the scope of the present disclosure is not limited to the combination of inventive features defined by each claim, but may be defined by any desired combination of specific features among all the respective disclosed features. [Explanation of symbols]

[0144] 100A~100C...Example system 110, 720A~720C, 820, 825, 920A~920C, 1020, 1120A, 1120B, 1220...Fish 120…Fish tank 130…Stereo camera 160…Cloud 162…Database 170, 1370…Network 200~600...Example processing 710, 810, 815, 910, 1010, 1110, 1210...Images 730A~730C, 930, 1030, 1230...Fish body shape approximated by a polygon with pentagons or more 830, 835, 840...center of gravity 850…Distance between centers of gravity 860…First direction 865…Second direction 870…Distance between centers of gravity in the first direction 875…Second-direction distance between centers of gravity 940, 1240...coordinates corresponding to the mouth 950A~950B, 1040...First line segment 960...Line segment that is not the first line segment 1050…Second line segment 1060...A line segment that is not a second line segment 1250...The longest line segment connecting the coordinate corresponding to the mouth and another coordinate among the coordinates that make up the fish body shape 1260...A line segment equivalent to the length of a fish's fork

Claims

1. a fish body shape recognition step in which a computer recognizes a fish body shape approximated by a polygon having pentagons or more sides in each of a right image and a left image captured by a stereo camera, and a fish is recognized by recognizing the fish body shape; a center of gravity determination step in which the computer determines a center of gravity of each of the recognized one or more fish in the recognized image of the fish based on the fish body shape; a correspondence determination step in which a computer determines whether a fish recognized in the right image and a fish recognized in the left image correspond to the same fish based on the center of gravity; a real size determining step in which the computer determines a real size of the same fish based on a size of the same fish in at least one of the right image and the left image; A method comprising: the right and left images have a common coordinate system; The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is: The distance between the coordinates of the center of gravity in the right image for the fish recognized in the right image and the coordinates of the center of gravity in the left image for the fish recognized in the left image is less than or equal to a predetermined distance. With the condition that, a distance in a first direction between coordinates of the center of gravity in the right image of the fish recognized in the right image and coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a first predetermined distance, and a distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance. With the condition that including one of method.

2. 2. The method of claim 1 , A step in which the computer determines, for at least one of the one or more recognized fish, coordinates corresponding to a mouth among coordinates constituting the fish body shape in the recognized image of the fish; A step in which the computer determines, as a first line segment, a line segment that connects a coordinate corresponding to the mouth and another coordinate among the coordinates that configure the fish body shape in the recognized image of the fish; a step of determining a size of the fish in the recognized image based on the first line segment by the computer; A method comprising:

3. 3. The method according to claim 2, wherein the on-image size determination step comprises: a step in which the computer determines, for the fish, a second line segment that is a longest line segment that is perpendicular to the first line segment and connects two coordinates that constitute the fish body shape in the recognized image of the fish; the computer determines a size of the fish in the recognized image based on the second line segment; A method comprising:

4. The method of claim 2 , wherein the size of the fish in the recognized image of the fish is a length of the first line segment.

5. The method of claim 3 , wherein the size of the fish in the recognized image of the fish is a length of the second line segment.

6. 2. The method of claim 1 , The fish includes a fish having convex portions on both ends of a tail fin, The fish body shape is a fish body shape excluding the convex parts at both ends of the tail fin. method.

7. 7. The method of claim 6, The fish body shape recognition step includes: A step in which the computer recognizes, based on a machine learning model, a polygonal approximation of the fish body shape in each of the right image and the left image, excluding the convex portions at both ends of the tail fin. Including, The machine learning model is trained based on a plurality of training data, and the plurality of training data includes: As input, an image of a fish is The output is a polygonal approximation of the fish body shape, excluding the convex parts at both ends of the tail fin. Including training data, method.

8. 8. The method according to claim 7, wherein the plurality of learning data include learning data including images of a first type of fish and learning data including images of a second type of fish different from the first type.

9. The method according to claim 7 , wherein all of the images of fish included in the plurality of learning data are images of fish captured from a predetermined direction.

10. 2. The method of claim 1 , determining, for each of the one or more recognized fish, an orientation of the fish in the recognized image; Further comprising: The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is The orientation of the fish recognized in the right image in the right image coincides with the orientation of the fish recognized in the left image in the left image. The further condition includes: method.

11. The method of claim 1 , wherein the size of the same fish in the at least one image is the larger of the size of the same fish in the right image and the size of the same fish in the left image.

12. The method according to claim 1 , further comprising the step of excluding fish whose center of gravity coordinates in the right image or the left image are outside a predetermined range from the fish recognized in that image.

13. 2. The method according to claim 1 , wherein in the actual size determination step, the actual size of the same fish is determined based on a disparity determined from the center of gravity in the right and left images of the same fish and also on characteristics of the stereo camera.

14. a fish body shape recognition step of recognizing a fish body shape approximated by a polygon having 5 or more sides in each of a right image and a left image captured by a stereo camera, and recognizing the fish body shape to thereby recognize the fish; a center of gravity determination step of determining a center of gravity of each of the recognized one or more fish in the recognized image of the fish based on the fish body shape; a correspondence determination step of determining, based on the center of gravity, a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish; an actual size determining step of determining an actual size of the same fish based on a size of the same fish in at least one of the right image and the left image; A program for causing a computer to execute a method including the steps of: the right and left images have a common coordinate system; The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is: The distance between the coordinates of the center of gravity in the right image for the fish recognized in the right image and the coordinates of the center of gravity in the left image for the fish recognized in the left image is less than or equal to a predetermined distance. With the condition that, a distance in a first direction between coordinates of the center of gravity in the right image of the fish recognized in the right image and coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a first predetermined distance, and a distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance. With the condition that including one of program.

15. a fish body shape recognition step of recognizing a fish body shape approximated by a polygon having 5 or more sides in each of a right image and a left image captured by a stereo camera, and recognizing the fish body shape to thereby recognize the fish; a center of gravity determination step of determining a center of gravity of each of the recognized one or more fish in the recognized image of the fish based on the fish body shape; a correspondence determination step of determining, based on the center of gravity, a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish; an actual size determining step of determining an actual size of the same fish based on a size of the same fish in at least one of the right image and the left image; A system configured to execute the right and left images have a common coordinate system; The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is: The distance between the coordinates of the center of gravity in the right image for the fish recognized in the right image and the coordinates of the center of gravity in the left image for the fish recognized in the left image is less than or equal to a predetermined distance. With the condition that, a distance in a first direction between coordinates of the center of gravity in the right image of the fish recognized in the right image and coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a first predetermined distance, and a distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance. With the condition that including one of system.

Images