Actual size determination method for fish and program and system

Abstract

To provide an improved method, etc. to determine the actual size of a fish.SOLUTION: A method includes a fish body shape recognition step where a computer recognizes a fish body shape approximated by a polygon of five or more sides in each of the right image and left image captured by the stereo camera, where a fish is recognized when the fish body shape is recognized, a center of gravity determination step where a computer determines the center of gravity in the recognized image of each of one or more recognized fish based on the fish body shape, a correspondence determination step where the computer determines, based on the center of gravity, the fish recognized in the right image and the fish recognized in the left image corresponding to the same fish, and an actual size determination step where the computer determines the actual size of the same fish based on the size in at least one of the right image and the left image for the same fish.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to determining the actual size of fish. [Background technology]

[0002] In order to calculate the value of aquaculture pens, techniques have been developed to determine the actual size of the fish in the pens.

[0003] In this regard, a fish size calculation device is known that can calculate the actual size of migrating fish without contacting the fish (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6694039 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present disclosure is to provide an improved method for determining the actual size of a fish. [Means for solving the problem]

[0006] According to an embodiment of the present disclosure, there is provided a method including: a fish body shape recognition step in which a computer recognizes a fish body shape approximated by a polygon with pentagons or more sides in each of a right image and a left image captured by a stereo camera, wherein the fish is recognized by recognizing the fish body shape; a gravity center determination step in which the computer determines, for each of one or more recognized fish, a gravity center in the recognized image of the fish based on the fish body shape; a correspondence determination step in which the computer determines, based on the gravity center, a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish; and an actual size determination step in which the computer determines the actual size of the same fish based on the size of the same fish in at least one of the right image and the left image.

[0007] In one embodiment, the method may include a step in which the computer determines, for at least one of the one or more recognized fish, a coordinate corresponding to the mouth among the coordinates constituting the fish body shape in the recognized image of the fish; a step in which the computer determines, as a first line segment, the longest line segment connecting the coordinate corresponding to the mouth and another coordinate among the coordinates constituting the fish body shape in the recognized image of the fish; and an on-image size determination step in which the computer determines the size of the fish in the recognized image of the fish based on the first line segment.

[0008] In one embodiment, the size determination step on the image may include a step in which the computer determines, for the fish, as a second line segment in the recognized image of the fish, the longest line segment that is perpendicular to the first line segment and connects two of the coordinates that make up the fish body shape, and a step in which the computer determines the size of the fish in the recognized image of the fish based on the second line segment.

[0009] In one embodiment, the size of the fish in the recognized image of the fish may be the length of the first line segment.

[0010] In one embodiment, the size of the fish in the recognized image of the fish may be the length of the second line segment.

[0011] In one embodiment, the fish includes a fish having convex portions at both ends of a tail fin, and the fish body shape may be a fish body shape excluding the convex portions at both ends of the tail fin.

[0012] In one embodiment, the fish shape recognition step includes a step in which a computer recognizes the fish body shape in each of the right image and the left image, which is approximated as a polygon and in which the convex portions at both ends of the tail fin are excluded, based on a machine learning model, and the machine learning model is trained based on a plurality of training data, and the plurality of training data may include, as input, images of a fish, and, as output, training data including the fish body shape approximated as a polygon and in which the convex portions at both ends of the tail fin are excluded.

[0013] In one embodiment, the plurality of learning data may include learning data including images of a first type of fish and learning data including images of a second type of fish different from the first type.

[0014] In one embodiment, all of the images of fish included in the plurality of learning data may be images of fish captured from a predetermined direction.

[0015] In one embodiment, the right image and the left image have a common coordinate system, and the conditions for determining that a fish recognized in the right image and a fish recognized in the left image correspond to the same fish may include a condition that the distance between the coordinates of the center of gravity in the right image for the fish recognized in the right image and the coordinates of the center of gravity in the left image for the fish recognized in the left image is less than or equal to a predetermined distance.

[0016] In one embodiment, the right image and the left image have a common coordinate system, and the conditions for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish may include the condition that the distance in a first direction between the coordinates of the center of gravity in the right image for the fish recognized in the right image and the coordinates of the center of gravity in the left image for the fish recognized in the left image is less than or equal to a first predetermined distance, and the distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance.

[0017] In one embodiment, the method further includes a step in which the computer determines, for each of the one or more recognized fish, the orientation of the fish in the recognized image, and the condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish may further include a condition that the orientation in the right image of the fish recognized in the right image matches the orientation in the left image of the fish recognized in the left image.

[0018] In one embodiment, the size of the same fish in the at least one image may be the larger of the size of the same fish in the right image and the size of the same fish in the left image.

[0019] In one embodiment, the method may further include a step of excluding fish whose centroids in the right image or the left image are outside a predetermined range from the fish recognized in the image.

[0020] In one embodiment, in the actual size determination step, the actual size of the same fish may be determined based on the parallax determined from the center of gravity in the right image and the left image of the same fish and also based on the characteristics of the stereo camera.

[0021] According to an embodiment of the present disclosure, there is provided a program for causing a computer to execute a method including: a fish body shape recognition step for recognizing a fish body shape approximated by a polygon with pentagons or more sides in each of a right image and a left image captured by a stereo camera, wherein the fish is recognized by recognizing the fish body shape; a gravity center determination step for determining a gravity center for each of one or more recognized fish based on the fish body shape in the recognized image of the fish; a correspondence determination step for determining a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish based on the gravity center; and an actual size determination step for determining the actual size of the same fish based on the size of the same fish in at least one of the right image and the left image.

[0022] According to an embodiment of the present disclosure, there is provided a system configured to execute the following steps: a fish body shape recognition step for recognizing a fish body shape approximated by a polygon with pentagons or more sides in each of a right image and a left image captured by a stereo camera, wherein the fish is recognized by recognizing the fish body shape; a gravity center determination step for determining a gravity center for each of one or more recognized fish based on the fish body shape in the recognized image of the fish; a correspondence determination step for determining, based on the gravity center, a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish; and an actual size determination step for determining the actual size of the same fish based on the size of the same fish in at least one of the right image and the left image. [Effects of the Invention]

[0023] According to one embodiment of the present disclosure, the actual size of a fish can be determined based on images captured by a stereo camera. [Brief explanation of the drawings]

[0024] [Figure 1A] FIG. 1 is a block diagram of an example system 100A. [Figure 1B] FIG. 1 is a block diagram of an example system 100B. [Figure 1C] FIG. 1 is a block diagram of an example system 100C. [Figure 2] 2 is a flowchart of an example process 200. [Figure 3] 3 is a flowchart of an example process 300. [Figure 4] 4 is a flowchart of an example process 400. [Figure 5] 5 is a flowchart of an example process 500. [Figure 6] 6 is a flow chart of an example process 600. [Figure 7A] FIG. 10 is a diagram for explaining a polygonal approximation of a fish body shape. [Figure 7B] FIG. 10 is a diagram for explaining a polygonal approximation of a fish body shape. [Figure 7C] FIG. 10 is a diagram for explaining a polygonal approximation of a fish body shape. [Figure 8] FIG. 10 is a diagram for explaining the distance between centers of gravity. [Figure 9A] FIG. 10 is a diagram illustrating a first line segment. [Figure 9B] FIG. 10 is a diagram illustrating a first line segment. [Figure 9C] FIG. 10 is a diagram illustrating a first line segment. [Figure 10] FIG. 10 is a diagram illustrating a second line segment. [Figure 11A] FIG. 10 is a diagram for explaining an image of a fish captured from a predetermined direction. [Figure 11B] FIG. 10 is a diagram for explaining an image of a fish captured from a predetermined direction. [Figure 12A] FIG. 10 is a diagram for explaining the shape of a fish body excluding the convex portions at both ends of the tail fin. [Figure 12B] FIG. 10 is a diagram for explaining the shape of a fish body excluding the convex portions at both ends of the tail fin. [Figure 13] FIG. 2 is a diagram illustrating the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0025] 1 System according to an embodiment of the present disclosure FIG. 1A is a block diagram of an example system 100A according to one embodiment of the present disclosure.

[0026] 110 denotes a fish. System 100A is intended to at least measure the actual size of fish 110. Note that, in this disclosure, measuring the actual size of fish 110 includes determining the actual size of fish 110.

[0027] Reference numeral 120 denotes a fish tank in which fish 110 are swimming. In Fig. 1A, two fish 110 are shown in the fish tank 120, but it should be noted that in reality, there may be more than two fish in the fish tank 120. The fish tank 120 may be installed outdoors, such as in the ocean or a lake, or may be installed indoors.

[0028] Reference numeral 130 denotes a stereo camera for capturing images of the fish 110. The stereo camera 130 includes two cameras, which are arranged so that when capturing an image of an object such as the fish 110, a parallax occurs in the horizontal direction (x-axis direction) between an image captured by one of the cameras and an image captured by the other. Hereinafter, the image captured by one of the two cameras will be referred to as a right image, and the image captured by the other camera will be referred to as a left image. The right and left images may share a common coordinate system, but this is not a limitation. The stereo camera 130 may capture the right and left images as still images, or may capture the right and left images as one frame of a moving image. The stereo camera 130 has certain known characteristics, which may include, but are not limited to, the focal length of the camera (or lenses or optical systems), the distance between the two cameras (or lenses or optical systems), the pixel pitch (mm / pixel) of the image sensor, or the size (mm) of one pixel of the image sensor.

[0029] Reference numeral 140 denotes a first device configured to acquire right and left images captured by the stereo camera 130 and upload them to the cloud 160 via a network 170. The first device 140 may be a computer configured with one or more computers.

[0030] 1A, the first device 140 is illustrated as being connected to the stereo camera 130 via a separate wire (which may be, but is not limited to, a USB cable), but the first device 140 and the stereo camera 130 may be connected via a network 170. The connection between the first device 140 and the stereo camera 130 may be wired or wireless.

[0031] The connection between the first device 140 and the network 170 may be wired or wireless.

[0032] Reference numeral 150 denotes a second device configured to obtain processing results from the cloud 160 via a network 170. The second device 150 may be a computer configured with one or more computers.

[0033] The processing results obtained from the cloud 160 may include the actual size of each of the one or more fish 110. Alternatively, the processing results obtained from the cloud 160 may include information derived from the actual size of each of the one or more fish 110 (for example, but not limited to, statistical processing of the actual size of each of the one or more fish 110).

[0034] The connection between the second device 150 and the network 170 may be wired or wireless.

[0035] Reference numeral 160 denotes a so-called cloud that processes the right and left images captured by the stereo camera 130. The cloud 160 may be a computer configured by one or more computers.

[0036] The connection between the cloud 160 and the network 170 may be wired or wireless.

[0037] Reference numeral 162 denotes a calculation execution unit. The calculation execution unit 162 is an abstraction of the calculation function that the cloud 160 has.

[0038] A database 164 is an abstraction of the storage function of the cloud 160.

[0039] Reference numeral 166 denotes a trained model. The trained model 166 is an abstraction of a machine learning model that is trained based on multiple pieces of training data and that is used for calculations by the cloud 160. The machine learning model may be any machine learning model, for example, a model based on deep learning, but is not limited to this. The entity of the trained model may be configured by a program that implements the machine learning model and is stored in the database 164, and one or more trained parameters that are used in the program.

[0040] A network is shown at 170. The network 170 may consist of one or more networks, including the Internet.

[0041] 1B is a block diagram of an exemplary system 100B according to an embodiment of the present disclosure. Differences from the exemplary system 100A will be described below.

[0042] 142 is The right and left images captured by the stereo camera 130 are acquired and uploaded to the cloud 160 via the network 170, and Acquires processing results from the cloud 160 via the network 170 The device 142 may be a computer configured with one or more computers. The device 142 corresponds to a combination of the first device 140 and the second device 150 in the exemplary system 100A.

[0043] FIG. 1C is a block diagram of an exemplary system 100C according to one embodiment of the present disclosure. Differences from the exemplary system 100A will be described below.

[0044] A device 144 is configured to acquire and process the right and left images captured by the stereo camera 130 and obtain the results. The device 144 may be a computer configured with one or more computers. The device 144 corresponds to the integration of the first device 140, the second device 150, and the cloud 160 in the exemplary system 100A.

[0045] Systems according to an embodiment of the present disclosure are not limited to the exemplary systems 100A to 100C.

[0046] 2. Processing according to an embodiment of the present disclosure 2 is a flowchart of an exemplary process 200 according to an embodiment of the present disclosure. Note that each step included in the exemplary process 200 may be performed by a computer. This computer may correspond to, but is not limited to, the first device 140, the second device 150, or the cloud 160 in the exemplary system 100A, the device 142 or the cloud 160 in the exemplary system 100B, or the device 144 in the exemplary system 100C.

[0047] Accordingly, one embodiment of the present disclosure may be a method including the exemplary process 200. Another embodiment of the present disclosure may be a program (a computer program, including a program stored in a computer-readable storage medium or a non-transitory computer-readable medium) that causes a computer to execute a method including the exemplary process 200, or a computer-readable storage medium or a non-transitory computer-readable medium that stores the program. Yet another embodiment of the present disclosure may be a system (including one or more computers) configured to execute a method including the exemplary process 200. The same applies to the exemplary processes 300 to 600 described below.

[0048] Reference numeral 210 denotes a fish body shape recognition step for recognizing a fish body shape approximated by a polygon having five or more sides in each of a right image and a left image captured by a stereo camera (for example, stereo camera 130; the same applies below). In the present disclosure, a fish is recognized by recognizing the fish body shape.

[0049] It will be appreciated that, according to step 210, more than one fish may be recognized from an image.

[0050] Any method for recognizing the shape of a fish in an image may be used. For example, the method may recognize the shape of a fish in an image based on a machine learning model, but is not limited to this. The machine learning model is trained based on multiple training data, and the multiple training data may include, as input, images of a fish and, as output, training data including a polygonal approximation of the shape of a fish, but is not limited to this. Such a machine learning model may take an image as input and, if a fish is included in the image, output the shape of the fish. In other words, such a machine learning model may not output the shape of the fish if the input image does not include a fish. The polygonal approximation of the shape of a fish may be represented by the coordinates of the vertices of a polygon in the image, but is not limited to this.

[0051] The polygonal approximation of the fish body shape will be described below with reference to FIGS. 7A to 7C.

[0052] 710 shows an example image which may be a right or left image.

[0053] 720A-720C show the fish captured in the example image 710.

[0054] 730A to 730C show the shape of the fish approximated by a polygon with five or more sides. The points in the fish shape 730 are the coordinates of the vertices of the polygon, and the dashed lines are lines connecting the coordinates of the vertices.

[0055] A fish body shape may not include a portion of the fish. For example, fish body shapes 730A to 730C do not include the dorsal or pelvic fins of fish 720A to 720C. However, fish body shapes 730A to 730C may include the dorsal or pelvic fins of fish 720A to 720C. Furthermore, fish body shape 730B includes the entire tail fin of fish 720B, while fish body shape 730C does not include a portion of the tail fin of fish 720C. In particular, fish 720B and 720C have convex portions 740 at both ends of their tail fins, while fish body shape 730C has a fish body shape that excludes the convex portions 740 at both ends of their tail fins.

[0056] Returning to FIG. 2, 220 indicates a centroid determination step of determining, for each of the one or more recognized fish, a centroid in the recognized image of the fish based on the shape of the fish body.

[0057] The center of gravity may be determined as coordinates in the image. The method for determining the center of gravity based on the shape of the fish body is arbitrary, and may be, for example, determined by averaging the coordinates of pixels in the image that are included inside the shape of the fish body, but is not limited to this.

[0058] 230 shows a correspondence determination step for determining whether a fish recognized in the right image and a fish recognized in the left image correspond to the same fish based on the center of gravity.

[0059] The conditions for determining that a fish recognized in the right image and a fish recognized in the left image correspond to the same fish are arbitrary. The conditions may include, for example, a condition that the distance between the coordinates of the center of gravity in the right image of the fish recognized in the right image and the coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a predetermined distance. The conditions may also include, for example, a condition that the distance in a first direction between the coordinates of the center of gravity in the right image of the fish recognized in the right image and the coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a first predetermined distance, and that the distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance. In any case, the conditions are not limited thereto. The first and second directions may be, respectively, one and the other of the horizontal direction (x-axis direction) and vertical direction (y-axis direction) of the image, but are not limited thereto.

[0060] The distance between the centers of gravity will be described below with reference to FIG.

[0061] Reference numeral 810 denotes one of the example right and left images, and reference numeral 815 denotes the other of the example right and left images. It is assumed that the images 810 and 815 share a common coordinate system (for example, the coordinates of the four corners of the images 810 and 815 are the same).

[0062] 820 shows the fish recognized in image 810 and 825 shows the fish recognized in image 815.

[0063] 830 indicates the coordinates of the center of gravity of the fish 820 in the image 810, and 835 indicates the coordinates of the center of gravity of the fish 825 in the image 815.

[0064] 840 shows the coordinates of the center of gravity of the fish 820 in the image 810 in the image 815.

[0065] 850 indicates the distance between the coordinates 830 (840) of the center of gravity in image 810 for fish 820 recognized in image 810 and the coordinates 835 of the center of gravity in image 815 for fish 825 recognized in image 815.

[0066] 860 indicates an example first direction, and 865 indicates an example second direction that is perpendicular to the first direction 860.

[0067] 870 is the distance in a first direction between the coordinate 830 (840) of the center of gravity in image 810 for fish 820 recognized in image 810 and the coordinate 835 of the center of gravity in image 815 for fish 825 recognized in image 815, and 875 indicates the distance in a second direction 865 perpendicular to the first direction 860 between the coordinate 830 (840) of the center of gravity in image 810 for fish 820 recognized in image 810 and the coordinate 835 of the center of gravity in image 815 for fish 825 recognized in image 815.

[0068] Furthermore, the predetermined distance (including the first predetermined distance and the second predetermined distance; the same applies hereinafter in this paragraph) used in this condition may be determined by any method. For example, the predetermined distance may be determined based on the characteristics of the stereo camera, the expected range of distances from the stereo camera to the fish, the expected size of the fish, the expected range of the magnitude of the parallax, etc. In particular, when the first direction (or the second direction) is the horizontal direction of the image, it is preferable that the first predetermined distance (or the second predetermined distance) be determined based at least on the expected range of the magnitude of the parallax (the range of possible parallax), etc. Furthermore, the optimum value of the predetermined distance may be determined by experiment. In any case, the method of determining the predetermined distance is not limited to this.

[0069] The reason for using distance 850 or both distances 870 and 875 to determine whether they correspond to the same fish is as follows.

[0070] As described above, the two cameras included in the stereo camera 130 are arranged so that when an object such as a fish 110 is captured, parallax basically occurs in the horizontal direction (x-axis direction) between the right and left images. This means that ideally, only a misalignment in the horizontal direction (x-axis direction) occurs between the fields of view of the two cameras. However, based on the unique knowledge of the inventors of the present disclosure, it has been found that in practice, particularly with stereo cameras capturing images underwater, a misalignment in the vertical direction (y-axis direction) of the images may also occur between the fields of view of the two cameras. In such cases, it is difficult to determine whether the right and left images correspond to the same fish using only the vertical positional relationship of the fish 110 in the right and left images.

[0071] Furthermore, even under the assumption that there is no misalignment in the vertical direction (y-axis direction) of the images in the fields of view of the two cameras, multiple fish 110 may be recognized in the right or left image, and the vertical positions of these multiple fish 110 in the images may match. Even in such cases, it is difficult to determine whether the fish 110 in the right and left images correspond to the same fish using only the relationship between their vertical positions.

[0072] Returning to FIG. 2, 240 shows an actual size determination step for determining the actual size of the same fish based on the size in at least one of the right and left images of the same fish.

[0073] The size of the fish in the image may be, but is not limited to, the total length or fork length of the fish in the image.

[0074] The size of the same fish in at least one of the right and left images may be, for example, the size of the same fish in one of the right and left images. The size may also be the larger of the size of the same fish in the right image and the size of the same fish in the left image. The size may also be the smaller of the size of the same fish in the right image and the size of the same fish in the left image. The size may also be the average of the size of the same fish in the right image and the size of the same fish in the left image. In any case, the size is not limited to this.

[0075] The method for determining the actual size of the same fish is arbitrary, and for example, the actual size of the same fish may be determined using the principle of a known stereo camera, but is not limited to this. When the principle of a known stereo camera is used, the actual size of the same fish will be determined based on the parallax determined from the center of gravity of the right image and the left image of the same fish, and also on the characteristics of the stereo camera. That is, according to the principle of a known stereo camera, The size of the same fish in at least one of the right and left images; disparities determined from the centroids in the right and left images of the same fish; Characteristics of stereo cameras and The actual size of the same fish can be determined based on the above. For example, the distance between the same fish and the stereo camera may be determined based on the disparity determined from the center of gravity in the right and left images of the same fish and the characteristics of the stereo camera (particularly, the focal length of the camera (lens or optical system) and the distance between the two cameras (lens or optical system)), the correspondence relationship (mm / pixel) between one pixel of the image sensor and the actual size at that distance may be determined based on the distance and the characteristics of the stereo camera (particularly, the focal length of the camera (lens or optical system) and the size of one pixel of the image sensor), and the actual size of the same fish may be determined based on the correspondence relationship and the size (number of pixels in the image) of the same fish in at least one of the right and left images, but the method for determining the actual size of the same fish is not limited to this. Note that the disparity determined from the center of gravity in the right and left images of the same fish may be, but is not limited to, the difference in the horizontal direction (x-axis direction) of the coordinates of the center of gravity in the right and left images (corresponding to distance 875 in FIG. 8 ).

[0076] The exemplary process 200 may include, for example, a step (not shown) after step 220 and before step 230, of excluding fish whose centers of gravity in the right or left image are outside a predetermined range from the fish recognized in that image. Fish recognized near the edges of an image are undesirable for purposes of determining the actual size of the fish due to various reasons, such as image distortion near the edges. This step allows such undesirable fish to be excluded from the recognized fish.

[0077] 3 is a flowchart of an example process 300 that may be further included in example process 200 (and example process 600, described below). Example process 300 may be performed after step 210 and before step 240 of example process 200, but is not limited to this.

[0078] 310 shows a step of determining, for at least one of the one or more fish recognized in step 210 (hereinafter referred to as "the fish"), the coordinates that constitute the fish body shape in the recognized image of the fish, corresponding to the mouth of the fish.

[0079] For example, if the fish shape is represented as the coordinates of the vertices of a polygon in the image, the coordinates that make up the fish shape may be the coordinates of the vertices, or may include the coordinates of the vertices and the coordinates of pixels in the image through which the line segments connecting adjacent vertices pass. In any case, the coordinates that make up the polygon are not limited to this.

[0080] Furthermore, any method can be used to determine the coordinates corresponding to the mouth among the coordinates constituting the fish body shape. For example, one of the coordinates of the convex part of the fish body shape may be determined as the coordinates corresponding to the mouth. Furthermore, for example, the coordinates corresponding to the mouth may be determined based on a machine learning model. Such a machine learning model is trained based on a plurality of training data, and the training data may include training data containing, as input, coordinates constituting the fish body shape and, as output, coordinates corresponding to the mouth. In any case, the method for determining the coordinates corresponding to the mouth among the coordinates constituting the fish body shape is not limited to this.

[0081] 320 indicates a step of determining, as a first line segment, the longest line segment connecting the coordinate corresponding to the mouth and another coordinate among the coordinates constituting the shape of the fish in the recognized image of the fish.

[0082] The first line segment will be described below with reference to FIGS. 9A to 9C.

[0083] 910 shows an example image which may be a right or left image.

[0084] 920A to 920C show the fish captured in the sample image 910.

[0085] 930 shows the shape of the fish approximated by a polygon having five or more sides.

[0086] 940 indicates the coordinates corresponding to the mouth.

[0087] Reference numerals 950A to 950C indicate the longest line segments (first line segments) connecting the coordinates corresponding to the mouth with other coordinates constituting the fish body shape. The lengths of the first line segments 950A and 950C correspond to the total lengths of the fish 920A and 920C, and the length of the first line segment 950B corresponds to the fork length of the fish 920B.

[0088] Reference numeral 960 denotes a line segment that connects the coordinate corresponding to the mouth and another coordinate among the coordinates that make up the shape of the fish body, but is not the longest line segment.

[0089] As is apparent, first line segment 950A and first line segment 950C are not parallel. That is, exemplary process 300 allows for the determination of the first line segment (whose length, as described above, may be the total length or fork length of the fish) regardless of the inclination of the fish on a plane parallel to the recognized image of the fish.

[0090] 3, step 330 may include determining the length of the first line segment as the size of the fish in the recognized image of the fish. Thus, the total length of fish 920A and 920C may be determined according to first line segments 950A and 950C, and the fork length of fish 920B may be determined according to first line segment 940B.

[0091] Step 330 may also include determining another line segment based on the first line segment to determine the size of the fish in the recognized image.

[0092] FIG. 4 is a flow chart of an example process 400 that the on-image size determination step 330 may include.

[0093] 410 shows a step of determining, as a second line segment for the fish, the longest line segment that is perpendicular to the first line segment determined in step 320 and connects two of the coordinates that make up the fish body shape in the recognized image of the fish.

[0094] The second line segment will be described below with reference to FIG.

[0095] 1010 shows an example image which may be a right or left image.

[0096] 1020 shows the fish that is imaged in the example image 1010.

[0097] Reference numeral 1030 indicates the shape of the fish approximated by a polygon having five or more sides.

[0098] 1040 indicates the first line segment.

[0099] Reference numeral 1050 denotes the longest line segment (second line segment) that is perpendicular to the first line segment 1040 and connects two of the coordinates that make up the shape of the fish body. The length of the second line segment 1050 corresponds to the body height of the fish 920.

[0100] Reference numeral 1060 denotes a line segment that is perpendicular to the first line segment 1040 and connects two of the coordinates that make up the shape of the fish, but is not the longest line segment.

[0101] Returning to FIG. 4, 420 shows the step of determining the size of the same fish in at least one image based on the second line segment.

[0102] Step 420 may include, for example, determining the length of the second line segment as the size of the fish in at least one image. Thus, the height of the fish 1020 according to the second line segment 1050 may be determined as the size of the fish in at least one image.

[0103] FIG. 5 is a flow chart of an example process 500 that the fish shape recognition step 210 may include.

[0104] 510 indicates a step of recognizing a polygonal approximation of a fish shape, including the entire tail fin, in each of the right and left images based on a first machine learning model.

[0105] The first machine learning model is trained based on a first plurality of training data, which may include, for example, training data including an image of a fish as input and a polygonal approximation of the fish body shape including the entire tail fin as output, but is not limited thereto.

[0106] The plurality of first learning data (and the plurality of third learning data described later) may include learning data containing images of a first type of fish and learning data containing image data of a second type of fish different from the first type. Such learning data will enable recognition of the body shapes of a plurality of types of fish from images.

[0107] Furthermore, all of the images of fish included in the plurality of first learning data (and the plurality of third learning data described later) may be images of fish captured from a predetermined direction.

[0108] This will be explained below with reference to Figures 11A and 11B. 1110 shows an example image, which may be a right or left image. 1120A and 1120B show fishes imaged in example image 1110. Fishes 1120A and 1120B are imaged from different directions. It can be seen that fish 1120B is tilted towards the depth of the image compared to fish 1120A.

[0109] By setting all images of fish included in multiple learning data to images of the fish captured from a predetermined direction, it is possible to prevent fish 1120B from being recognized by setting the predetermined direction to the direction in which fish 1120A is captured. While the total length and fork length of the fish can be used to determine the size of the fish, it will be understood that the total length and fork length of fish 1120A can be determined more accurately than those of fish 1120B. In other words, the predetermined direction for capturing images of the fish may be a direction in which an image can be captured in which the size of the fish in the image can be determined more accurately.

[0110] Returning to FIG. 5, 520 indicates a step of recognizing a fish body shape, in which the convex portions at both ends of the tail fin are excluded from the fish body shape including the entire tail fin, based on the second machine learning model.

[0111] This will be explained below with reference to Figures 12A and 12B.

[0112] 1210 shows an example image which may be a right or left image.

[0113] Reference numeral 1220 denotes a fish captured in the sample image 1210. The fish 1220 has convex portions 1225 on both ends of its tail fin.

[0114] Reference numeral 1230 denotes a fish body shape approximated by a polygon having five or more sides. However, Fig. 12A shows the fish body shape 1230 including the entire tail fin, while Fig. 12B shows the fish body shape 1230 excluding the convex portions at both ends of the tail fin.

[0115] 1240 indicates the coordinates corresponding to the mouth.

[0116] Reference numeral 1250 denotes the longest line segment that connects the coordinate corresponding to the mouth and another coordinate among the coordinates that make up the shape of the fish body.

[0117] 1260 indicates a line segment corresponding to the fork length of the fish 1220.

[0118] As is clear, in Figure 12B, line segment 1250 and line segment 1260 are the same, while in Figure 12A, line segment 1250 and line segment 1260 are different. This is because, in the case of fish 1220 having convex portions 1225 at both ends of the tail fin, the distance from the mouth to both ends of the tail fin may be longer than the fork length of fish 1220.

[0119] That is, the exemplary process 500 enables the fork length of a fish having convex portions 1225 on both ends of its tail fin to be derived as the size in the image by the exemplary process 300.

[0120] The second machine learning model is trained based on a second plurality of training data. The second plurality of training data may include, for example, a polygonal approximation of a fish body shape including the entire tail fin as input, and a polygonal approximation of a fish body shape excluding the convex portions at both ends of the tail fin as output, but is not limited thereto.

[0121] It goes without saying that in fish shape recognition step 210, the polygonal approximation of the fish shape, excluding the convex portions at both ends of the tail fin, may be directly recognized based on a machine learning model in each of the right and left images captured by the stereo camera, rather than using exemplary process 500. Such a machine learning model (hereinafter referred to as a "third machine learning model") is trained based on multiple third training data, and the multiple third training data may include, for example, training data including captured images of a fish as input and a polygonal approximation of the fish shape, excluding the convex portions at both ends of the tail fin, as output, but is not limited to this.

[0122] In addition, both the first machine learning model (which recognizes the shape of the fish body including the entire tail fin) and the third machine learning model (which recognizes the shape of the fish body excluding the convex portions at both ends of the tail fin) may be prepared and used depending on the type of fish being considered. For example, the third machine learning model may be used for fish types that have convex portions at both ends of the tail fin (for example, fish types with forked, crescent, or indented tail fins, but are not limited to these), and the first machine learning model may be used for fish types that do not.

[0123] The computer may determine the type of fish captured in the image and selectively use the first machine learning model and the third machine learning model based on the determined type of fish. The method for determining the type of fish captured in the image is arbitrary, and for example, the type of fish captured in the image may be determined based on a machine learning model. Such a machine learning model may be trained based on multiple training data, and the training data may include, as input, an image of the fish and, as output, training data including the type of fish. In any case, the method for determining the type of fish captured in the image is not limited to this.

[0124] Alternatively, the computer may determine whether the fish in the image has convex portions at both ends of its tail fin, and selectively use the first machine learning model if it determines that the fish does not have convex portions at both ends, and the third machine learning model if it determines that the fish has convex portions at both ends. Any method for determining whether the fish in the image has convex portions at both ends of its tail fin may be used. For example, the computer may determine whether the fish in the image has convex portions at both ends of its tail fin based on a machine learning model. Such a machine learning model may be trained based on multiple training data, and the training data may include, as input, an image of the fish, and, as output, training data indicating whether the fish has convex portions at both ends of its tail fin. In any case, the method for determining whether the fish in the image has convex portions at both ends of its tail fin is not limited to this.

[0125] The fish targeted by an embodiment of the present disclosure may include a fish having convex portions at both ends of the tail fin. It will be understood that, according to the exemplary process 500 and the processes described above, the fish body shape recognized in step 210 may be a fish body shape excluding the convex portions at both ends of the tail fin.

[0126] 6 is a flowchart of an example process 600 according to an embodiment of the present disclosure. Note that each step included in the example process 600 may be executed by a computer. The example process 600 can be used as a substitute for the example process 200, and the same steps as those in the example process 200 are denoted by the same reference numerals.

[0127] 610 shows the step of determining, for each of the one or more fish recognized in step 210, the orientation of the fish in the recognized image.

[0128] The method for determining the orientation of the fish is arbitrary.

[0129] For example, the exemplary process 300 may be further executed to determine the orientation of the fish from the positional relationship between the coordinates of the center of gravity of the fish determined in step 220 and the coordinates corresponding to the fish's mouth determined in step 310. For example, a vector pointing from the coordinates of the center of gravity of the fish to the coordinates corresponding to the fish's mouth may be taken as the orientation of the fish. Also, for example, the lateral (x-axis direction) component of the vector of the recognized image of the fish may be taken as the orientation of the fish.

[0130] Also, for example, exemplary process 300 may be further performed to determine an orientation for the fish from a relationship between the center of gravity for the fish determined in step 220 and the first line segment for the fish determined in step 330.

[0131] In any case, the method for determining the orientation of a fish is not limited to this.

[0132] 620 shows a correspondence determination step for determining whether a fish recognized in the right image and a fish recognized in the left image correspond to the same fish based on the center of gravity.

[0133] The conditions for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish are arbitrary. The conditions can be, for example, The distance between the coordinates of the center of gravity in the right image of the fish recognized in the right image and the coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a predetermined distance, and The orientation of the fish recognized in the right image in the right image matches the orientation of the fish recognized in the left image in the left image. The condition may include, for example, The distance in a first direction between the coordinates of the center of gravity in the right image of a fish recognized in the right image and the coordinates of the center of gravity in the left image of a fish recognized in the left image is less than or equal to a first predetermined direction, and the distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance, and The orientation of the fish recognized in the right image in the right image matches the orientation of the fish recognized in the left image in the left image. In any case, the condition is not limited to this. The first direction and the second direction may be the horizontal direction (or x-axis direction) and the vertical direction (or y-axis direction) of the image, but are not limited to this.

[0134] 3. Computer An example of a hardware configuration of a computer that can be used to implement an embodiment of the present disclosure will be described below.

[0135] 13 shows an example of a computer hardware configuration. As shown in the figure, the computer 1300 mainly includes, as hardware resources, a processor 1310, a main memory device 1320, an auxiliary memory device 1330, an input / output interface 1340, and a communication interface 1350, which are interconnected via a bus line 1360 including an address bus, a data bus, a control bus, etc. Note that an interface circuit (not shown) may be interposed between the bus line 1360 and each hardware resource as appropriate. Note that some of these components (e.g., the communication interface 1350) may not be included in the computer 1300.

[0136] The processor 1310 controls the entire computer or at least part of the computer, such as a CPU or microprocessor. Note that one computer may include multiple processors 1310. In such cases, the term "processor" in the above description may be a general term for the multiple processors 1310.

[0137] The main memory device 1320 is a volatile memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM) that provides a working area for the processor 1310.

[0138] The auxiliary storage device 1330 is a non-volatile memory such as an HDD, SSD, or flash memory that stores software programs and data. The programs and data are loaded from the auxiliary storage device 1330 to the main storage device 1320 via the bus line 1360 at any time. The auxiliary storage device 1330 may also be called a computer-readable storage medium, a non-transitory computer-readable medium, or a computer-readable storage medium. Note that the programs include instructions that cause the processor to perform desired processing.

[0139] The input / output interface 1340 presents information and / or receives information input, and may include a digital camera, LiDAR sensor, keyboard, mouse, display, touch panel display, microphone, speaker, various sensors, etc.

[0140] The communication interface 1350 is connected to a network 1370 configured from one or more of the Internet, a local area network (LAN), etc., and transmits and receives data via the network 1370. The communication interface 1350 and the network 1370 may be connected by wire or wirelessly. The communication interface 1350 may also acquire information related to the network, such as information related to Wi-Fi access points and information related to communication carrier base stations.

[0141] It will be apparent to those skilled in the art that the cooperation of the hardware resources and software exemplified above enables the computer 1300 to function as desired means, execute desired steps, and achieve desired functions.

[0142] 4. Conclusion Up to this point, the embodiments of the present disclosure have been described, but it goes without saying that the present disclosure is not limited to the above-described embodiments and may be embodied in various different forms within the scope of its technical concept.

[0143] Furthermore, the scope of the present disclosure is not limited to the exemplary embodiments shown and described, but includes all embodiments that achieve equivalent effects to those intended by the present disclosure. Furthermore, the scope of the present disclosure is not limited to the combinations of inventive features defined by each claim, but may be defined by any desired combination of specific features among all the respective disclosed features. [Explanation of symbols]

[0144] 100A~100C...Example system 110, 720A~720C, 820, 825, 920A~920C, 1020, 1120A, 1120B, 1220...Fish 120...Fish tank 130...Stereo camera 160...Cloud 162...Database 170, 1370…Network 200~600...Example processing 710, 810, 815, 910, 1010, 1110, 1210...Images 730A~730C, 930, 1030, 1230...Fish body shape approximated by polygons with pentagons or more 830, 835, 840…center of gravity 850...Distance between centers of gravity 860…First direction 865…Second direction 870: Distance between centers of gravity in the first direction 875...2nd direction distance between centers of gravity 940, 1240...coordinates corresponding to the mouth 950A~950B, 1040...First line segment 960...Line segment that is not the first line segment 1050...Second line segment 1060...A line segment that is not the second line segment 1250...The longest line segment connecting the coordinate corresponding to the mouth and another coordinate among the coordinates that make up the shape of the fish body 1260...The line segment equivalent to the length of the fish's fork

Claims

1. a fish body shape recognition step in which a computer recognizes a fish body shape approximated by a polygon having pentagons or more sides in each of a right image and a left image captured by a stereo camera, and a fish is recognized by recognizing the fish body shape; a center of gravity determination step in which the computer determines a center of gravity of each of the one or more recognized fish in the recognized image of the fish based on the fish body shape; a correspondence determination step in which a computer determines whether a fish recognized in the right image and a fish recognized in the left image correspond to the same fish based on the center of gravity; an actual size determination step in which the computer determines the actual size of the same fish based on the size of the same fish in at least one of the right image and the left image; A method comprising:

2. 10. The method of claim 1, a step of determining, by the computer, for at least one of the one or more recognized fish, coordinates corresponding to a mouth among coordinates constituting the fish body shape in the recognized image of the fish; a step in which the computer determines, in the recognized image of the fish, a line segment that connects a coordinate corresponding to the mouth and another coordinate among the coordinates that form the shape of the fish, as a first line segment; an on-image size determination step in which the computer determines a size of the fish in the recognized image of the fish based on the first line segment; A method comprising:

3. 3. The method according to claim 2, wherein the on-image size determination step comprises: a step in which the computer determines, for the fish, as a second line segment, the longest line segment that is perpendicular to the first line segment and connects two coordinates that constitute the fish body shape in the recognized image of the fish; determining a size of the fish in the recognized image based on the second line segment; A method comprising:

4. The method of claim 2 , wherein the size of the fish in the recognized image of the fish is the length of the first line segment.

5. The method of claim 3 , wherein the size of the fish in the recognized image of the fish is the length of the second line segment.

6. 10. The method of claim 1, The fish includes a fish having convex portions on both ends of a tail fin, The fish body shape is a fish body shape excluding the convex parts at both ends of the tail fin, method.

7. 7. The method of claim 6, The fish body shape recognition step includes: A step in which the computer recognizes, based on a machine learning model, a polygonal approximation of the fish body shape in each of the right image and the left image, from which the convex portions at both ends of the tail fin have been removed. Including, The machine learning model is trained based on a plurality of training data, and the plurality of training data includes: As input, we take an image of a fish. The output is a polygonal approximation of the fish body shape, excluding the convex parts at both ends of the tail fin. Including training data, method.

8. 8. The method according to claim 7, wherein the plurality of training data include training data including images of a first type of fish and training data including images of a second type of fish different from the first type.

9. The method according to claim 7 , wherein all of the images of fish included in the plurality of learning data are images of fish captured from a predetermined direction.

10. 10. The method of claim 1, the right and left images have a common coordinate system; The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is: The distance between the coordinates of the center of gravity in the right image of the fish recognized in the right image and the coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a predetermined distance. Including the condition that method.

11. 10. The method of claim 1, the right and left images have a common coordinate system; The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is: a distance in a first direction between the coordinates of the center of gravity in the right image of the fish recognized in the right image and the coordinates of the center of gravity in the left image of the fish recognized in the left image is less than or equal to a first predetermined distance, and a distance in a second direction perpendicular to the first direction is less than or equal to a second predetermined distance. Including the condition that method.

12. 11. The method of claim 10, determining, by the computer, for each of the one or more recognized fish, the orientation of the fish in the recognized image; Further comprising: The condition for determining that the fish recognized in the right image and the fish recognized in the left image correspond to the same fish is: The orientation of the fish recognized in the right image in the right image matches the orientation of the fish recognized in the left image in the left image. It further includes the condition that method.

13. 2. The method of claim 1, wherein the size of the same fish in the at least one image is the larger of the size of the same fish in the right image and the size of the same fish in the left image.

14. The method according to claim 1 , further comprising the step of excluding fish whose centroids in the right image or the left image are outside a predetermined range from the fish recognized in the image.

15. 2. The method according to claim 1, wherein in the actual size determination step, the actual size of the same fish is determined based on a parallax determined from the center of gravity in the right image and the left image of the same fish and also on characteristics of the stereo camera.

16. a fish body shape recognition step of recognizing a fish body shape approximated by a polygon having pentagons or more sides in each of a right image and a left image captured by a stereo camera, and recognizing the fish body shape to thereby recognize the fish; a center of gravity determination step of determining a center of gravity of each of the recognized one or more fish in the recognized image of the fish based on the fish body shape; a correspondence determination step of determining, based on the center of gravity, a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish; an actual size determining step of determining an actual size of the same fish based on a size of the same fish in at least one of the right image and the left image; A program for causing a computer to execute a method including the steps of:

17. a fish body shape recognition step of recognizing a fish body shape approximated by a polygon having pentagons or more sides in each of a right image and a left image captured by a stereo camera, and recognizing the fish body shape to thereby recognize the fish; a center of gravity determination step of determining a center of gravity of each of the recognized one or more fish in the recognized image of the fish based on the fish body shape; a correspondence determination step of determining, based on the center of gravity, a fish recognized in the right image and a fish recognized in the left image that correspond to the same fish; an actual size determining step of determining an actual size of the same fish based on a size of the same fish in at least one of the right image and the left image; A system configured to run

Images