Computer program, model generation method, inference method, and inference device

Abstract

A computer program, model generation method, inference method, and inference apparatus are provided for inferring the size of aquatic organisms. The computer program causes a computer to perform the following processes: acquiring image data obtained from photographing aquatic organisms; inputting the acquired image data into a first learning model that outputs position data of a specified part of the aquatic organism given the image data, thereby acquiring position data of the specified part of the photographed aquatic organism; inputting the acquired image data into a generation model that generates a segmented image of the aquatic organism given the image data, thereby generating a segmented image of the photographed aquatic organism; and inferring the size of the photographed aquatic organism based on the position data of the specified part and the segmented image.

Description

Technical Field

[0001] This invention relates to computer programs, model generation methods, inference methods, and inference devices. Background Technology

[0002] The size of farmed fish in a fishpond is crucial information for determining feeding amounts and harvesting times. Previously, various analytical services were available to automate the measurement of fish length in the water. These services measured length by extracting multiple feature points from the fish body and measuring the distances between these feature points (e.g., between the snout and the tail fork).

[0003] Patent document 1 discloses an information processing device that detects two characteristic parts (snout and tail fin) of a fish from an image obtained by photographing the fish, and calculates the size of the fish based on the length between the detected characteristic parts.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: International Publication No. 2018 / 061925 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, since the optimal location for measuring the body height of fish is not necessarily characteristic, it is not possible to measure body height with high precision in terms of measurement accuracy and biological stability, making it difficult to infer the size of aquatic organisms such as fish.

[0009] The present invention was made in view of this situation, and its purpose is to provide a computer program, a model generation method, an inference method, and an inference device capable of inferring the size of aquatic organisms.

[0010] Methods for solving problems

[0011] The computer program of the present invention causes a computer to perform the following processes: acquiring image data obtained from photographing aquatic organisms; inputting the acquired image data into a first learning model that outputs position data of a specified part of the aquatic organism when the image data is input, thereby obtaining position data of the specified part of the photographed aquatic organism; inputting the acquired image data into a generation model that generates a segmented image of the aquatic organism when the image data is input, thereby generating a segmented image of the photographed aquatic organism; and inferring the size of the photographed aquatic organism based on the position data of the specified part and the segmented image.

[0012] The computer program of the present invention causes a computer to perform the following processes: inputting the acquired image data into a second learning model that outputs first position data of an aquatic organism and second position data of a predetermined part of the aquatic organism when image data is input, thereby outputting the first position data and the second position data; inputting image data of a second region containing the second position data into the first learning model; and inputting image data of a first region containing the first position data into the generation model.

[0013] The computer program of the present invention causes a computer to perform the following process: calculate body height based on a straight line orthogonal to the line connecting the caudal fin and the snout, wherein the specified portion includes the caudal fin and the snout.

[0014] The computer program of the present invention causes a computer to perform the following processes: inputting image data captured by a stereo camera into the first learning model, and calculating the three-dimensional position of the snout and tail fork based on the obtained position data of the specified parts; generating a body height auxiliary line based on the calculated three-dimensional position; and projecting the generated body height auxiliary line onto the segmented image in two dimensions to calculate the body height.

[0015] The computer program of the present invention causes a computer to perform the following process: calculate the body height using a height auxiliary line located at a predetermined proportion of the distance between the muzzle and the tail fork, starting from the position of the muzzle or the tail fork.

[0016] The computer program of the present invention causes a computer to perform the following processes: inputting image data of multiple frames captured by a stereo camera into the first learning model, and calculating the three-dimensional position of the tail fork in each frame based on the obtained position data of a specified part; determining the displacement of the tail fork according to each frame based on the calculated three-dimensional position of the tail fork; and selecting a frame for inferring the size of the aquatic organism based on the determined displacement.

[0017] The computer program of the present invention causes a computer to perform the following processes: displaying the inferred position of the caudal fin or snout based on the position data output by the first learning model; accepting corrections to the displayed inferred position of the caudal fin or snout; and causing the first learning model to relearn based on the accepted corrected position and the image data when the corrected position was accepted.

[0018] The computer program of the present invention causes a computer to perform the following processes: displaying a segmented image generated by the generative model; accepting corrections to the displayed segmented image; and relearning the generative model based on the corrected segmented image and the image data when the corrections were accepted.

[0019] The computer program of the present invention causes the computer to perform the following process: to assign a fork length or body height to an image of the aquatic organisms being raised in a fishpond for display.

[0020] The computer program of the present invention causes a computer to perform the following process: classify the estimated size of multiple aquatic organisms cultured in a fishpond into multiple levels of precision, and display the number of aquatic organisms in each of the classification levels.

[0021] The computer program of the present invention causes a computer to perform the following process: displaying the distribution of dimensions of at least one of the fork length and body height of a plurality of aquatic organisms cultured in a fishpond.

[0022] The model generation method of the present invention includes: acquiring first training data including image data obtained by photographing aquatic organisms and position data of specified parts of the aquatic organisms; acquiring second training data including the image data and segmented images of the aquatic organisms; generating a first learning model based on the first training data, such that when image data obtained by photographing aquatic organisms is input, the model outputs position data of specified parts of the aquatic organisms; and generating a generation model based on the second training data, such that when image data obtained by photographing aquatic organisms is input, the model generates segmented images of the aquatic organisms.

[0023] The model generation method of the present invention includes: obtaining third training data including image data obtained by photographing aquatic organisms, first position data of the aquatic organisms, and second position data of a specified part of the aquatic organisms; and generating a second learning model based on the third training data, so as to output the first position data of the aquatic organisms and the second position data of the specified parts of the aquatic organisms when the image data obtained by photographing aquatic organisms is input.

[0024] The inference method of the present invention includes: acquiring image data obtained by photographing aquatic organisms; inputting the acquired image data into a first learning model that outputs position data of a specified part of the aquatic organism when the image data is input, thereby obtaining position data of the specified part of the photographed aquatic organism; inputting the acquired image data into a generation model that generates a segmented image of the aquatic organism when the image data is input, thereby generating a segmented image of the photographed aquatic organism; and inferring the size of the photographed aquatic organism based on the position data of the specified part and the segmented image.

[0025] The inference apparatus of the present invention comprises: a first acquisition unit that acquires image data obtained by photographing aquatic organisms; a second acquisition unit that inputs the image data acquired by the first acquisition unit into a first learning model that outputs position data of a predetermined part of the aquatic organism when image data is input, thereby acquiring position data of the predetermined part of the photographed aquatic organism; a generation unit that inputs the image data acquired by the first acquisition unit into a generation model that generates a segmented image of the aquatic organism when image data is input, thereby generating a segmented image of the photographed aquatic organism; and an inference unit that infers the size of the photographed aquatic organism based on the position data of the predetermined part and the segmented image.

[0026] Invention Effects

[0027] According to the present invention, it is possible to infer the size of aquatic organisms. Attached Figure Description

[0028] Figure 1 This is a block diagram illustrating an example of the configuration of the inference device in this embodiment.

[0029] Figure 2 This is a schematic diagram illustrating an example of the structure of the AI ​​department.

[0030] Figure 3 This is a schematic diagram illustrating the process of inferring the size of a fish.

[0031] Figure 4 This is a schematic diagram illustrating an example of the structure of the second learning model.

[0032] Figure 5 This is a schematic diagram illustrating an example of the structure of the first learning model.

[0033] Figure 6 This is a schematic diagram illustrating an example of the structure of a generative model.

[0034] Figure 7 This is a schematic diagram illustrating an example of a machine learning-based generative method for each model.

[0035] Figure 8 This is a schematic diagram illustrating an example of fish frame pairing between cameras.

[0036] Figure 9 This is an example diagram showing a pairing list of fish frames and part frames.

[0037] Figure 10 This is a schematic diagram illustrating an example of pairing fish with different body parts.

[0038] Figure 11 This is a diagram representing an example of a list of fish.

[0039] Figure 12 This is a block diagram illustrating an example of the structure of a tracking unit.

[0040] Figure 13 This is a block diagram illustrating an example of the structure of the tail beat removal section.

[0041] Figure 14 This is a schematic diagram representing an example of an approximate straight line representing the trajectory of a tail fork.

[0042] Figure 15 This is a block diagram illustrating an example of the structure of the measuring unit.

[0043] Figure 16 This is a schematic diagram illustrating an example of body height measurement.

[0044] Figure 17 This is a schematic diagram illustrating an example of automatic measurement of fish size.

[0045] Figure 18 This is a schematic diagram illustrating an example of the composition of the inference result DB.

[0046] Figure 19 This is a block diagram illustrating another example of the configuration of the inference device in this embodiment.

[0047] Figure 20 This is a schematic diagram showing the first example of the inference result.

[0048] Figure 21 This is a schematic diagram illustrating one example of the main factors contributing to error.

[0049] Figure 22 This is a schematic diagram showing the second example of the inference result.

[0050] Figure 23 This is a schematic diagram showing the third example of the inference result.

[0051] Figure 24 This is a schematic diagram showing the fourth example of the inference result.

[0052] Figure 25 This is a flowchart illustrating an example of the processing sequence of an inference device.

[0053] Figure 26 This is a flowchart illustrating an example of the processing sequence of an inference device. Detailed Implementation

[0054] The embodiments of the present invention will be described below. Figure 1This is a block diagram illustrating an example of the configuration of the inference device 100 according to this embodiment. The inference device 100 includes an input unit 10, an AI unit 20, a pairing unit 30, a tracking unit 40, a tailbeat removal unit 50, a measurement unit 60, an inference unit 70 serving as an inference unit, and an output unit 80.

[0055] The camera unit 200 is, for example, a waterproof stereo camera installed at a predetermined position in the water of a fishpond. The camera unit 200 is capable of photographing aquatic life such as fish swimming in the fishpond. Alternatively, the camera unit 200 can also be a single-lens camera. Furthermore, other photographing devices can be used instead of a camera. Hereinafter, fish will be used as an example of aquatic life for explanation. The camera unit 200 outputs captured image data (moving images). The captured image data consists of images in multiple frame units (frame images).

[0056] The inference device 100 can be a personal computer or a server on the Internet (cloud server). When the inference device 100 is a personal computer, it can directly obtain captured image data from the camera unit 200. When the inference device 100 is a server on the Internet, it can obtain captured image data from the camera unit 200 via a client terminal device (not shown).

[0057] The input unit 10 acquires the captured image data from the camera unit 200. The input unit 10 then outputs the acquired image data to the AI ​​unit 20.

[0058] AI Unit 20 is a model generated through machine learning that outputs the data needed to infer the size of a fish based on captured image data. The fish's size includes, for example, fork length and body height. AI Unit 20 will be explained in detail below. Fork length is the body length, specifically the length from the front of the fish's upper jaw (snout) to the most concave part of the central part of the caudal fin fork (tail fork). Body height is the vertical distance from the dorsal edge to the ventral edge of the fish, specifically the distance from the base of the pelvic fin to the dorsal edge.

[0059] Figure 2 This is a schematic diagram illustrating an example of the configuration of the AI ​​unit 20. The AI ​​unit 20 includes a first learning model 21 as a second acquisition unit, a generation model 22 as a generation unit, a second learning model 23, and an image cropping unit 24 as the first acquisition unit. The inference unit 70 can infer the size of the fish based on the positional data of the parts output by the first learning model 21 and the segmented image data of the fish output by the generation model 22. In this case, the camera unit 200 may also be a single-lens camera.

[0060] The second learning model 23 is generated through machine learning, which, given the input image data, outputs the position data of an individual fish (the first position data of the aquatic organism) and the position data of a specific part of the aquatic organism (the second position data of a specific part of the aquatic organism). The position data output by the second learning model 23 is the coordinates (x, y) or the region (x, y, width, height) on the image.

[0061] The image cropping unit 24, based on the location data (coordinates or region) of a single fish output by the second learning model 23, crops the image of the surrounding area containing the coordinates or region to obtain single fish image data. The single fish image contains the entire fish. The image cropping unit 24, based on the location data (coordinates or region) of a part output by the second learning model 23, crops the image of the surrounding area containing the coordinates or region to obtain part image data. The part image contains an image of a specified part of the fish. The image cropping unit 24 outputs single fish image data to the generation model 22 and part image data to the first learning model 21. Alternatively, the image cropping unit 24 can be located outside the AI ​​unit 20.

[0062] Regarding the first learning model 21, it is generated through machine learning, so that when the part image data output by the image cropping unit 24 is input, it outputs the position data of the part. Regarding the generation model 22, it is generated through machine learning, so that when the single fish image data output by the image cropping unit 24 is input, it outputs segmented fish image data.

[0063] Figure 3 This is a schematic diagram illustrating the process of inferring the size of a fish. In Figure 3 For convenience, let's assume a fish is reflected in the captured image. As described above, the second learning model 23, in cooperation with the image cropping unit 24, outputs image data including single fish image data (image data of a first region containing first position data) and part image data (image data of a second region containing second position data) when the captured image data is input. Figure 3 As shown, a single fish image is an image of a rectangular region surrounding the entire fish. Single fish image data includes data of the rectangular region and data of the class (e.g., fish) within that region. A part image includes an image of the snout (P1) as a specified part, and an image of the caudal fin (or caudal fork) (P2) as a specified part. Part image data includes data of the rectangular region and data of the class (e.g., snout and caudal fin) within that region.

[0064] The first learning model 21 outputs the position data of the part when given the image data of the part. The position data of the part is the position data of the snout P1 and the position data of the caudal fin P2, which are coordinates on the image. If the coordinate system of the image is set to XY, the position data can be represented by coordinates (x, y) or regions (x, y, width, height). In addition, in this case, the coordinates of the camera's optical axis can be represented by Z.

[0065] Generative model 22, given input single fish image data, outputs segmented image data. The segmented image is an image created by classifying the categories of each pixel in the single fish image. Figure 3 In the example, pixels classified as "fish body with fins removed" are set to "no pattern," while pixels classified as "outside the fish body" are represented as "with pattern." By segmenting the image, the boundary between the fish body and the outside of the fish body (the outline of the fish body) can be determined.

[0066] The inference unit 70 can infer the size of the fish based on the location data of the part and the segmented image data. For example... Figure 3 As shown, the length of the straight line L connecting the snout tip P1 and the caudal fin P2 can be deduced as the fork length. Furthermore, the length between the intersection of the straight line H, which is orthogonal to the straight line L, and the outline of the fish body can be deduced as the body height. Here, the position of the intersection of the straight line H and the straight line L can be set as a point on the straight line L at a predetermined proportion from the position of the snout tip P1 (e.g., 40% of the fork length).

[0067] As described above, since the positional data of specified parts (snout and caudal fin) can be combined with the segmented image, and the body height measurement can be performed based on the body height line H, which is orthogonal to the line representing the fork length, and the segmented image, the size of the fish can be inferred stably and with high accuracy even without the feature part for optimal body height measurement.

[0068] Furthermore, the pairing unit 30, described later, can perform body height measurement with higher accuracy through subsequent processing.

[0069] Figure 4This is a schematic diagram illustrating an example of the structure of the second learning model 23. The second learning model 23 can, for example, be constructed using a Faster R-CNN (Convolutional Neural Network). The second learning model 23 includes a CNN layer 231, a Region Proposal Network (RPN) 232, a Region-of-Interest Pool (ROI) 233, and a recognition network 234. The CNN layer 231 generates image features (feature maps) based on the input captured image data and outputs them to the RPN 232 and the ROI pool 233.

[0070] The Region Proposal Network (RPN) 232 calculates candidate regions based on the input image features and outputs them to the ROIPOOL 233. The Region Proposal Network (RPN) 232 can detect where an object is reflected in the captured image, that is, it can detect the region where the object is reflected and the shape of its rectangle.

[0071] ROIPOOL 233 connects the image features output from CNN layer 231 with the candidate regions output from Region Proposal Network (RPN) 232, and outputs the fixed-length ROI region features to recognition network 234.

[0072] The recognition network 234 recalculates the accurate region / category based on the features of the input ROI region, and classifies and outputs the region and its category. Here, the categories are single fish, snout, and tail fin. The second learning model 23 is not limited to Faster R-CNN, but can also be R-CNN, Mask R-CNN, YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), etc.

[0073] Figure 5This is a schematic diagram illustrating an example of the structure of the first learning model 21. The first learning model 21 can be constructed, for example, using RetinaNet. The first learning model 21 includes a Feature Pyramid Network 211, a category classifier 212, and a region regressor 213. The part image data output by the second learning model 23 is input into the first learning model 21. The part image data includes image data of the caudal fin and image data of the snout. The Feature Pyramid Network 211 computes feature layers constituting feature maps of various scales in a bottom-up direction, and upsamples feature maps from spatially coarser but semantically stronger higher-level features in a top-down direction to generate high-resolution features. Each feature is associated with a feature computed in the bottom-up direction.

[0074] Thus, the Feature Pyramid Network 211 can extract features with both high-order and low-order characteristics, achieving a good balance of accuracy between semantics and position.

[0075] Region regressor 213 outputs the positions of the snout and caudal fin. The positions of the snout and caudal fin are represented by coordinate values ​​on the image. Classifier 212 determines the category (snout and caudal fin) of the positions (coordinate values) output by region regressor 213. Through the first learning model 21, the position of the snout on the snout image and the position of the caudal fin on the caudal fin image can be determined with high accuracy. The first learning model 21 is not limited to RetinaNet; for example, it can also be SegNet, Mask R-CNN, SVM (Support Vector Machine), etc.

[0076] Figure 6 This is a schematic diagram illustrating an example of the structure of the generative model 22. The generative model 22 can be constructed, for example, using a U-Net. The generative model 22 includes encoders 221-225 and decoders 226-229. For the input single fish image data, the generative model 22 repeatedly performs convolution processing using encoders 221-225. Decoders 226-229 repeatedly perform upsampling (deconvolution) processing on the image convolved by encoders 225. When decoding the convolved image, the feature maps generated by encoders 224-221 are added to the image that has undergone deconvolution. The generative model 22 outputs a segmented image of the fish body from the input single fish image. By using the segmented image, the boundary between the fish body and the surrounding area (the outline of the fish body) can be determined.

[0077] Therefore, it can preserve the positional information lost due to convolution processing and output higher-precision segmentation (which pixel belongs to which category). The generative model is not limited to U-Net, but can also be GAN (Generative Adversarial Network), SegNet, etc.

[0078] Figure 7 This is a schematic diagram illustrating an example of a machine learning-based generative method for each model. Figure 7 In the diagram, A represents the method for generating the second learning model 23. The second learning model 23 is generated through machine learning, whereby, given captured image data as input for learning, it outputs position data of a single fish (first position data) and position data of a part of the fish (second position data). In this case, the teacher data uses position data of a single fish and position data of a part of the fish generated by labeling the captured image data as the input for learning. Multiple sets of learning data and teacher data are prepared, and machine learning is performed on the second learning model 23. The internal parameters of the second learning model 23 are updated to generate the second learning model 23 so that the output data closely approximates the teacher data.

[0079] Figure 7 In the diagram, B represents the method for generating the first learning model 21. The first learning model 21 is generated through machine learning, outputting positional data of a part when given image data of a part as input for learning. In this case, the teacher data uses positional data of the part created by annotating the image data of the part as input for learning. Multiple sets of learning data and teacher data are prepared, and machine learning is performed on the first learning model 21. The internal parameters of the first learning model 21 are updated to generate the first learning model 21 so that the output data closely approximates the teacher data.

[0080] Figure 7 In the diagram, C represents the generation method of generative model 22. Generative model 22 is generated through machine learning, producing segmented image data as the output, given single-fish image data as input for learning. In this case, the teacher data uses segmented image data created by annotating the single-fish image data as the input for learning. Multiple sets of learning data and teacher data are prepared, and machine learning is applied to generative model 22. The internal parameters of generative model 22 are updated to generate generative model 22 such that the output data closely approximates the teacher data.

[0081] This also allows the aforementioned models to be relearned. For example, such as... Figure 3As shown, based on the position data output by the first learning model 21, the inferred positions of the caudal fin and snout are displayed on the screen, and operations are performed to correct the displayed inferred positions of the caudal fin and snout. Correction of the inferred position is achieved, for example, by manipulating a mouse to move the pointer on the screen to the inferred position and performing a specified operation (touch, click, drag, etc.), thereby moving the inferred position to the correct location on the screen. The corrected position and the image data at the time of correction are prepared as training data, allowing the first learning model 21 to relearn based on the corrected position and the image data at the time of correction. Therefore, the accuracy of the position data output by the first learning model 21 is improved.

[0082] In addition, such as Figure 3 As shown, the segmented image generated by the generative model 22 is displayed on the screen, and operations are performed to correct the displayed segmented image. The segmented image can be corrected, for example, by launching an application such as drawing software, displaying the segmented image on the screen, moving the pointer on the screen to the desired position using a mouse, and performing a specified operation (touch, click, drag, etc.) to convert pixels outside the fish body into pixels of the fish body, or vice versa. The corrected segmented image and the image data used for correction are prepared as training data, and the generative model 22 can be relearned based on the corrected segmented image and the image data used for correction. As a result, the accuracy of the segmented image generated by the generative model 22 is improved.

[0083] Additionally, the position data of the individual fish, the snout position data, and the tail fin position data output by the second learning model 23 are displayed on the screen, and operations to correct the displayed position data are performed. Alternatively, in this case, from the viewpoint of improving workability and visibility, cropped images (individual fish image, snout image, and tail fin image) containing the positions of the individual fish, snout, and tail fin respectively, can also be displayed on the screen along with the position data. When correcting the position data of the individual fish, the position of the individual fish output by the second learning model 23 is displayed on the screen, and operations to correct the displayed position of the individual fish are performed. Correction of the individual fish position can be achieved, for example, by moving the individual fish on the screen using a mouse or similar device. When correcting the snout position, the snout position output by the second learning model 23 is displayed on the screen, and operations to correct the displayed snout position are performed. Correction of the snout position can be achieved, for example, by moving the snout position on the screen using a mouse or similar device. When correcting the tail fin position, the tail fin position output by the second learning model 23 is displayed on the screen, and operations to correct the displayed tail fin position are performed. The position of the caudal fin can be corrected, for example, by moving the caudal fin on the screen using a mouse. Alternatively, the position data corrected for relearning the first learning model 21 can also be used as the position data for the snout and caudal fin.

[0084] Using the corrected position of the individual fish and the image data from the time of correction as training data, the second learning model 23 can relearn based on the corrected position of the individual fish and the image data from the time of correction. Similarly, using the corrected position of the snout and the image data from the time of correction as training data, the second learning model 23 can relearn based on the corrected position of the snout and the image data from the time of correction. Likewise, using the corrected position of the caudal fin and the image data from the time of correction as training data, the second learning model 23 can relearn based on the corrected position of the caudal fin and the image data from the time of correction. Therefore, the accuracy of the position data of the individual fish and the position data of the body parts output by the second learning model 23 is improved.

[0085] Next, methods to further improve the accuracy of body height measurement will be explained. The following sections will describe the pairing process, tracking process, tail removal process, and body height measurement process in sequence. Additionally, the camera unit 200 is a stereo camera.

[0086] The pairing unit 30 performs pairing processing between cameras and pairing processing between fish and body parts. First, the pairing processing between cameras will be explained. The pairing processing between cameras includes fish frame pairing and body part frame pairing.

[0087] Figure 8 This is a schematic diagram illustrating an example of fish-frame pairing between cameras. Figure 8 In the image, the left-hand recognition image is an inferred image of a single fish based on the image captured by the left-hand camera of the stereo camera, while the right-hand recognition image is an inferred image of a single fish (also known as a fish frame) based on the image captured by the right-hand camera of the stereo camera. Figure 8 In the example, three fish frames are shown, but in actual images, the number of fish frames is not limited to three. Alternatively, instead of arranging the stereo camera in the left-right (horizontal) direction, it can be arranged in the up-down direction, or it can be arranged in an angle (e.g., an angle greater than 0 degrees and less than 90 degrees relative to the horizontal direction).

[0088] The matching unit 30 performs pairing on all fish frames within the left and right recognition images in turn. Figure 8 In the example, a pairing is performed between fish frames G1 in the left-hand recognition image and all fish frames G4, G5, and G6 in the right-hand recognition image. During pairing, the closest pair is selected within the allowable error range. Figure 8 In the example, for fish frame G1, fish frame G6 is selected as the closest pair. Similarly, for fish frames G2 and G3 in the left recognition image, the corresponding pairs are selected from the fish frames in the right recognition image.

[0089] Additionally, although not illustrated, part box matching can be performed in the same way as fish frame matching. The matching unit 30 performs matching on all part boxes in both the left and right recognition images alternately. Matching is performed between part boxes in the left recognition image and all part boxes in the right recognition image. During matching, the closest pair is selected within the allowable error range.

[0090] Figure 9 This is an example diagram showing a pairing list of fish frames and part frames. Figure 9 In the middle, A represents the fish frame pairing list. Figure 9 The "B" in the middle represents a list of paired parts. For example... Figure 9 As shown in Figure A, the fish frame G1 in the left recognition image and the fish frame G6 in the right recognition image form a pair. Similarly, the fish frame G2 in the left recognition image and the fish frame G4 in the right recognition image form a pair, and the fish frame G3 in the left recognition image and the fish frame G5 in the right recognition image form a pair.

[0091] like Figure 9 As shown in Figure B, the part bounding box g1 in the left recognition image and the part bounding box g12 in the right recognition image form a pair; the part bounding box g2 in the left recognition image and the part bounding box g10 in the right recognition image form a pair; and the part bounding box g3 in the left recognition image and the part bounding box g8 in the right recognition image form a pair. The same applies to other parts.

[0092] Figure 10 This is a schematic diagram illustrating an example of pairing fish with body parts. Through the pairing between the cameras described above, fish frames G1-G3 and body part frames g1-G6 are paired. Furthermore, fish frames G1-G3 and body part frames g1-G6 each have paired fish frames and body part frames. Figure 10 The image shows the paired fish frame and one side of the part frame (e.g., the left-hand recognition image).

[0093] The matching unit 30 performs matching on all fish frames and part frames in the recognition image in turn. In this case, matching can be performed using only the left recognition image, only the right recognition image, or both recognition images (left and right). Figure 10 In the example, pairing is performed between fish frame G1 and all part frames g1 to g6 within the recognition image. During pairing, the closest pair is selected within the allowable error range. The same process is performed for other fish frames G2 and G3.

[0094] Figure 11 This is a diagram representing an example of a list of fish. For example... Figure 11 As shown, fish frame G1 is paired with part frames g3 and g4. Similarly, fish frame G2 is paired with part frames g1 and g2, and fish frame G3 is paired with part frames g5 and g6. That is, the fish with ID 1 is paired with fish frame G1 and part frames g3 and g4, the fish with ID 2 is paired with fish frame G2 and part frames g1 and g2, and the fish with ID 3 is paired with fish frame G3 and part frames g5 and g6.

[0095] Through the above-described structure, for example, by photographing multiple fish in a fishpond, a corresponding fish frame and part frame can be established for each fish.

[0096] Figure 12 This is a block diagram illustrating an example of the configuration of the tracking unit 40. The tracking unit 40 tracks fish swimming in the fishpond between frames. By tracking, double counting of fish can be prevented. The tracking unit 40 includes a 2D tracking unit 41, a 3D conversion unit 42, and a 3D tracking unit 43.

[0097] The 2D tracking unit 41 uses the fish frame matching list, part frame matching list, and fish list generated in the matching unit 30 to track each fish. The 2D tracking unit 41 tracks the fish based on 2D images.

[0098] The 3D conversion unit 42, based on the fish frame pairing list, part frame pairing list, and fish list generated in the pairing unit 30, converts the 2D image of the fish into a 3D image using individual fish images captured by the left camera, individual fish images captured by the right camera, and the ranging principle of the stereo camera. Alternatively, the 3D positions of the snout and tail fork can be measured based on the principle of three-point measurement. The distance Z to each pixel of the 2D image of the fish can be calculated using Z = (B × F) / D. Here, B is the distance between the cameras, F is the focal length, and D is the parallax.

[0099] The 3D tracking unit 43 tracks each fish using the fish frame matching list, part frame matching list, and fish list generated in the matching unit 30. The 3D tracking unit 43 can track fish based on the position of the snout tip on a 3D image of the fish. This reduces the impact of the fish's tail wagging while swimming in the water.

[0100] Figure 13 This is a block diagram illustrating an example of the configuration of the tail removal unit 50. The tail removal unit 50 includes a tail fork 3D measurement unit 51, a planar projection unit 52, an approximate straight line calculation unit 53, and a frame removal unit 54.

[0101] The tail fork 3D measurement unit 51 measures the 3D position (X, Y, Z) of the tail fork (tail fin) on the image after it has been paired in the pairing unit 30. Here, the horizontal direction of the image can be set as the X-axis, the vertical direction as the Y-axis, and the optical axis direction of the camera as the Z-axis.

[0102] The planar projection unit 52 projects the 3D position of the tail fork onto the XZ plane. This allows the position of the tail fork to be set as viewed from the dorsal side of the fish. The planar projection unit 52 records the position of the tail fork as viewed from the dorsal side of the fish for each frame. This allows the position of the tail fork to be drawn on the XZ plane.

[0103] The approximate straight line calculation unit 53 calculates the approximate straight line of the position (tail fork trajectory) of the tail fork drawn on the XZ plane.

[0104] The frame removal unit 54 removes the frame corresponding to the tail fork if the deviation of the tail fork from the approximate straight line exceeds the allowable range. Alternatively, instead of calculating the approximate straight line after projecting the 3D position of the tail fork onto the plane, an approximate line in 3D space can be calculated based on the 3D position of the tail fork. In this case, the frame corresponding to the tail fork is removed when the deviation of the 3D position of the tail fork from the calculated 3D approximate line exceeds the allowable range.

[0105] Figure 14 This is a schematic diagram illustrating an example of an approximate straight line representing the trajectory of a tail fork. In Figure 14 In the diagram, the horizontal axis represents the position X of the tail fork, and the vertical axis represents the position Z of the tail fork. Figure 14In the diagram, O represents the measured position of the tail fork, t1 to t13 indicate the time frame, and the dashed line represents an approximate straight line of the tail fork trajectory. The tail fork position shifts between times t1 and t13. Figure 14 In the process, the position of the tail fork deviates from an approximately straight line at times t3 and t7–t9. Therefore, frames at times t3 and t7–t9 are removed, and the remaining frames at times t1–t2, t4–t6, and t10–t13 are used for the body height measurement described later. The deviation of the tail fork position from an approximately straight line is due to significant tail fork movement as the fish changes its course, and since this is a major factor contributing to errors in inferring the fish's size, frames corresponding to that tail fork are removed when the deviation exceeds an acceptable range. Furthermore, frame removal is not limited to methods using approximate straight lines; other machine learning methods such as SVM, neural networks, and clustering can also be used. Additionally, frames can be removed if the temporal change in the tail fork position exceeds a predetermined threshold.

[0106] The tail removal unit 50 outputs the frames other than the frames from the captured frames after removing the tail as the tail removal completed frame (the measurement object frame) to the measurement unit 60.

[0107] That is, the inference device 100 inputs the image data of multiple frames captured by the stereo camera into the first learning model 21, calculates the three-dimensional position of the tail fork in each frame based on the obtained position data of the specified parts, projects the calculated three-dimensional position onto a specified two-dimensional plane and determines the displacement of the tail fork according to each frame, and selects the frame for inferring the size of the fish based on the determined displacement.

[0108] Figure 15 This is a block diagram illustrating an example of the configuration of the measuring unit 60. The measuring unit 60 includes an optimal frame selection unit 61, a snout / tail fork 3D measuring unit 62, a body height auxiliary line generation unit 63, a body height auxiliary line projection unit 64, and a body height measuring unit 65.

[0109] The optimal frame selection unit 61 selects the frame corresponding to the moment when the tail fork is closest to the approximate straight line of the tail fork trajectory. Alternatively, instead of selecting the optimal frame, it may randomly select from the remaining frames after frame removal, or it may select a frame several frames before or after the removed frame.

[0110] The snout / tail fork 3D measurement unit 62 measures the 3D position of the snout and the 3D position of the tail fork in a selected frame. Alternatively, the 3D measurement can also use the 3D position of the snout tracked by the 3D tracking unit 43 and the 3D position of the tail fork measured by the tail fork 3D measurement unit 51.

[0111] The height auxiliary line generation unit 63 generates, for example, nine height auxiliary lines orthogonal to a 3D straight line connecting the 3D position of the snout and the 3D position of the tail fork measured in 3D, so as to divide the 3D straight line into ten parts.

[0112] The body height auxiliary line projection unit 64 projects the body height auxiliary line generated by the body height auxiliary line generation unit 63 onto a 2D plane (e.g., the XY plane).

[0113] The body height measurement unit 65 measures the body height based on the intersection point of the body height auxiliary line projected onto the 2D plane and the fish's outline (e.g., the outline on a segmented image). Furthermore, if the body height cannot be measured due to missing parts of the fish's outline, the optimal frame selection unit 61 can select a frame corresponding to the moment of the tail fork at a position approximately adjacent to the next straight line of the tail fork trajectory, and repeat the processes of the snout / tail fork 3D measurement unit 62, the body height auxiliary line generation unit 63, the body height auxiliary line projection unit 64, and the body height measurement unit 65. Missing parts of the fish's outline can be determined, for example, by comparing the distance between the intersection point of the body height auxiliary line and the fish's outline with a predetermined threshold. Alternatively, it can be determined by comparing the change in distance between the intersection points of adjacent body height auxiliary lines and the fish's outline with a predetermined threshold. Interpolation processing can also be performed on the missing parts.

[0114] Figure 16 This is a schematic diagram illustrating an example of height measurement. (For example...) Figure 16 As shown in Figure A, for example, nine body height auxiliary lines A1 to A9 are generated that are orthogonal to the 3D straight line L connecting the 3D position of the snout and the 3D position of the tail fork as measured in 3D, so as to divide the 3D straight line L into ten parts.

[0115] like Figure 16 As shown in Figure B, nine body height auxiliary lines A1 to A9 are projected onto a 2D plane (e.g., the XY plane). The body height is measured based on the intersection points of the body height auxiliary lines A1 to A9 projected onto the 2D plane with the fish's outline (e.g., the outline on a segmented image). Specifically, the body height can be measured as the distance H between the two points where the body height auxiliary line A4, which extends from the snout to a distance equivalent to 40% of the fork length L, intersects with the fish's outline. Furthermore, the number of body height auxiliary lines is an example, and is not limited to nine. Also, the ratio of 40% is an example, and is not limited to 40%.

[0116] Figure 17 This is a schematic diagram illustrating an example of automatic measurement of fish size. Figure 17 The diagram illustrates the fish IDs projected into the images at each frame number. For example, fish ID 0001 is projected into frames 1-10. In frame 7, represented by 0, the size of fish ID 0001 is automatically measured. Frame 7, measured automatically, is, for example, the frame with the smallest tail wag among frames 1-10.

[0117] Similarly, in frames 3-12, the fish with ID 0002 is projected. In frame 9 (represented by 0), the size of fish 0002 is automatically measured. In frames 5-9, the fish with ID 0003 is projected, but the size of fish 0003 is not automatically measured. Furthermore, in frames 7-14, the fish with ID 0004 is projected. In frame 13 (represented by 0), the size of fish 0004 is automatically measured. In frames 8-16, the fish with ID 0005 is projected. In frame 15 (represented by 0), the size of fish 0005 is automatically measured. Identification of the same individual between frames is performed by the tracking unit 40.

[0118] The measurement unit 60 outputs the measurement results to the inference unit 70. The inference unit 70 can collect the automatically measured fork length and body height of the fish in the fishpond and infer the size of the fish in the fishpond.

[0119] The output unit 80 can convert the result deduced by the inference unit 70 into displayable data and output it to external terminal devices, display devices, etc.

[0120] Figure 18 This is a schematic diagram illustrating an example of the structure of the inference result DB90. The inference result DB90 can be located inside the inference device 100, accessible from the inference device 100, or it can be located on an external data server, etc. The inference result DB90 can store the inference results of the inference device 100. For example, the inference result DB90 records a correspondence between the fish ID, fork length, body height, grade, and image and the fish ID. Additionally, although not shown, the position information (coordinate values) of the snout tip and the position information (coordinate values) of the tail fin (tail fork) can also be recorded in correspondence with the fish ID. The fish ID is, for example, an identifier for identifying fish in a fishpond. The grade is used to determine the measurement accuracy of the fish's size. Details of the grade will be described later. The image is an image of the fish, and a line representing the fork length and a line representing the body height can also be displayed over the fish image. Furthermore, the frame number, the fork length value, the body height value, and the aforementioned grade can also be displayed on the fish image.

[0121] Figure 19 This is a block diagram illustrating other examples of the configuration of the inference device 100 in this embodiment. For example... Figure 19 As shown, the inference device 100 can be, for example, a personal computer. The inference device 100 can be configured with a CPU 101, ROM 102, RAM 103, GPU 104, video memory 105, and a recording medium reading unit 106. The recording medium reading unit 106 (e.g., an optical disk drive) can read a computer program (computer program product) recorded on the recording medium 1 (e.g., an optically readable disk storage medium such as a CD-ROM) and store it in the RAM 103. Here, the computer program (computer program product) includes those described later. Figure 25as well as Figure 26 The processing order is described in the diagram. It can also be stored on a hard disk (not shown) and in RAM 103 during computer program execution.

[0122] By executing a computer program (computer program product) stored in RAM 103 via CPU 101, various processes in the input unit 10, AI unit 20, matching unit 30, tracking unit 40, tail removal unit 50, measurement unit 60, inference unit 70, and output unit 80 can be performed. The video memory 105 can temporarily store data and processing results used for various image processing tasks. Furthermore, the computer program (computer program product) can be downloaded from other computers or network devices via a network such as the Internet, instead of being read by the recording medium reading unit 106.

[0123] Next, the inference result displayed on the screen of the terminal device connected to the inference device 100 via the communication network will be explained. The following display processing is performed by the inference device 100 accessing the inference result DB90 according to a request from the terminal device, calculating the value for display if necessary, and outputting it to the terminal device. Furthermore, if the terminal device has already downloaded the inference result DB data, the downloaded data can be used by the terminal device's CPU and displayed on the display unit.

[0124] Figure 20 This is a schematic diagram showing the first example of the inference results. The inference results screen 300 displays various columns including grade, number of measurements, average fish weight, average fork length, average body height, average plumpness, average shooting distance, and average angle. Additionally, a box 301 displays an option to download detailed data for each grade; by clicking the "Save" icon 302, detailed data can be downloaded from the inference device 100. The downloaded data also includes motion images captured by the camera unit 200.

[0125] Grades are used to determine the accuracy of fish size measurements. For example, grade A can be defined as an error of less than 5%, grade B as an error of 5% or more but less than 10%, grade C as an error of 10% or more but less than 20%, and grade F as an error of 20% or more. However, the number of grade divisions and the definition of each grade are not limited to these.

[0126] Figure 21 This is a schematic diagram illustrating one example of the main factors contributing to error. Figure 21 In the figure, A represents the distribution of positional errors. The greater the errors in the snout position and the caudal fork (tail fin) position, the greater the errors in fork length and body height. The positional errors of the snout and caudal fork are calculated using the positional data of the snout and caudal fin (tail fork) output by the first learning model 21, and based on the reprojection error during 3D measurement. Figure 21 In the figure, B represents the error in the fish's body tilt (angle). The greater the tilt, the greater the error in body height. The tilt error reflects the angular deviation as a simple measurement position deviation in the body height error. The grade can be assigned based on the worse of the positional error or the tilt error.

[0127] The measured number represents the number of fish contained in each grade. Average fork length and average body height represent the average values ​​within each grade. Average fish weight and average fatness can be calculated using a prescribed formula based on the average fork length and average body height. In aquaculture, determining the weight of adult fish is particularly important. In practice, measuring the weight of adult fish using a weighing instrument requires considerable labor. However, according to this embodiment, the weight of the adult fish can be calculated without using an actual weighing instrument based on the fork length and body height inferred by the inference device 100. In this case, even without the optimal feature for measuring body height, the body height can be inferred stably and with high accuracy, thus allowing for the accurate inference of the adult fish's weight.

[0128] As described above, the inference device 100 categorizes the accuracy of the inferred size of multiple fish (aquatic organisms) raised in the fishpond into several levels and displays the fish in each level. Thus, by knowing the size range of the fish in the fishpond and even the reliability of the data, information can be provided for determining the feeding amount and harvesting time.

[0129] Figure 22 This is a schematic diagram representing a second display example of the inference result. The inference result screen 310 displays a distribution selection list 311 and a distribution display area 312. Figure 22 In the example, the distribution of fork length is selected from items such as fork length, body height, and fish weight, and the fork length distribution of fish in the pond is displayed in the distribution display area 312. Additionally, as detailed information on the fork length distribution, levels can also be displayed. Similar to fork length, the distribution of body height, fish weight, etc., can also be displayed.

[0130] As described above, the inference device 100 can display the distribution of at least one of the fork length and body height of multiple fish raised in a fishpond. Therefore, knowing the size distribution of the fish in the fishpond provides information for determining the feeding amount and harvesting time.

[0131] Figure 23 This is a schematic diagram representing a third display example of the inference results. The inference results screen 320 displays a grade selection list 321 and a fish measurement data display area 322. Figure 23 In the example, grade A is selected from grades A, B, C, and F. The measurement data for each fish belonging to grade A is displayed in fish measurement data display area 322. Figure 23 In the example, the measurement data includes fish weight, fork length, body height, and images, but is not limited to these. This can be achieved by manipulating box 323 in the image bar, as described later. Figure 24 As shown, it can display an image of the selected fish.

[0132] Therefore, it is possible to confirm the measurement data of each fish (individual) belonging to each grade.

[0133] Figure 24 This is a schematic diagram of the fourth display example representing the inference result. The selected fish image is displayed on the inference result screen 330. Alternatively, a line representing the fish's fork length and a line representing its body height can be displayed over the fish image. Additionally, the frame number, fork length value, body height value, fish weight value, and its class can also be displayed on the fish image.

[0134] As described above, the inference device 100 can display the fork length and body height of the fish image. This allows for the confirmation of measurement data for each individual fish of a given grade.

[0135] Figure 25 as well as Figure 26 This is a flowchart illustrating an example of the processing sequence of the inference device 100. The inference device 100 acquires captured image data (S11), inputs the acquired captured image data into the second learning model 23, and acquires individual fish image data and part image data from the image cropping unit 24 (S12). Here, the part image data includes snout image data and tail fin image data. The inference device 100 inputs the acquired part image data into the first learning model 21 to obtain part position data (S13). Here, the snout image data is input into the first learning model 21 to obtain the snout position information (coordinates or region values) output by the first learning model 21, and the tail fin image data is input into the first learning model 21 to obtain the tail fin position information (coordinates or region values) output by the first learning model 21.

[0136] The inference device 100 inputs the acquired individual fish image data into the generation model 22 to obtain segmented image data (S14), and infers (generates) the measurement points for the fork length based on the acquired position data of the parts and the segmented image data (S15). Here, the measurement points for the fork length (position information (coordinates or region values) of the snout and tail fin) are stored in the inference result DB90. Alternatively, the distance Z between the fish or a specified part of the fish and the camera can be calculated using the processing in steps S16 and S17 described later, and the size of the fish (fork length and body height) can be inferred.

[0137] The inference device 100 performs pairing between cameras (S16) and pairing of fish with body parts to generate a fish list (S17). The inference device 100 performs fish tracking processing (S18) and tail removal processing (S19). The inference device 100 selects the frame that is optimal for body height calculation (S20).

[0138] The inference device 100 determines whether body height calculation can be performed (S21). If body height calculation cannot be performed (S21 is no), step S20 is performed. If body height calculation can be performed (S21 is yes), the inference device 100 calculates the body height (S22). The body height calculation is performed through... Figure 16 The method shown in section B is used. The "calculation" of body height is synonymous with "measurement," but it also has the meaning of indirect measurement. The inference device 100 determines whether there are other fish (fish that have not been measured) (S23).

[0139] If other fish are present (S23 is yes), the inference device 100 repeats the processing after step S11. If no other fish are present (S23 is no), the inference device 100 collects the fish's calculation data (S24) and classifies the calculation data into levels according to the error (S25). The inference device 100 calculates the statistical value (e.g., average value) of the calculation data for each level (S26), creates the distribution for each inference item (e.g., fork length, body height, etc.) (S27), outputs the inference result (S28), and ends the processing.

[0140] The inference device 100 can be configured with a CPU, GPU, ROM, RAM, and a recording medium reading unit. The recording medium reading unit can read a computer program recorded on a recording medium and store it in RAM. By executing the computer program stored in RAM by the CPU or GPU, the processing performed in the inference device 100 can be executed. Alternatively, the computer program can be downloaded via a network such as the Internet, instead of being read by the recording medium reading unit.

[0141] According to this embodiment, since body height is measured by combining a segmented image with a body height line, the size of the fish can be inferred stably and with high accuracy even when there is no optimal location for body height measurement.

[0142] Furthermore, the extension or retraction of fins during swimming can reduce measurement accuracy. However, according to this embodiment, since the fish body with its fins removed is segmented, body height can be measured with high accuracy regardless of the influence of the fins.

[0143] Furthermore, extracting feature points with high precision from high-resolution images increases the computational load required for image processing. However, according to this embodiment, by combining the first learning model and the generation model, it is possible to combine the segmented image and the body height line to infer the size of the fish with high precision, and thus reduce the workload.

[0144] Explanation of reference numerals in the attached figures

[0145] 100 Inference device; 10 Input unit; 20 AI unit; 21 First learning model; 211 Feature Pyramid Network; 212 Classifier; 213 Region regressor; 22 Generative model; 221, 222, 223, 224, 225 Encoders; 226, 227, 228, 229 Decoders; 23 Second learning model; 24 Image cropping unit; 231 CNN layer; 232 Region proposal network; 233 ROIPOOL (Region of Interest Pool); 234 Recognition network; 30 Pairing unit; 40 Tracking unit; 41 2D tracking unit; 42 3D conversion unit; 43 3D tracking unit; 50 Tail wobble removal unit; 51 Tail fork 3D measurement unit; 52 Plane projection unit; 53 Approximate straight line calculation unit; 54 Frame removal unit; 60 Measurement unit; 61 Optimal frame selection unit; 62 Snout / tail fork 3D measurement unit; 63 Body height auxiliary line generation unit; 64 Body height auxiliary line projection unit; 65 Body height measurement unit; 70 Inference unit; 80 Output unit; 90 Inference result DB; 101 CPU; 102 ROM; 103 RAM; 104 GPU; 105 Video memory; 106 Recording medium reading unit; 200 Camera unit.

Claims

1. A computer program product comprising a computer program that causes a computer to perform the following processes: To obtain image data from photographing aquatic life; A first learning model, which outputs positional data of a specified part of an aquatic organism when given image data, is given the acquired image data to obtain positional data of the specified part of the aquatic organism captured in the image. The image data is input into a generative model that generates segmented images of aquatic organisms given image data, thereby generating the captured segmented images of aquatic organisms. The size of the captured aquatic organism is inferred based on the location data of the specified area and the segmented image; The image data captured by the stereo camera is input into the first learning model, and the three-dimensional position of the snout and tail fork is calculated based on the obtained position data of the specified parts; Generate volume height guide lines based on the calculated 3D position; as well as The generated body height auxiliary lines are projected two-dimensionally onto the segmented image to calculate the body height.

2. The computer program product as described in claim 1, The computer program causes the computer to perform the following processes: A second learning model, which outputs first position data of an aquatic organism and second position data of a specified part of the aquatic organism when image data is input, takes the acquired image data as input and outputs the first position data and the second position data. Input image data of the second region containing the second location data into the first learning model; as well as Image data of a first region containing the first location data is input into the generative model.

3. The computer program product as described in claim 1, The specified areas include the caudal fin and the snout. The computer program causes the computer to perform a process that calculates body height based on a straight line orthogonal to the line connecting the tail fin and the snout.

4. The computer program product as described in claim 2, The specified areas include the caudal fin and the snout. The computer program causes the computer to perform a process that calculates body height based on a straight line orthogonal to the line connecting the tail fin and the snout.

5. The computer program product as described in claim 1, The computer program causes the computer to perform the following process: calculate the body height using a height guideline located at a predetermined proportion of the distance between the snout and the tail fork, starting from the position of the snout or the tail fork.

6. The computer program product as described in any one of claims 1 to 5, The computer program causes the computer to perform the following processes: The first learning model is input with image data of multiple frames captured by a stereo camera, and the three-dimensional position of the tail fork in each frame is calculated based on the obtained position data of the specified parts. Based on the calculated three-dimensional position of the tail fork, the displacement of the tail fork is determined for each frame; as well as Based on the determined displacement, a frame is selected to infer the size of the aquatic organism.

7. The computer program product as described in any one of claims 1 to 5, The computer program causes the computer to perform the following processes: Based on the position data output by the first learning model, the inferred position of the caudal fin or snout is displayed; Corrections to the inferred position of the tail fin or snout as shown; and Based on the corrected location and the image data at the time of the corrected location, the first learning model is relearned.

8. The computer program product as described in any one of claims 1 to 5, The computer program causes the computer to perform the following processes: Display the segmented image generated by the generative model; Accept corrections to the displayed segmented image; and The generative model is relearned based on the corrected segmented image and the image data at the time of the correction.

9. The computer program product as described in any one of claims 1 to 5, The computer program causes the computer to perform the following processing: assigning fork length or body height to the images of the aquatic organisms being raised in the fishpond for display.

10. The computer program product as described in any one of claims 1 to 5, The computer program causes the computer to perform the following processing: classify the estimated size of multiple aquatic organisms raised in the fishpond into multiple levels of precision, and display the number of aquatic organisms in each level of classification.

11. The computer program product as described in any one of claims 1 to 5, The computer program causes the computer to perform the following processing: displaying the size distribution of at least one of the fork length and body height of multiple aquatic organisms cultured in the fishpond.

12. An inference method, comprising: To obtain image data from photographing aquatic life, A first learning model, which outputs positional data of specified parts of aquatic organisms given image data as input, obtains positional data of the specified parts of the aquatic organisms captured in the image data. A generative model that generates segmented images of aquatic organisms from input image data is used to generate segmented images of the captured aquatic organisms. The size of the captured aquatic organism is inferred based on the location data of the specified area and the segmented image. Image data captured by a stereo camera is input into the first learning model, and the three-dimensional positions of the snout and tail fork are calculated based on the obtained position data of the specified parts. Volume height guide lines are generated based on the calculated 3D position. The generated body height auxiliary lines are projected two-dimensionally onto the segmented image to calculate the body height.

13. A reasoning device, comprising: The first acquisition unit acquires image data obtained from photographing aquatic life; The second acquisition unit inputs the image data acquired by the first acquisition unit into a first learning model that outputs position data of a specified part of an aquatic organism when image data is input, thereby acquiring position data of the specified part of the aquatic organism captured in the photograph. The generation unit inputs image data acquired by the first acquisition unit into a generation model that generates segmented images of aquatic organisms based on input image data, thereby generating the captured segmented images of the aquatic organisms; and The inference unit infers the size of the captured aquatic organism based on the location data of the specified region and the segmented image. The inference device inputs image data captured by a stereo camera into the first learning model, calculates the three-dimensional position of the snout and tail fork based on the obtained position data of the specified parts, generates a body height auxiliary line based on the calculated three-dimensional position, and projects the generated body height auxiliary line onto the segmented image in two dimensions to calculate the body height.

Images