A method and system for acquiring 3D phenotype of sphenotyped fish based on monocular multi-view

Abstract

The present invention discloses a method and system for acquiring 3D phenotypes of a fish of the family Catulidae based on monocular multi-views, the method comprising: acquiring video data of the fish of the family Catulidae; reconstructing and denoising a three-dimensional model of the fish of the family Catulidae based on the acquired video data of the fish of the family Catulidae in combination with a multi-eye stereo vision algorithm; inputting the denoised three-dimensional point cloud data of the fish of the family Catulidae into a trained segmentation model, and outputting phenotypic traits of different parts of the fish of the family Catulidae; training the segmentation model, comprising: constructing a training set and a test set of the fish of the family Catulidae; constructing a segmentation model based on a PointNet++ algorithm, including point set feature learning and point feature propagation; training the constructed segmentation model using a training set, and testing the trained segmentation model using a test set, and outputting the trained segmentation model. The present invention obtains a three-dimensional model of the fish of the family Catulidae through three-dimensional reconstruction, removes background noise through a filtering algorithm, and finally obtains the phenotypic traits of the fish of the family Catulidae through a segmentation model based on a deep learning algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of agricultural information technology, and in particular relates to a method for measuring and acquiring phenotypic characteristics of a fish of the family Catulidae by integrating 2D / 3D computer vision technology, and specifically provides a method and system for acquiring 3D phenotypes of a fish of the family Catulidae based on monocular multi-views. Background Art

[0002] Under the low oxygen and low temperature environment of the plateau, the fish of the family Catulidae have evolved some characteristics that can adapt to various environments. By analyzing the Catulidae fish of different biogeographic and environmental distribution, the study of their environmental mutual assistance, reproduction and breeding, and changes in fish community structure are of great significance.

[0003] According to current research, the research technology for automatic phenotypic measurement in three-dimensional space is relatively scarce, and it is difficult to ensure that the phenotypic trait values ​​can be obtained from every shooting angle in two-dimensional graphics. The information on the two-dimensional image is planar, and the length or width of various fins of the family of fish can only be measured in a single measurement with small errors. When obtaining the top, front, and side views of the fish, it is also difficult to ensure that the fins are fully displayed in the three images, and there is a possibility of curling. It is difficult to ensure the accuracy of the subsequent measurement of traits when obtaining data, and it is difficult to avoid human intervention. The image data set obtained is used in the artificial intelligence algorithm, resulting in data errors in the subsequent prediction of the positions of various traits of the fish, and the measurement results obtained are quite different from the actual situation. In addition, based on the obtained two-dimensional image data of the fish, it is difficult to obtain some phenotypic information, such as the chest length information of the fish, the surface area information of the fish, the volume information of the fish, and other phenotypic data, which are very important for the body condition detection of the fish. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method and system for acquiring 3D phenotypes of Tucatidae fish based on monocular multi-views, which obtains a three-dimensional model of Tucatidae fish through three-dimensional reconstruction, removes background noise through a filtering algorithm, and finally obtains the phenotypic traits of Tucatidae fish through a segmentation model based on a deep learning algorithm.

[0005] According to one aspect of the present invention, a method for acquiring 3D phenotype of a fish of the family Catanidae based on a monocular multi-view is provided, comprising:

[0006] Get video data of fish of the family Catanidae;

[0007] Based on the acquired video data of the family Catulidae, combined with the multi-eye stereo vision algorithm, the three-dimensional model of the family Catulidae is reconstructed and denoised;

[0008] The denoised three-dimensional point cloud data of the family Catulidae is input into the trained segmentation model to output the phenotypic traits of different parts of the family Catulidae; wherein the training of the segmentation model includes:

[0009] Construct training and testing sets of 3D point cloud data samples of Cichlidae fish;

[0010] Building a segmentation model based on the PointNet++ algorithm, the segmentation model includes a point set feature learning module and a point feature propagation module;

[0011] The constructed segmentation model is trained using the training set, and the trained segmentation model is tested using the test set, and the trained segmentation model is output.

[0012] As a further technical solution, the method further includes:

[0013] The lengths of the real reference object and the reference object in the three-dimensional model are respectively obtained to obtain the proportional relationship between the real reference object and the reference object in the three-dimensional model;

[0014] According to the proportional relationship and in combination with the phenotypic shapes of different parts of the fish of the family Catulidae, the phenotypic data of different parts of the fish of the family Catulidae are obtained.

[0015] As a further technical solution, reconstruction of the three-dimensional model of the family of fish is carried out, including:

[0016] According to the acquired video data of the catfish, pictures of each catfish at different rotation angles are obtained to form an image data set of the catfish;

[0017] Performing sparse reconstruction according to the image dataset of the fish of the family Catanidae;

[0018] Based on the sparse reconstruction results and combined with the multi-eye stereo vision algorithm, dense reconstruction is performed to obtain the reconstructed three-dimensional model of the Ctenopharyngodontidae fish.

[0019] As a further technical solution, the reconstructed 3D model of the family of fish is denoised, including:

[0020] Using texture mapping, the reconstructed mesh data and texture image are mapped into mesh data containing RGB information;

[0021] Segmentation is performed according to the random sampling consistency algorithm to obtain the segmentation foreground and background;

[0022] Based on the grid data containing RGB information, the background point cloud is removed by combining the HSV space color segmentation algorithm;

[0023] Statistical filtering and Euclidean clustering are used to further remove background noise.

[0024] As a further technical solution, after obtaining the removed background point cloud, it also includes: extracting the reference object in the three-dimensional model based on the color segmentation method; measuring the length, height or radius of the reference object in the three-dimensional model.

[0025] As a further technical solution, in the segmentation model constructed based on the PointNet++ algorithm, the point set feature learning module includes two set abstraction layers, each of which includes sampling, grouping and PointNet layers; the input of the point feature propagation module is the number of channels and multiple multi-layer perceptron structures, and a hierarchical propagation strategy based on distance interpolation and cross-level jump links is adopted, and the output is the score of the category of each point.

[0026] According to one aspect of the present invention, a system for acquiring 3D phenotypes of fish of the family Ctenidae based on monocular multi-view is provided, comprising:

[0027] An input module is used to obtain video data of fish of the family Catanidae;

[0028] The reconstruction and denoising module is used to reconstruct and denoise the three-dimensional model of the catfish based on the acquired catfish video data and combined with the multi-eye stereo vision algorithm;

[0029] The part segmentation module is used to input the denoised three-dimensional point cloud data of the family of fish into the trained segmentation model, and output the phenotypic traits of different parts of the family of fish; wherein the training of the segmentation model includes:

[0030] Construct training and testing sets of 3D point cloud data samples of Cichlidae fish;

[0031] Building a segmentation model based on the PointNet++ algorithm, the segmentation model includes a point set feature learning module and a point feature propagation module;

[0032] The constructed segmentation model is trained using the training set, and the trained segmentation model is tested using the test set, and the trained segmentation model is output.

[0033] As a further technical solution, the system further includes a phenotype measurement module, which is used to execute the following instructions:

[0034] The lengths of the real reference object and the reference object in the three-dimensional model are respectively obtained to obtain the proportional relationship between the real reference object and the reference object in the three-dimensional model;

[0035] According to the proportional relationship and in combination with the phenotypic shapes of different parts of the fish of the family Catulidae, the phenotypic data of different parts of the fish of the family Catulidae are obtained.

[0036] According to one aspect of the present specification, there is provided an electronic device, comprising: at least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the described method.

[0037] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the method described above.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1) The present invention uses a 3D reconstruction method to obtain a 3D model of a fish of the family Catulidae, uses point cloud filtering to remove noise, obtains data for segmentation, and performs component segmentation on the fish of the family Catulidae by using the segmentation task in the PointNet++ model to obtain the phenotypic traits of different parts of the fish of the family Catulidae. Furthermore, the present invention can also measure the phenotypic traits after the component segmentation, and obtain the volume, surface area and other data of the fish in the 3D model.

[0040] 2) The present invention obtains images of the Ctenopharyngodontidae fish at different angles through video and performs three-dimensional reconstruction based on the image data. Compared with the existing method of obtaining point cloud data by laser scanning, the present invention can ensure that the contact time between people and fish is short, and ensure that the body condition of the fish is only slightly affected, so that the fish can still grow healthily when it returns to its growth environment.

[0041] 3) The present invention can reconstruct a clear three-dimensional model through the cooperation of sparse reconstruction and dense reconstruction.

[0042] 4) The present invention uses a method of performing layer-by-layer denoising after texture mapping, which can efficiently, quickly and accurately achieve point cloud denoising.

[0043] 5) The present invention is efficient in segmenting the phenotypes of the same anterior bream and has a certain generalization ability.

[0044] 6) The present invention can achieve rapid and accurate phenotypic measurement through the proportional relationship between the real reference object and the reference object in the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A schematic flow chart of a method for acquiring phenotypes of Cichlidae fish based on monocular multi-views provided in an embodiment of the present invention.

[0047] Figure 2 A schematic diagram of a video recording device for Cichlidae fish provided in an embodiment of the present invention.

[0048] Figure 3 A schematic diagram of reconstructing a three-dimensional model of a fish of the family Catanidae provided in an embodiment of the present invention.

[0049] Figure 4 A schematic diagram of a denoising process provided by an embodiment of the present invention.

[0050] Figure 5 A schematic diagram of a labeled three-dimensional point cloud image for phenotypic segmentation of a fish of the family Tuna provided in an embodiment of the present invention.

[0051] Figure 6 A schematic diagram of the three-dimensional reconstruction effect of a fish of the family Catanidae provided in an embodiment of the present invention.

[0052] Figure 7 A schematic diagram of the structure of a system for acquiring phenotypes of Cichlidae fish based on monocular multi-views provided in an embodiment of the present invention.

[0053] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] At present, there is some research progress in the measurement of fish body size at home and abroad, but there are deficiencies in the measurement of fish phenotypic traits by obtaining the three-dimensional spatial model of fish. The present invention obtains the three-dimensional model of fish by adopting three-dimensional reconstruction technology, removes the noise of fish by filtering algorithm, and finally processes the three-dimensional information of fish by deep learning algorithm to obtain the phenotypic traits of fish, such as fish head, tail, pectoral fin, pelvic fin, anal fin, dorsal fin, etc. The present invention provides effective technical support for the breeding and cultivation of fish by studying the automatic acquisition of phenotypes of fish of the family Catulidae.

[0055] It should be noted that the 3D point cloud information can capture the overall appearance and details of the fish, provide higher accuracy and more morphological information, and the 3D point cloud data of the fish body can measure complex indicators such as volume, surface area and 3D morphological characteristics. Generally speaking, after obtaining the 3D data of the fish, we also need to deal with the noise point problem. At present, many laser scanning devices or depth cameras can be used to obtain the 3D appearance of an object, thereby obtaining the point cloud morphology of the object, that is, the 3D coordinate information of the object. However, since the laser scanning method takes a long time to obtain the point cloud, the contact time with the fish is too long, which affects the body condition of the fish. The use of video acquisition can ensure that the contact time between people and fish is short, and the body condition of the fish is only slightly affected, so that the fish can still grow healthily when it returns to its growth environment.

[0056] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0057] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely exemplary and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps may be further decomposed, while some operations / steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0059] The embodiment of the present invention provides a method for obtaining 3D phenotype of a fish of the family Ctenidae based on a monocular multi-view. Figure 1 As shown, including:

[0060] Step 1, obtaining video data of fish of the family Catanidae;

[0061] Step 2, based on the acquired video data of the catfish, combined with a multi-eye stereo vision algorithm, reconstruct and denoise the three-dimensional model of the catfish;

[0062] Step 3: Input the denoised 3D point cloud data of the Catulidae fish into the trained segmentation model to output the phenotypic traits of different parts of the Catulidae fish.

[0063] Among them, the training of the segmentation model includes: constructing a training set and a test set of three-dimensional point cloud data samples of Tuatara fish; constructing a segmentation model based on the PointNet++ algorithm, and the segmentation model includes a point set feature learning module and a point feature propagation module; using the training set to train the constructed segmentation model, and using the test set to test the trained segmentation model, and outputting the trained segmentation model.

[0064] The embodiment of the present invention uses a 3D reconstruction method to obtain a 3D model of a fish of the family Catulidae, uses point cloud filtering to remove noise, obtains data for segmentation, and performs component segmentation on the fish of the family Catulidae by using the segmentation task in the PointNet++ model to obtain the phenotypic traits of different parts of the fish of the family Catulidae. Compared with the prior art, the present invention has the advantages of short contact time with the fish, clear reconstructed 3D model, efficient point cloud denoising, and efficient phenotypic segmentation.

[0065] Based on the phenotypic traits of different parts of the Tuna fish obtained in the aforementioned embodiment, the method described in the embodiment of the present invention also includes: respectively obtaining the length of the real reference object and the reference object in the three-dimensional model to obtain the proportional relationship between the real reference object and the reference object in the three-dimensional model; according to the proportional relationship, combined with the phenotypic shapes of different parts of the Tuna fish, the phenotypic data of different parts of the Tuna fish are obtained.

[0066] The embodiment of the present invention obtains the size of the reference object in the three-dimensional model based on the phenotypic shape obtained above, as well as the background point cloud data after the three-dimensional model reconstruction and denoising, and then combines the size of the real reference object to measure the phenotypic traits after the component segmentation, thereby obtaining the volume, surface area and other data of the fish. Compared with the existing technology, it has the advantage of fast and accurate phenotypic measurement.

[0067] As a preferred embodiment, the method for obtaining 3D phenotype of a fish of the family Ctenidae based on monocular multi-view provided in the embodiment of the present invention further includes, before step 1:

[0068] Set up the environment for shooting fish, which requires a slow rotating table with controllable speed, a base plate, a flat table top, an LED light with adjustable light intensity, an industrial camera, an adjustable bracket for fixing the industrial camera, and a regular reference object for measuring length.

[0069] After setting up the shooting environment for the fish, the fish of the family Channa are shot to obtain a video. Specifically, through the proof of the previous experiment, the appropriate light intensity and the appropriate camera position are selected, and the duration of shooting a 360-degree panoramic view of a fish is determined.

[0070] It should be noted that in order to avoid the reflection problem caused by the fish surface being too wet, it is necessary to wipe the fish surface with a soft paper towel to avoid poor reconstruction effect and affect the subsequent reconstruction results. In addition, in order to avoid the death of fish due to lack of water and oxygen in the shooting site, it is necessary to prepare a small water-filled container with good oxygen in the water so that the fish can be placed in the water in time after the shooting is completed.

[0071] As an implementation method, using Figure 2 The platform shown in the figure, after ensuring that the shooting environment has negligible impact on outdoor light, uses a camera and a rotating circular platform to obtain video data of about 300 fish. Among them, each video obtained using a 360-degree rotating platform is about 100 seconds (this duration is the result of a preliminary experiment under the premise of preparing to shoot a large number of fish of the family Cichlidae. Too short a time will result in poor intercepted frames, and too long a time will easily affect the body condition of the fish). After the shooting is completed, each fish photographed is numbered to facilitate the subsequent work.

[0072] As a preferred embodiment, the embodiment of the present invention provides a method for obtaining 3D phenotypes of Tuna fish based on monocular multi-views, wherein step 2 also includes: performing three-dimensional reconstruction using the software Colmap tool in combination with the open source library OpenMVS that integrates the MVS algorithm (multi-eye stereo vision) until the three-dimensional models of all the fish are reconstructed.

[0073] Optionally, step 2 specifically includes: step 2.1, sparse reconstruction; step 2.2, dense reconstruction.

[0074] Specifically, after selecting suitable fish videos, the Colmap tool is used to perform sparse reconstruction of the fish of the family Catanidae. The main function of sparse reconstruction is to locate and reconstruct the three-dimensional coordinates of some image feature points.

[0075] As an implementation method, Figure 3 As shown, the process of sparse reconstruction includes:

[0076] Step 2.11, use ffmpeg (video processing tool) to get pictures of each fish at different rotation angles, and use the parameter setting of capturing one picture every 1 second to get an image dataset. Taking a 100 second long video as an example, a dataset of 100 images is finally obtained. Each image in this image set contains the camera intrinsic parameter and camera extrinsic parameter information.

[0077] Step 2.12, enter the sparse reconstruction process in Colmap, and use the command line program to operate Colmap to reduce manual real-time viewing and operation in the GUI. Sparse reconstruction uses the incremental SfM algorithm (Structure from Motion) to obtain the sparse point cloud and camera pose information of the fish. First, obtain the file path of the multi-view image set, then use the corresponding point search method to perform feature extraction and feature matching, then perform matching optimization based on geometric conditions, and establish a scene graph. Then enter the reconstruction of the incremental SfM algorithm, first initialize the algorithm, then use triangulation and pose estimation to obtain the overall shape of the fish, and finally use the bundle adjustment method to optimize it to obtain a sparse point cloud, that is, the fuzzy shape of the three-dimensional model.

[0078] Furthermore, after obtaining the sparse point cloud, the MVS algorithm (multi-view stereo vision) is used to obtain the dense point cloud of the catfish. Before dense reconstruction, the OpenMVS open source library and configuration files are downloaded, and the image files to be reconstructed and the camera pose data file paths are set. The dense reconstruction process still uses the control command line to set parameters and save file paths, avoiding multiple manual operations, reducing the interaction time between people and computers and other devices, and realizing a simple end-to-end process.

[0079] As an implementation method, Figure 3As shown in Figure 2, the dense reconstruction process includes:

[0080] Step 2.21, obtain the Colmap interface, extract sparse point cloud and camera parameters, etc.

[0081] Step 2.22, dense reconstruction obtains a point cloud that is as complete as possible. Under the known camera parameters of the image, find points in space with photometric consistency, and then perform scene stereo matching by finding matching corresponding points from images of different perspectives to restore three-dimensional information from two-dimensional images.

[0082] Step 2.23, perform mesh surface reconstruction, and perform surface reconstruction by estimating the best receiving input point cloud surface to obtain a three-dimensional model of the mesh data class of the family Tuatara.

[0083] Step 2.24, refine the mesh. Since the model obtained in the previous step has many coarse meshes on the surface and there is still a big gap compared to the real three-dimensional structure, mesh refinement is used. The obtained model is consistent with the real fish of the shooting scene in details.

[0084] Step 2.25, calculate clear and accurate texture. In order to obtain the color information of the fish surface, texture processing is required.

[0085] Step 2.26, finally color the mesh. The mesh coloring process uses an algorithm to map the texture information obtained in the previous step to the mesh data of the fish, so that the model viewed using the 3D model visualization tool has clear texture and color.

[0086] Step 2.27, organize the entire process as a pipeline, set the parameters of each link, and run it with one click to achieve an end-to-end reconstruction process, and finally obtain a three-dimensional model of the catfish, reducing human-computer interaction operations.

[0087] As a preferred embodiment, the embodiment of the present invention provides a method for acquiring 3D phenotype of Tussidae fish based on monocular multi-view, wherein step 2 further includes: step 2.3, a noise removal process of Tussidae fish.

[0088] Specifically, after obtaining the 3D model of the sphenoid fish, it is necessary to segment the background and foreground for reconstruction. The foreground includes the sphenoid fish, while the background is mostly the rotating platform surface and reference objects. The reference objects will be processed separately later.

[0089] As an implementation method, Figure 4 As shown in Figure 1, the noise removal process of the catfish includes:

[0090] Step 2.31, using texture mapping, maps the mesh data and texture image obtained in the above reconstruction into a mesh data containing RGB information. The specific ideas include: reading the file saved by the three-dimensional mesh and the texture image information, respectively obtaining the vertex and face data of the mesh data, and obtaining the color information of the texture image, and then mapping the UV coordinates to the pixel position of the texture image, mapping the UV coordinates of the triangular patch to the pixel position on the texture image, and obtaining the color information on the texture image. Finally, to process the patch information of the mesh data, it is necessary to traverse the data of each patch, extract its vertex UV coordinates, and combine them with the vertex index of the patch according to the mapped pixel position to form the final patch data, and obtain each point and its corresponding RGB information.

[0091] Step 2.32, use the Python script to process the data file saved in step 2.31 and convert it into a pcd format (data storage format, containing information such as x, y, z, r, g, b). After step 2.31 is completed, the data file storage characteristics are the spatial coordinate information of the XYZ axis of one point per line and the number of rows corresponding to the RGB color information, so it is very simple to use the script for batch processing.

[0092] Step 2.33, use the RANSAC algorithm (random sampling consensus algorithm) to get the plane equation, and then use the plane equation to segment the foreground and background, save them in pcd format, and record the plane equation information. The specific process of this algorithm is: use the open3d library to read the file, use the RANSAC algorithm to segment the plane, including parameters (threshold of point-to-plane distance, minimum number of points for fitting the plane, maximum number of iterations of the algorithm), get the plane equation coefficient, extract the point cloud in the plane, and the remaining point cloud, which is the main data information of the three-dimensional model. At the same time, since the plane also contains some point cloud data of the fish close to the bottom surface, further denoising is required.

[0093] Step 2.34, since the RGB information of the three-dimensional coordinate points is obtained by step 2.31, the HSV space color segmentation algorithm is used to process the foreground and background point clouds, and the background point clouds are removed by setting appropriate H, S, and V parameter thresholds. In step 2.32, we obtain the RGB information of each point, convert RGB to obtain H, S, and V information, that is, hue value, saturation value, and tone value. Only points that meet the set threshold can be retained, and all other points that do not meet the threshold are removed. However, at the same time, the background points close to the Tuatridae have similar RGB information to the Tuatridae. If the thresholds of H, S, and V are changed, the Tuatridae points will be removed together with the background points, which will affect the next step of phenotypic segmentation, so further processing is required.

[0094] Step 2.35, use statistical filtering to remove the remaining noise. In step 2.34, the object point cloud in the foreground needs to be merged with the object point cloud in the background to obtain the complete three-dimensional point cloud data of the catfish. The background points that are not completely filtered in the background, that is, the noise, are also integrated into the entire three-dimensional model. The statistical filtering algorithm can be used to remove these background noise points that are close to the catfish. The overall steps of the algorithm are: first, a KD tree is established to accelerate the nearest neighbor search, and the sum of the distances to the nearest zero point is calculated for each point, and a threshold is calculated based on these distance values, and then the points with a distance less than the threshold are filtered according to this threshold as the output after filtering.

[0095] Step 2.36, due to the color information caused by the environment in the three-dimensional reconstruction process, when using the above five steps, there will be such a situation that when using spatial color segmentation, the background surface information far away from the object cannot be completely removed. By using the Euclidean clustering segmentation algorithm, two point clusters far apart in three-dimensional space can be separated, so that the fish of the family of catfish can be separated from the distant background points, and the final three-dimensional point cloud of the fish of the family of catfish can be obtained. The process of the Euclidean clustering segmentation algorithm is: read the point cloud data from the given data file path, and convert the coordinates and color information of the point cloud into NumPy arrays. Further, the DBSCAN algorithm (density-based spatial clustering algorithm) is used to cluster the point cloud, and the clustering parameters are determined by setting the search radius and the minimum number of samples. Next, the algorithm divides the point cloud into multiple clusters according to the clustering labels, that is, the segmentation results are obtained, and they are saved in sequence (because the reference object needs to be measured, it needs to be saved as the actual measurement value to compare with the measurement value in the point cloud).

[0096] Step 2.37, in the point cloud of the removed background plane, the 3D model of the reference object is included. The reference object (the color difference between the reference object and the background plane is large) is extracted by using the color segmentation method. The height and radius length of the reference object are measured using the CloudCompare tool. In the real reference object, the real height and radius length of the reference are measured using precision measuring instruments, such as vernier calipers, to obtain the proportional relationship between the real length and the length in the 3D model after 3D reconstruction.

[0097] The effect of 3D model reconstruction is shown in the figure Figure 6 shown.

[0098] As a preferred embodiment, the embodiment of the present invention provides a method for obtaining 3D phenotypes of Tufa fish based on monocular multi-views, wherein step 3 further includes: performing phenotypic segmentation on Tufa fish using a deep learning algorithm. The phenotypic trait segmentation of Tufa fish is performed using the component segmentation task in the PointNet++ model.

[0099] As an implementation method, the training of the segmentation model includes:

[0100] Step 3.1, use the method of step 1 to obtain the video of the family of fish, and use the method of step 2 to obtain the reconstructed and denoised three-dimensional point cloud data to form a data set for training and testing, and then divide the data set into a test set and a training set in an 8:2 ratio according to the allocation idea of ​​each model data set in the deep learning algorithm.

[0101] Step 3.2, before labeling, the point cloud of the dataset is downsampled, and then the point clouds of the two datasets are classified and labeled, such as Figure 5 The labels are divided into 8 categories, namely the fish's head, trunk, tail, pectoral fin A, pectoral fin B, pelvic fin A, pelvic fin B, and dorsal fin.

[0102] Step 3.3, after using the annotation tool CloudCompare to classify and annotate the point cloud of 300 fish, check the data format to ensure that the point data can be obtained through this file format during training. The PointNet++ model is trained for 200 rounds in total, and the training weight file is obtained after the training is completed. The weight file is then used for prediction, and the prediction is as above. Figure 5 The phenotypic segmentation results are obtained, and the head, tail, dorsal fin, pectoral fin, and pelvic fin of the fish of the family Tuatariidae are obtained. In the subsequent processing, the length or width value of different phenotypes is measured by using the proportional relationship between the real object and the reconstructed three-dimensional space.

[0103] The segmentation model is implemented using the PointNet++ algorithm. Its segmentation task is mainly divided into two modules. The first module is point set feature learning, which includes two set abstraction layers. The set abstraction layer includes sampling, grouping, and PointNet layers. The input is N(d+C) and the output is N'(d+C'). N is the number of sampling points, d is the coordinate information, C is the number of features, N' is the number of sub-sampled points, and C' is the number of new features. The input size of the sampling layer is 2048 (the number of points). Iterative farthest point sampling is used to obtain N', which is 512 sampling points. The input and output of the grouping layer are N (d+C), N'´ K´(d+C), K is the number of points in the area of ​​the centroid point, set to 3, and the grouping strategy is based on radius ball query. In the PointNet layer, the input is the point set after sampling and grouping, that is, N'´ K'(d+C), and the output is N'(d +C'). The first step is to convert the coordinates of the local area points into the local coordinate system relative to the centroid point. The second step is to pass through the multi-layer perceptron network (MLP), and the third step is to perform maximum pooling (MaxPooling) on ​​the features. The second module is the feature propagation of the points. The input is the number of channels and multiple multi-layer perceptron structures. It adopts a hierarchical propagation strategy based on distance interpolation and cross-level jump links. The connection features use the unit point network (Unit Pointnet), which is mainly composed of Conv1d (1´1 convolution), BatchNorm (batch normalization), and Dropout. It is used for the shared fully connected layer and ReLU (activation function) layer to update the feature vector of the point. Repeat the process of the second module above. The final output result is the score of the category of each point.

[0104] As a preferred embodiment, the embodiment of the present invention provides a method for acquiring 3D phenotypes of Ctenopharyngodontidae fish based on monocular multi-views. After obtaining the predicted segmentation result, the classified point cloud is used to measure the length and width, and the length of each reference object is measured to obtain the ratio of each true value to the measured value in the three-dimensional reconstructed model. The true length and width of each category (i.e., various traits of the fish) are calculated by using the ratio.

[0105] Specifically, by comparing the length of the reference object in the three-dimensional model with the length of the reference object in the real model, a proportional relationship is obtained. The length of each segmented fish phenotype, such as the fish head, trunk, and tail, is measured, and the true value of the fish phenotype is measured using the proportional relationship obtained by measuring the reference object in the above-mentioned method.

[0106] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a 3D phenotype acquisition system for a fish of the family Catulidae based on a monocular multi-view, which is used to execute the 3D phenotype acquisition method for a fish of the family Catulidae based on a monocular multi-view in the above method embodiment.

[0107] See also Figure 7 The system includes: an input module for acquiring video data of the family of fish; a reconstruction and denoising module for reconstructing and denoising a three-dimensional model of the family of fish based on the acquired video data of the family of fish in combination with a multi-eye stereo vision algorithm; a part segmentation module for inputting the denoised three-dimensional point cloud data of the family of fish into a trained segmentation model, and outputting phenotypic traits of different parts of the family of fish; wherein the training of the segmentation model includes: constructing a training set and a test set of three-dimensional point cloud data samples of the family of fish; constructing a segmentation model based on the PointNet++ algorithm, the segmentation model including a point set feature learning module and a point feature propagation module; using the training set to train the constructed segmentation model, and using the test set to test the trained segmentation model, and outputting the trained segmentation model.

[0108] The embodiment of the present invention provides a 3D phenotype acquisition system for Ctenidae fish based on monocular multi-view, which is aimed at phenotype acquisition and measurement of Ctenidae fish. Figure 7 Several modules in the 3D reconstruction method are used to obtain the 3D model of the Catulidae fish, and point cloud filtering is used to remove noise to obtain data for segmentation. The Catulidae fish are segmented into parts by using the segmentation task in the PointNet++ model to obtain the phenotypic traits of different parts of the Catulidae fish.

[0109] It should be noted that the system embodiment provided by the present invention is used to implement the methods in the above method embodiment as well as the methods in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set. The principle is basically the same as the principle of the above system embodiment provided by the present invention. As long as the technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above system embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining the technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above system embodiment to obtain the corresponding system class embodiments, which are used to implement the methods in other method class embodiments. For example:

[0110] Based on the content of the above system embodiment, as a preferred embodiment, a 3D phenotype acquisition system for Cichlidae fish based on monocular multi-view provided in the embodiment of the present invention further includes a phenotype measurement module for executing the following instructions:

[0111] The lengths of the real reference object and the reference object in the three-dimensional model are respectively obtained to obtain the proportional relationship between the real reference object and the reference object in the three-dimensional model;

[0112] According to the proportional relationship and in combination with the phenotypic shapes of different parts of the fish of the family Catulidae, the phenotypic data of different parts of the fish of the family Catulidae are obtained.

[0113] Based on the contents of the above system embodiments, as a preferred embodiment, a 3D phenotype acquisition system for fish of the family Ctenidae based on monocular multi-view is provided in the embodiments of the present invention.

[0114] Based on the content of the above system embodiment, as a preferred embodiment, a 3D phenotype acquisition system for Cichlidae fish based on monocular multi-view is provided in the embodiment of the present invention, wherein the reconstruction and denoising module further includes:

[0115] According to the acquired video data of the catfish, pictures of each catfish at different rotation angles are obtained to form an image data set of the catfish;

[0116] Performing sparse reconstruction according to the image dataset of the fish of the family Catanidae;

[0117] Based on the sparse reconstruction results and combined with the multi-eye stereo vision algorithm, dense reconstruction is performed to obtain the reconstructed three-dimensional model of the Ctenopharyngodontidae fish.

[0118] Based on the content of the above system embodiment, as a preferred embodiment, a 3D phenotype acquisition system for Cichlidae fish based on monocular multi-view is provided in the embodiment of the present invention, wherein the reconstruction and denoising module further includes:

[0119] Using texture mapping, the reconstructed mesh data and texture image are mapped into mesh data containing RGB information;

[0120] Segmentation is performed according to the random sampling consistency algorithm to obtain the segmentation foreground and background;

[0121] Based on the grid data containing RGB information, the background point cloud is removed by combining the HSV space color segmentation algorithm;

[0122] Statistical filtering and Euclidean clustering are used to further remove background noise.

[0123] Based on the contents of the above system embodiments, as a preferred embodiment, a 3D phenotype acquisition system for Ctenopharyngodontidae based on monocular multi-view is provided in the embodiments of the present invention, and the reconstruction and denoising module also includes: extracting reference objects in the three-dimensional model based on a color segmentation method; measuring the length, height or radius of the reference objects in the three-dimensional model.

[0124] Based on the content of the above system embodiment, as a preferred embodiment, a 3D phenotype acquisition system for Cichlidae fish based on monocular multi-view is provided in the embodiment of the present invention, wherein the segmentation module further includes:

[0125] The point set feature learning includes two set abstraction layers, each of which includes sampling, grouping and PointNet layers; the input of the point feature propagation is the number of channels and multiple multi-layer perceptron structures, and a hierarchical propagation strategy based on distance interpolation and cross-level skip links is adopted, and the output is the score of the category of each point.

[0126] The method of the embodiment of the present invention is implemented by relying on electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 8 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. At least one processor calls the logic instructions in at least one memory to execute all or part of the steps of the method provided by the aforementioned various method embodiments.

[0127] In addition, when the logic instructions in the above-mentioned at least one memory are implemented in the form of a software functional unit and sold or used as an independent product, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or partly in other words, the part that contributes to the prior art or part of the technical solution is embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (a personal computer, a server, or a network device) to perform all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and various media for storing program codes.

[0128] The system embodiments described above are merely illustrative, wherein the units described as separate components are or are not physically separated, and the components shown as units are or are not physical units, located in one place, or distributed on multiple network units. Some or all of the modules are selected according to actual conditions to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement without creative effort.

[0129] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for acquiring 3D phenotype of a fish of the family Catanidae based on monocular multi-view, characterized in that: include: Use cameras and a 360-degree rotating circular platform to obtain video data of catfish; Based on the acquired video data of the family Catulidae, combined with the multi-eye stereo vision algorithm, the three-dimensional model of the family Catulidae is reconstructed and denoised; The reconstruction of the 3D model of the family Chaetodontidae includes sparse reconstruction and dense reconstruction; The sparse reconstruction process includes: first obtaining the file path of the multi-view image set, then using the corresponding point search method to perform feature extraction and feature matching, followed by matching optimization based on geometric conditions and establishing a scene graph; then entering the incremental SfM algorithm for reconstruction, first initializing the algorithm, then using triangulation and posture estimation to obtain the overall shape of the fish, and finally using the bundle adjustment method for optimization to obtain a sparse point cloud; The dense reconstruction process includes: obtaining the Colmap interface and extracting the sparse point cloud; obtaining the most complete point cloud possible through dense reconstruction, finding points with photometric consistency in space under the known camera parameters of the image, and then performing scene stereo matching by finding matching corresponding points from images of different perspectives and recovering three-dimensional information from two-dimensional images; performing mesh surface reconstruction by estimating the best receiving input point cloud surface and performing surface reconstruction to obtain a three-dimensional model of the mesh data class of the family of fish; refining the mesh; processing the texture; coloring the mesh, the mesh coloring process is to map the texture information obtained in the previous step to the mesh data of the fish using an algorithm, so that the model viewed using a three-dimensional model visualization tool has clear texture and color; The denoised three-dimensional point cloud data of the family Catulidae is input into the trained segmentation model to output the phenotypic traits of different parts of the family Catulidae; wherein the training of the segmentation model includes: Construct training and testing sets of 3D point cloud data samples of Cichlidae fish; A segmentation model is constructed based on the PointNet++ algorithm, and the segmentation model includes a point set feature learning module and a point feature propagation module; in the point set feature learning module, the coordinates of the points in the local area are converted into a local coordinate system relative to the centroid point, and then the features are max-pooled after passing through a multi-layer perceptron network; in the point feature propagation module, the input is the number of channels and multiple multi-layer perceptron structures, and a hierarchical propagation strategy based on distance interpolation and cross-level jump links is adopted, while the connection features adopt a unit point network, which is used for the shared fully connected layer and the activation function layer to update the feature vector of the point; The constructed segmentation model is trained using the training set, and the trained segmentation model is tested using the test set, and the trained segmentation model is output.

2. According to claim 1, a method for acquiring 3D phenotype of a fish of the family Ctenidae based on monocular multi-view, characterized in that: The method further comprises: The lengths of the real reference object and the reference object in the three-dimensional model are respectively obtained to obtain the proportional relationship between the real reference object and the reference object in the three-dimensional model; According to the proportional relationship and in combination with the phenotypic shapes of different parts of the fish of the family Catulidae, the phenotypic data of different parts of the fish of the family Catulidae are obtained.

3. According to claim 1, a method for acquiring 3D phenotype of a fish of the family Catanidae based on monocular multi-view, characterized in that: Reconstruction of 3D models of catfish, including: According to the acquired video data of the catfish, pictures of each catfish at different rotation angles are obtained to form an image data set of the catfish; Performing sparse reconstruction according to the image dataset of the fish of the family Catanidae; Based on the sparse reconstruction results and combined with the multi-eye stereo vision algorithm, dense reconstruction is performed to obtain the reconstructed three-dimensional model of the Ctenopharyngodontidae fish.

4. According to claim 3, a method for acquiring 3D phenotype of a fish of the family Ctenidae based on monocular multi-view, characterized in that: De-noising the reconstructed 3D model of the family Catanidae, including: Using texture mapping, the reconstructed mesh data and texture image are mapped into mesh data containing RGB information; Segmentation is performed according to the random sampling consistency algorithm to obtain the segmentation foreground and background; Based on the grid data containing RGB information, the background point cloud is removed by combining the HSV space color segmentation algorithm; Statistical filtering and Euclidean clustering are used to further remove background noise.

5. According to claim 4, a method for acquiring 3D phenotype of a fish of the family Catanidae based on monocular multi-view, characterized in that: After obtaining the removed background point cloud, the method further includes: extracting a reference object in the three-dimensional model based on a color segmentation method; and measuring the length, height or radius of the reference object in the three-dimensional model.

6. The method for acquiring 3D phenotype of a fish of the family Ctenidae based on monocular multi-view according to claim 1, characterized in that: In the segmentation model constructed based on the PointNet++ algorithm, the point set feature learning module includes two set abstraction layers, each of which includes sampling, grouping and PointNet layers; the input of the point feature propagation module is the number of channels and multiple multi-layer perceptron structures, and a hierarchical propagation strategy based on distance interpolation and cross-level jump links is adopted, and the output is the score of the category of each point.

7. A 3D phenotype acquisition system for Cichlidae fish based on monocular multi-view, characterized in that: include: An input module is used to obtain video data of sphenoid fish using a camera and a 360-degree rotating circular platform; The reconstruction and denoising module is used to reconstruct and denoise the three-dimensional model of the catfish based on the acquired catfish video data and combined with the multi-eye stereo vision algorithm; The reconstruction of the 3D model of the family Chaetodontidae includes sparse reconstruction and dense reconstruction; The sparse reconstruction process includes: first obtaining the file path of the multi-view image set, then using the corresponding point search method to perform feature extraction and feature matching, followed by matching optimization based on geometric conditions and establishing a scene graph; then entering the incremental SfM algorithm for reconstruction, first initializing the algorithm, then using triangulation and posture estimation to obtain the overall shape of the fish, and finally using the bundle adjustment method for optimization to obtain a sparse point cloud; The dense reconstruction process includes: obtaining the Colmap interface and extracting the sparse point cloud; obtaining the most complete point cloud possible through dense reconstruction, finding points with photometric consistency in space under the known camera parameters of the image, and then performing scene stereo matching by finding matching corresponding points from images of different perspectives and recovering three-dimensional information from two-dimensional images; performing mesh surface reconstruction by estimating the best receiving input point cloud surface and performing surface reconstruction to obtain a three-dimensional model of the mesh data class of the family of fish; refining the mesh; processing the texture; coloring the mesh, the mesh coloring process is to map the texture information obtained in the previous step to the mesh data of the fish using an algorithm, so that the model viewed using a three-dimensional model visualization tool has clear texture and color; The part segmentation module is used to input the denoised three-dimensional point cloud data of the family of fish into the trained segmentation model, and output the phenotypic traits of different parts of the family of fish; wherein the training of the segmentation model includes: Construct training and testing sets of 3D point cloud data samples of Cichlidae fish; A segmentation model is constructed based on the PointNet++ algorithm, and the segmentation model includes a point set feature learning module and a point feature propagation module; in the point set feature learning module, the coordinates of the points in the local area are converted into a local coordinate system relative to the centroid point, and then the features are max-pooled after passing through a multi-layer perceptron network; in the point feature propagation module, the input is the number of channels and multiple multi-layer perceptron structures, and a hierarchical propagation strategy based on distance interpolation and cross-level jump links is adopted, while the connection features adopt a unit point network, which is used for the shared fully connected layer and the activation function layer to update the feature vector of the point; The constructed segmentation model is trained using the training set, and the trained segmentation model is tested using the test set, and the trained segmentation model is output.

8. The system for acquiring 3D phenotype of a fish of the family Ctenidae based on monocular multi-view according to claim 7, characterized in that: The system also includes a phenotype measurement module, which is used to execute the following instructions: The lengths of the real reference object and the reference object in the three-dimensional model are respectively obtained to obtain the proportional relationship between the real reference object and the reference object in the three-dimensional model; According to the proportional relationship and in combination with the phenotypic shapes of different parts of the fish of the family Catulidae, the phenotypic data of different parts of the fish of the family Catulidae are obtained.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 6.

10. A non-transitory computer readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the method of any one of claims 1 to 6.

Images