A fish, shrimp and shellfish three-dimensional phenotype detection device and method based on multi-view stereo vision

Abstract

The application discloses a fish, shrimp and shellfish three-dimensional phenotype detection device and method based on multi-view stereovision, which comprises a rigid outer frame body, a control module, a sample support frame, a rotary table module, a state monitoring camera and an image acquisition module; the rotary table module drives the image acquisition module to collect and shoot around the sample support frame; the method is characterized in that a dynamic synchronization mechanism of state monitoring by the state monitoring camera and image acquisition by the image acquisition module is adopted to obtain high-quality multi-view images of live aquatic organisms; based on accurate camera pose and sample mask, a parameterized statistical shape model is fused to perform initial contour fitting and joint optimization reconstruction, thereby obtaining a high-precision three-dimensional mesh model; finally, various phenotype parameters are extracted through predefined key points in the model. The application realizes automatic, high-precision and rapid detection of the three-dimensional phenotype of live fish, shrimp and shellfish, and can effectively overcome the measurement interference caused by the breathing movement of fish, shrimp and shellfish and the body surface reflection.

Description

Technical Field

[0001] This invention relates to the field of live aquatic organism measurement technology, specifically to a three-dimensional phenotypic detection device and method for fish, shrimp, and shellfish based on multi-view stereo vision, which is particularly suitable for non-contact, automated three-dimensional morphological measurement and phenotypic parameter extraction of live fish, shrimp, and shellfish. Background Technology

[0002] Morphological phenotypes in fish, shrimp, and shellfish are core indicators in genetic breeding, germplasm evaluation, and healthy aquaculture. Precise parameters such as body length, head length, body color, volume, and surface area are crucial for analyzing the relationship between genotype and phenotype, screening superior varieties, and evaluating aquaculture effectiveness.

[0003] Currently, the measurement of phenotypes in live aquatic organisms mainly relies on contact-based manual measurement methods and two-dimensional image-based methods. However, the former is cumbersome, inefficient, and prone to causing stress damage to the organisms; the latter loses depth information, making it impossible to obtain key three-dimensional parameters such as volume, and the measurement results are easily affected by the shooting angle and the organism's posture, resulting in limited accuracy. These limitations restrict the efficiency and quality of phenotype collection for fish, shrimp, and shellfish. This leads to a lack of information dimensions in the phenotypic datasets upon which breeding analysis depends, particularly the inability to obtain key three-dimensional geometric parameters such as volume, surface area, and surface morphology, making it difficult to construct a complete phenotypic model that accurately correlates with genotype and higher-order economic traits (such as body condition).

[0004] Furthermore, while 3D measurement technologies such as laser scanning and industrial CT exist, their equipment is expensive, their operation is complex, and they are often unsuitable for rapid, non-destructive testing of live aquatic organisms. Therefore, there is a need in this field for a technical solution that can achieve rapid, non-destructive, and high-precision 3D phenotypic detection of live fish, shrimp, and shellfish to overcome the shortcomings of existing technologies in terms of efficiency, accuracy, and information dimension. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a device and method for detecting the three-dimensional phenotype of fish, shrimp and shellfish based on multi-view stereo vision, which can realize automated, high-precision and rapid detection of the three-dimensional phenotype of live fish, shrimp and shellfish.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: I. A 3D Phenotypic Detection Device for Fish, Shrimp, and Shellfish Based on Multi-View Stereo Vision It includes a rigid outer frame, a control module, a weighing module, a sample support frame, a turntable module, a status monitoring camera, and an image acquisition module; The rigid outer frame has a built-in cavity, and the weighing module is arranged inside the cavity. The sample support frame is used to support the test object arranged on the upper end of the weighing module, and the corresponding sample support frame can be replaced according to the type of test object. The two ends of the turntable module are respectively fixed to the inner walls of the corresponding two sides of the cavity. The image acquisition module is mounted on the turntable module and rotates with the turntable module. The status monitoring camera is arranged inside the cavity. The cavity also integrates a control module, which is electrically connected to the weighing module, the turntable module, the image acquisition module, and the status monitoring camera to acquire data from the weighing module, control the movement of the turntable module and receive motion information fed back by the turntable module, control the image acquisition module to acquire images of the test object, and monitor the breathing status of the test object.

[0007] The sample support frame can be replaced with a corresponding structure depending on the object being tested: When the test subject is fish, the sample support frame consists of three independent support components that are parallel to the length of the fish. Two of the independent support components are symmetrically arranged to support the head and body of the fish, and the other independent support component is arranged at the rear to support the tail of the fish. When the test subject is shrimp, the sample support frame is a horizontal platform used to place the shrimp; When the object of the test is a shellfish, the sample support frame consists of two symmetrical clamping components on both sides, each of which is equipped with a flexible pin structure for clamping the shellfish.

[0008] The turntable module includes a first turntable and a second turntable, and the image acquisition module includes a rigid mounting beam and multiple cameras. The first turntable and the second turntable are symmetrically fixed on the inner walls of the corresponding two sides of the cavity, and are connected by a rigid mounting beam with a U-shaped central protrusion. The first turntable and the second turntable are synchronously rotating electric turntables and are both electrically connected to the control module, so that the rigid mounting beam rotates around a horizontal axis. All the cameras are spaced along the length of the rigid mounting beam on the central protrusion and are all facing the sample support frame. The height of the sample support frame is lower than the horizontal axis formed by the first turntable and the second turntable, so that the detection object placed on the sample support frame is on the horizontal axis.

[0009] The status monitoring camera is fixedly connected to the weighing module via an arc-shaped connecting rod, and the status monitoring camera faces the sample support frame for fish to support the head of the fish. The installation of the status monitoring camera does not interfere with the rotation of the turntable module or the image acquisition module. The cavity inside the rigid outer frame is also equipped with a lighting module to provide a light source.

[0010] II. A method for detecting three-dimensional phenotypes of fish, shrimp, and shellfish The detection method is as follows: S1: Place the object to be tested on the corresponding sample support frame; The detection object in step S1 is a live fish, shrimp, or shellfish sample. Step S2 uses different collection methods depending on the type of detection object in step S1. When the object of detection is a live fish sample, the state monitoring camera monitors the gill cover status of the live fish sample, and the image acquisition module acquires images only when the gill cover is closed, thereby obtaining a multi-view image sequence of the live fish sample. When the object of detection is a live shrimp or shellfish sample, it is not necessary to monitor it through the status monitoring camera. Instead, the image acquisition module can synchronously acquire images at a predetermined angle to obtain a multi-view image sequence of the live shrimp or shellfish sample. S2: The status monitoring camera monitors the detection object. Based on the category of the detection object and the detection result of the status monitoring camera, the turntable module is selectively controlled to rotate to drive the image acquisition module to acquire images of the detection object and obtain a multi-view image sequence composed of multiple multi-view images. S3: Determine the camera pose corresponding to each of the multi-view images based on the rotation angle information of the image acquisition module and the fixed relative pose parameters pre-calibrated by each camera in the image acquisition module. The specific method for determining the camera pose in step S3 is as follows: a world coordinate system is established with the intersection of the horizontal axis formed by the first turntable and the second turntable in the image acquisition module and the center of the sample support frame as the origin of the world coordinate system; the rigid transformation matrix corresponding to the image acquisition module in the world coordinate system is calculated according to the rotation angle of the first turntable and the second turntable; and the camera pose of each camera in each of the multi-view images is calculated according to the preset extrinsic parameters of each camera relative to the rigid mounting beam of the image acquisition module. S4: Perform foreground segmentation on each multi-view image in the multi-view image sequence and extract the sample pixel region mask corresponding to each multi-view image; S5: Using the multi-view images, the corresponding camera poses, and the corresponding sample pixel region masks, three-dimensional reconstruction is performed by fusing the preset parameterized model of the corresponding detection object to generate an optimized three-dimensional mesh model. Step S5 specifically involves: Starting with the pre-defined parametric model of the corresponding detection object, initial registration is performed using a contour-based fitting method. The projected contour of the pre-defined parametric model under each camera pose is rendered, and the difference between the pre-defined parametric model and the corresponding sample pixel region mask is minimized to obtain a preliminary 3D mesh model. The preliminary 3D mesh model is then simultaneously optimized using a joint optimization algorithm to obtain an optimized 3D mesh model. The joint optimization algorithm obtains the optimal 3D mesh model by minimizing a composite objective function, which includes at least one photometric consistency loss based on the multi-view image and one shape prior error term based on the preliminary 3D mesh model. The preset parameterized model is a statistical shape model, trained using 3D sample data of fish, shrimp, and shellfish. This 3D sample data is obtained through multi-view... Figure 3 The 3D reconstruction method was used to process fish, shrimp, and shellfish samples. S6: Extract the corresponding phenotypic parameters of the detected object based on the predefined shape parameters and feature points of the optimized 3D mesh model.

[0011] In step S6, the phenotypic parameters include the linear size parameters, surface area, and volume corresponding to the detected object. The predefined feature points include: for fish, head feature points, trunk feature points, fin feature points, and tail feature points based on fish anatomy; for shrimp, cephalothorax feature points, abdominal segment feature points, and telson feature points based on shrimp anatomy; for shellfish, apex feature points, anterior end feature points, posterior end feature points, ventral margin feature points, and ventral margin feature points based on shell geometry. The predefined shape parameters are: for fish, principal component coefficients of the corresponding preset parametric model; for shrimp, joint angle vectors and shape principal component coefficients based on the corresponding preset parametric model; and for shellfish, spherical harmonic function expansion coefficients. The extraction of multiple phenotypic parameters of the detection object in step S6 specifically includes: the position information of predefined feature points calculated by predefined shape parameters of the optimized three-dimensional mesh model, and linear size parameters calculated based on the Euclidean distance between the feature points. The linear size parameters are for different detection objects, including one or more of the following for fish samples: body length, body height, body width, and head length; one or more of the following for shrimp samples: total length, carapace length, abdominal segment width, and carapace height; and one or more of the following for shell samples: shell length, shell width, shell height, and shell thickness. The surface area and volume of the detection object are obtained by geometric integration of the optimized three-dimensional mesh model.

[0012] The beneficial effects of this invention are as follows: This invention enables automated, high-precision, and rapid detection of the three-dimensional phenotypes of live fish, shrimp, and shellfish, effectively overcoming measurement interference caused by the respiratory movements and surface reflections of these organisms.

[0013] 1. Through the synchronous control mechanism of "state monitoring-dynamic triggering", a consistent physiological state benchmark (i.e. gill cover closure) is set for the acquisition of multi-view images of fish, thereby ensuring the geometric consistency of the input data itself and fundamentally avoiding the three-dimensional reconstruction error caused by the breathing movement of live fish; for shrimp and shellfish, the process is simplified through direct acquisition mode, reflecting the versatility of the device.

[0014] 2. By integrating the hardware foundation of "known camera precise pose", the initial registration of "sample mask contour fitting" and the algorithmic constraints of "statistical shape model prior", an efficient and robust joint optimization reconstruction framework was constructed, which enables the complete and accurate 3D geometric model to be recovered even in areas with weak texture and high reflectivity of the sample, thereby achieving high-precision, high-completeness and high-efficiency 3D reconstruction.

[0015] 3. A replaceable support structure is designed. This invention features a specially designed fish support frame whose support surface angle conforms to the fish's surface, providing stable support at key locations while minimizing obstruction of the fish's surface. Simultaneously, it ensures that deformation in the fish's contact area remains within a controllable range. The position and height of the tail support are flexibly adjustable to accommodate fish of different sizes. Furthermore, flat support structures and clamping structures are designed specifically for shrimp and shellfish. This design achieves a good balance between ease of operation, animal welfare, and the integrity of 3D reconstruction.

[0016] 4. Through highly integrated hardware and software co-design and automated processes, efficient and standardized "one-click" 3D phenotypic data acquisition is achieved. This system features an integrated design from hardware layout and control logic to software processing, with each module intelligently scheduled and synchronized by a central unit. Users only need to place the sample and close the light-blocking door; the entire subsequent process, from multi-view image synchronous acquisition and 3D model reconstruction to phenotypic parameter extraction and report generation, can be completed automatically. This deeply collaborative "end-to-end" design encapsulates complex operations within the system, greatly simplifying manual operation steps and enabling convenient and rapid acquisition of standardized 3D phenotypic data for fish, shrimp, and shellfish. Attached Figure Description

[0017] Figure 1 A schematic diagram of the structure of the fish, shrimp and shellfish three-dimensional phenotypic detection device based on multi-view stereo vision provided by the present invention; Figure 2 This is a schematic diagram of the fish body support frame structure; Figure 3 This is a schematic diagram of a shellfish clamping structure; Figure 4 This is a schematic diagram showing the arrangement of the image acquisition module and the status monitoring camera; Figure 5 This is a schematic diagram of the detection method process.

[0018] Among them, 1 is the rigid outer frame, 2 is the touch screen, 3 is the light shielding door panel, 4 is the control module, 5 is the weighing module, 6 is the sample support frame, 7 is the first turntable, 8 is the second turntable, 9 is the status monitoring camera, 10 is the image acquisition module, 11 is the network switch, 12 is the standard light source on the upper panel, 13 is the standard light source on the lower panel, 14 is the sliding component, 15 is the height adjustment knob, 16 is the clamping component, 17 is the flexible ejector pin structure, and 18 is the arc-shaped connecting rod. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0021] A three-dimensional phenotypic detection device for fish, shrimp, and shellfish based on multi-view stereo vision includes a rigid outer frame 1, a control module 4, a weighing module 5, a sample support frame 6, a turntable module, a network switch 11, a status monitoring camera 9, and an image acquisition module 10. These modules work collaboratively through precise mechanical and electrical connections to form a complete automated measurement platform. The core technology of this device lies in its active sensing and control system. Specifically, this system does not passively image; instead, it achieves dynamic and synchronous image capture of samples under specific physiological states through a closed-loop collaboration of a rotating multi-view image acquisition module, a camera dedicated to synchronous physiological state monitoring, a precision motion control module, and a data processing center. This aims to fundamentally overcome the interference of micro-movements in live samples on the accuracy of three-dimensional reconstruction, providing geometrically consistent high-quality input data for subsequent high-precision three-dimensional phenotypic analysis.

[0022] The rigid outer frame 1 is constructed of aluminum alloy profiles and a cover plate, providing a stable overall structure. It also contains a cavity where the weighing module 5 is located. A sample support frame 6 supports the object being tested, positioned on top of the weighing module 5. Different sample support frames 6 can be used depending on the type of object being tested, thus achieving integrated functionality of stable support for the object's posture and simultaneous acquisition of weight data. The two ends of the turntable module are fixed to the corresponding inner walls of the cavity. An image acquisition module 10 is mounted on the turntable module and rotates with it to capture images of the object being tested. A status monitoring camera 9 is located within the cavity to monitor the object's breathing status. A control module 4 is also integrated within the cavity. The high-resolution industrial camera of the image acquisition module 10 and the status monitoring camera 9 are electrically connected to the control module 4 via a network switch 11. Specifically, the control module 4, acting as the central control core, is connected to the image acquisition module 10, the status monitoring camera 9, and the weighing module 5 via a communication bus and the network switch 11. It is used to determine the gill cover status and trigger synchronous image acquisition, while simultaneously reading data from the weighing module 5 and running 3D reconstruction and phenotypic parameter extraction algorithms.

[0023] Specifically, the control module 4 is electrically connected to the weighing module 5, the turntable module, the image acquisition module 10, and the status monitoring camera 9, thereby enabling it to acquire data from the weighing module 5, control the movement of the turntable module and receive the motion angle information fed back by the turntable module, control the image acquisition module 10 to acquire images of the object being tested, and monitor the breathing status of the object being tested.

[0024] The control module 4 is the data processing and control center of the device, integrating a high-performance GPU for running complex visual algorithms. The touchscreen display 2, serving as the human-machine interface, is integrated into the rigid outer frame 1. The control module 4 communicates with the three cameras of the image acquisition module 10 and the status monitoring camera 9 via a network switch 11, enabling centralized scheduling and synchronization of command issuance and data acquisition. Simultaneously, the control module 4 communicates with the two turntables and the weighing module via a serial communication interface to achieve control and data transmission.

[0025] The sample support frame 6 can be replaced with the appropriate structure depending on the object being tested: like Figure 2As shown, when the test subject is fish, the fish support frame is specifically designed to stably and with minimal interference support live fish under anesthesia. The sample support frame 6 consists of three independent support members, all parallel to the length of the fish. Two of these independent support members are symmetrically arranged to support the fish's head and body. Their ends extend upwards at a preset angle to support the fish's head from below. The support surfaces of these two members are tilted inwards to conform to the natural contour of the fish's abdomen, thereby providing stable support while minimizing obstruction of the fish's lateral field of vision. Another independent support is arranged at the rear to support the fish's tail. Specifically, the support for the head and body of the fish is fixed in position, and its support surface is slightly tilted inward to fit the fish body. The support for the fish's tail is a sliding component 14, which is adjustable in horizontal position. Its height can be adjusted by the height adjustment knob 15 to fine-tune the height of the tray. By adjusting, the fish tail and the main axis of the fish body can be kept roughly at the same horizontal plane to ensure that the fish body is upright and to meet the measurement needs of fish of different lengths. When the test subject is shrimp, the sample support frame 6 is a horizontal platform used to stably place the shrimp. Specifically, the surface is covered with a flexible anti-slip material to stabilize the shrimp and ensure that the shrimp maintains a natural posture and does not slip during the collection process. like Figure 3 As shown, when the object of detection is a shellfish, the sample support frame 6 consists of two symmetrical clamping components 16 on both sides. Each clamping component 16 is equipped with a flexible pin structure 17 for clamping the shellfish. Specifically, the shellfish clamping structure includes two symmetrical clamping components 16, each with a flexible pin structure 17, which provides gentle point contact clamping from both sides of the shellfish to avoid damage and ensure stability. This component achieves stable clamping of the shellfish through a built-in spring. This design provides sufficient stability while minimizing occlusion of the shellfish surface, facilitating multi-view image acquisition.

[0026] All three support structures are connected to the weighing module 5 by snap-fit, and their height is designed to be near the rotation axis of the first turntable 7 and the second turntable 8 to ensure that the entire sample is located within the rotating scanning plane of the image acquisition module 10.

[0027] The turntable module includes a first turntable 7 and a second turntable 8. The image acquisition module 10 includes a rigid mounting beam and multiple high-resolution industrial cameras. The first turntable 7 and the second turntable 8 are symmetrically fixed on the corresponding inner walls of the cavity, and are connected by a U-shaped aluminum alloy rigid mounting beam with a central protrusion. Specifically, the two ends of the beam are rigidly connected to the output shafts of the first turntable 7 and the second turntable 8 respectively via flanges. The first turntable 7 and the second turntable 8 are high-precision synchronously rotating electric turntables and are both electrically connected to and controlled by the control module 4. The control module 4 connects to the drivers of the two turntables via a serial bus and sends synchronous motion commands to drive them to rotate at the same angular velocity, thereby driving the image acquisition module 10 spanning between them to rotate smoothly and accurately around the sample support frame 6. The turntables provide real-time feedback of high-precision angle information to the control module 4. The first turntable 7 and the second turntable 8 are driven by servo motors and equipped with absolute encoders, and their rotation axes are strictly aligned to form a horizontal axis. This allows the rigidly mounted crossbeam to rotate around a horizontal axis. Specifically, both ends of the rigidly mounted crossbeam are connected to the first turntable 7 and the second turntable 8, respectively. The image acquisition module 10 is the core sensor group for acquiring the three-dimensional information of the sample, and its position and layout are as follows: Figure 1 and Figure 4 As shown. The multiple high-resolution industrial cameras specifically consist of three global shutter industrial cameras, each equipped with a fixed-focus lens. The focusing distance of all lenses is pre-adjusted to the center plane of the sample support frame 6. The relative positions and orientations of the three industrial cameras have been precisely calibrated and remain fixed, thus forming a multi-view observation array with known geometric relationships, providing the hardware foundation for subsequent 3D reconstruction algorithms. All high-resolution industrial cameras are spaced apart along the length of the rigid mounting beam on the central protrusion, with all camera lenses facing the sample support frame 6. The height of the sample support frame 6 is slightly lower than the horizontal axis formed by the first turntable 7 and the second turntable 8, ensuring that the object to be detected placed on the sample support frame 6 is on the horizontal axis.

[0028] Furthermore, the proposed camera and lens parameters are as follows: resolution 1920*1200; sensor Sony IMX249 Global shutter CMOS; target size 1 / 1.2”; pixel size 5.86μm*5.86μm; frame rate 41fps; focal length 6mm.

[0029] The status monitoring camera 9 is an independent high-frame-rate industrial camera, fixedly connected to the weighing module 5 via an arc-shaped connecting rod 18. The status monitoring camera 9 faces the sample support frame 6, which supports the head of the fish, and is specifically positioned near the head support end of the fish support frame. Its optical axis is at the same height as the rotation axis of the image acquisition module 10 to continuously monitor the real-time status of the fish's gill cover area. The status monitoring camera 9 is close to the horizontal axis in height, and its installation does not interfere with the rotation of the turntable module or the image acquisition of the image acquisition module 10. Specifically, as shown... Figure 4 As shown, its installation position satisfies the following precise spatial relationships: horizontally, the camera is adjacent to the front end of the left-side support; vertically, its optical axis is approximately aligned with the rotation axes of the first turntable 7 and the second turntable 8. Furthermore, the layout of the camera body and its support is carefully planned to ensure that it is completely outside the rotational motion envelope of the image acquisition module 10 and does not obstruct the field of view of any acquisition camera. This camera is used to continuously capture and output a real-time video stream of the fish's gill cover area. That is, its body and connectors are located outside the rotational path and field of view of the image acquisition module 10.

[0030] The cavity within the rigid outer frame 1 also houses an illumination module for providing a light source. Specifically, the illumination module consists of a standard light source 12 on the upper panel and a standard light source 13 on the lower panel. Both are large-area LED planar light sources, rigidly fixed to the center of the top and bottom of the chamber by brackets, respectively, and together vertically illuminating the sample support frame 6 area placed in the center. The light source surfaces are covered with a light-diffusing plate to ensure uniform and soft output light, thereby creating a relatively uniform, stable imaging environment free from external stray light interference.

[0031] It also includes a light-shielding door panel 3. A switchable light-shielding door panel 3 is installed on one side of the rigid outer frame 1. Both the rigid outer frame 1 and the light-shielding door panel 3 are made of opaque material. The light-shielding door panel 3 is connected to the rigid outer frame 1 via hinges, and a sealing strip is provided between it and the frame when closed to ensure the optical sealing of the imaging chamber. To create an ideal lighting environment, all inner wall surfaces of the cavity of the rigid outer frame 1 and the inner surface of the light-shielding door panel 3 are covered with a high-reflectivity diffuse white film. This design allows for sufficient and uniform diffuse reflection of light, effectively eliminating shadows and specular reflection hotspots. The diffuse reflective material covering the inner walls of the cavity of the rigid outer frame 1 and the inner surface of the light-shielding door panel 3 together form a closed imaging chamber with a diffuse reflective inner surface.

[0032] It also includes a touch screen 2, which is arranged on the outer wall of the rigid outer frame 1 and electrically connected to the control module 4, serving as a human-machine interface.

[0033] A method for detecting three-dimensional phenotypes of fish, shrimp, and shellfish is as follows: S1: Place the subject under appropriate anesthesia on the corresponding sample support frame 6, record the weight, close the light shield, and start the testing process; S1 is as follows: After the operator turns on the device, the device will automatically turn on the lighting system and reset the image acquisition module to its initial position. The operator opens the light-shielding door 3 and places the live fish to be tested, after being moderately anesthetized, horizontally on the fish support frame with the head to the left and the tail to the right. According to the actual length of the fish, the operator manually fine-tunes the front-to-back position and height of the right support to keep the fish upright and stable, and the main axis of the fish roughly parallel to the rotation axis of the image acquisition module 10. After closing the light-shielding door 3, the operator selects the fish detection mode on the touch screen 2 and clicks the start button. The control module 4 automatically reads the weight data from the weighing module 5 and starts the detection process. The method for shrimp or shellfish is similar: place the sample on a flat plate or clamping structure, select the appropriate mode, and start the detection process.

[0034] S2: The control module 4 monitors the detection object through the status monitoring camera 9. Based on the category of the detection object and the detection result of the status monitoring camera 9, it selectively controls the turntable module to rotate and drive the image acquisition module 10 to acquire images of the detection object to obtain a multi-view image sequence composed of multiple multi-view images. The testing object in step S1 is a live fish, shrimp, or shellfish sample after moderate anesthesia. Step S2 uses different collection methods depending on the type of testing object in step S1. When the object of detection is a live fish sample, the gill cover status of the live fish sample is monitored in real time by the status monitoring camera 9, and the image acquisition module 10 performs synchronous image acquisition only when the gill cover is closed, thereby obtaining multiple sets of multi-view image sequences from different perspectives acquired when the live fish sample is relatively still. When the object of detection is a live shrimp or shellfish sample, it is regarded as a geometrically invariant object. It does not need to be monitored by the status monitoring camera 9. Instead, the image acquisition module 10 directly performs synchronous image acquisition at a predetermined angle to obtain a multi-view image sequence of the live shrimp or shellfish sample.

[0035] Specifically, S2a: Control the first turntable 7 and the second turntable 8 to synchronously drive the image acquisition module 10 to rotate to a predetermined acquisition angle and remain stationary at that angle; S2b: If the object being detected is fish, the state monitoring camera 9 monitors the degree of closure of the fish's gill covers in real time; for fish, the image acquisition module 10 is immediately triggered to acquire images synchronously only when the degree of closure of the gill covers reaches or exceeds the preset threshold; for shrimp or shellfish, acquisition is triggered directly. S2c: After the acquisition is completed, control the turntable to rotate to the next predetermined acquisition angle, and repeat steps S2a to S2c until all predetermined angles are acquired.

[0036] Furthermore, if the object being detected is fish, the control turntable drives the image acquisition module 10 to rotate to a predetermined angle and remain stationary. Simultaneously, the status monitoring camera 9 begins to capture a video stream of the fish's gill cover area in real time. The video frames are analyzed in real time using a pre-trained deep learning model to quantify the degree of gill cover closure. For example, a lightweight fish gill cover key point detection model based on YOLOv8-Pose is used as the core inference algorithm of the status monitoring camera 9. This model predefines anatomical key points on the upper and lower edges of the gill cover in the fish's gill cover area. By calculating the ratio of the Euclidean distance between the upper and lower edge key points to the reference length from the fish's eye to its snout, a quantitative value C of the degree of gill cover closure is obtained. When C exceeds a preset threshold and remains stable for several consecutive frames, it is determined to be in a gill cover closed state. The method infers the degree of gill cover closure from real-time video frames to obtain a quantitative value. If this value exceeds a preset threshold, the three industrial cameras in the image acquisition module 10 are immediately triggered to simultaneously acquire images, ensuring that the fish is in a relatively static physiological state, i.e., the closed phase of its respiratory cycle, at the moment of acquisition. After one acquisition is completed, the turntable steps to the next angle, for example, acquiring images every 15°, for a total of 24 positions. The above "position-wait-trigger" cycle is repeated. This method ensures that all multi-view images used for 3D reconstruction have a consistent physiological state by strictly limiting each acquisition to the closed state of the gill cover, thereby providing geometrically consistent input data for subsequent reconstruction and avoiding reconstruction errors caused by the periodic deformation of the fish. S3: Based on the rotation angle information of the image acquisition module 10 and the fixed relative pose parameters pre-calibrated by each high-resolution industrial camera in the image acquisition module 10, determine the precise camera pose corresponding to the acquisition time of each multi-view image. The specific method for determining the precise camera pose in step S3 is as follows: A world coordinate system is established with the intersection of the horizontal axis formed by the first turntable 7 and the second turntable 8 in the image acquisition module 10 and the central horizontal plane of the sample support frame 6 as the origin of the world coordinate system; based on the rotation angles fed back by the first turntable 7 and the second turntable 8, the instantaneous rigid transformation matrix corresponding to the image acquisition module 10 in the world coordinate system is calculated; based on the preset extrinsic parameters of each high-resolution industrial camera relative to the rigid mounting beam of the image acquisition module 10, the camera pose of each high-resolution industrial camera at each multi-view image acquisition moment is calculated. This pose has absolute scale and no cumulative error, providing deterministic geometric constraints for subsequent 3D reconstruction.

[0037] Specifically, for each acquired image, the system directly associates it with the precise world coordinate system pose established for that camera at that location during system pre-calibration, based on its camera number and corresponding acquisition location. This provides deterministic geometric constraints with absolute scale and no cumulative error for subsequent 3D reconstruction. For each acquired image, based on its camera number, the three cameras in the image acquisition module 10 have pre-calibrated relative poses and corresponding acquisition positions, i.e., the precise rotation angles fed back by the absolute encoders of the first turntable 7 and the second turntable 8, to calculate its pose in the world coordinate system. Specifically, the method is as follows: the intersection of the rotation axis of the image acquisition module 10 and the center plane of the sample support frame 6 is taken as the origin of the world coordinate system; the instantaneous rigid transformation matrix of the image acquisition module 10 as a whole in the world coordinate system is calculated based on the rotation angles fed back by the turntables; and then, combined with the pre-calibrated extrinsic parameters of each camera relative to the mounting beam, the precise pose of each camera at each acquisition moment is derived. This pose has absolute scale and no cumulative error, providing deterministic geometric constraints for subsequent 3D reconstruction.

[0038] S4: Perform foreground segmentation on each multi-view image in the multi-view image sequence and extract the sample pixel region mask corresponding to each multi-view image; Before performing 3D reconstruction using multi-view images, foreground segmentation is first performed on each image to extract a precise mask of the fish's pixel region. Thanks to the enclosed chamber, uniform illumination, and simple background provided by the device, the fish exhibits high contrast and clear contours in the images, greatly simplifying the segmentation task. In a preferred embodiment, a deep learning-based semantic segmentation model can be used to achieve fully automated mask generation. The obtained mask effectively eliminates interference from the support frame and background, ensuring that subsequent 3D matching and optimization are strictly focused on the fish itself, providing robust preprocessing assurance for generating a high-precision model.

[0039] S5: By utilizing multi-view images, corresponding precise camera poses, and corresponding sample pixel region masks, 3D reconstruction is performed by fusing the preset parameterized model of the corresponding detection object to generate an optimized 3D mesh model. This method effectively fills in the data missing caused by fish surface reflection or weak texture and suppresses noise by constraining the model to conform to the statistical shape prior of fish during optimization, including overall streamlined shape, left-right symmetry, biologically reasonable proportional distribution, and surface smoothness. This overcomes the common difficulties of traditional methods in 3D reconstruction of live fish.

[0040] Step S5 is as follows: Starting with the pre-defined parametric model of the corresponding detection object, initial registration is performed using a contour-based fitting method. The projected contour of the pre-defined parametric model under each high-resolution industrial camera pose is rendered, and the difference between the pre-defined parametric model and the corresponding sample pixel region mask is minimized to obtain a preliminary 3D mesh model. Starting from this, the shape parameters, pose, and surface texture mapping of the preliminary 3D mesh model are simultaneously optimized using a joint optimization algorithm to obtain an optimized 3D mesh model.

[0041] The joint optimization algorithm obtains the optimal 3D mesh model by minimizing a composite objective function, which includes at least one photometric consistency loss based on multi-view images and one shape prior error term based on the preliminary 3D mesh model. The preset parametric model is a statistical shape model, trained using a large amount of historical 3D fish, shrimp, and shellfish sample data. This large amount of historical 3D fish, shrimp, and shellfish sample data is obtained through traditional multi-view... Figure 3 The dimensional reconstruction method was obtained by processing a large number of fish, shrimp and shellfish samples, and the preset parameterized model can characterize samples of different sizes and shapes by adjusting a set of low-dimensional shape parameters.

[0042] The shape prior error term is used to constrain the optimized 3D model to conform to one or more of the inherent structural, morphological, and surface characteristics of fish, shrimp, and shellfish. Specifically, the shape prior error term constrains the model's shape parameters, ensuring they remain within a reasonable distribution range defined by the statistical shape model for fish, shrimp, and shellfish. This prevents the reconstruction results from exhibiting deformities or abnormal shapes that do not conform to biological laws, thus guaranteeing that the generated 3D model of fish, shrimp, and shellfish has a natural appearance, bilateral symmetry, and a reasonable proportional distribution. The shape prior error term penalizes the degree to which the shape parameters deviate from the reasonable shape distribution defined by the statistical shape model.

[0043] More specifically, 3D reconstruction is performed by fusing parametric standard fish body models using multi-view images and their precise camera poses, along with fish body masks obtained from S4. First, the average fish body shape is used as the initial model parameters, and a contour-based fitting method is employed for initial registration: the parametric standard fish body model is rendered using PyTorch3D's differentiable rendering technology. This statistical shape model, constructed by analyzing a large amount of 3D fish sample data, can characterize morphological variations such as body length, height, and width through low-dimensional shape parameters. The projected contours under each camera pose are minimized, and the IoU loss between the model and the fish body mask is minimized, thus obtaining the preliminary model shape parameters. Starting from this, a joint optimization algorithm is used to simultaneously optimize the model's shape parameters, pose, and surface texture mapping. The optimization objective is to minimize the composite function E: ; Among them, the photometric consistency error term Used to measure the color difference between the projected image rendered by the model from all viewpoints and the actual photographed image, it is calculated only within the fish body area to ensure that the texture and appearance of the reconstructed model are highly consistent with the actual fish body; and These are the weighting coefficients for the shape prior and the keypoint loss term, respectively. For shape prior error term, For key loss items; Photometric consistency error term The calculation formula is: ; Where V is the total number of viewpoints; v is the viewpoint index; The projected image of model M rendered by combining surface texture T at the v-th camera pose; For the fish body mask at the v-th viewpoint; This is a real-world image taken from the v-th perspective; Shape prior error term The shape parameters used to constrain the model ensure that they remain within a reasonable distribution range defined by the statistical shape model of fish, preventing deformities or abnormal shapes in the reconstruction results that do not conform to biological laws. This guarantees that the generated 3D fish model has a natural streamlined appearance, bilateral symmetry, and reasonable proportional distribution. The calculation formula is as follows: ; Where α is the shape parameter vector in the current iteration; μ is the mean vector of the statistical shape model; It is the inverse of the covariance matrix; Key point loss item This is used to ensure the precise correspondence of the positions of local anatomical features such as fins, gill covers, snout, and tail forks, thus providing reliable geometric anchor points for subsequent automatic extraction of phenotypic parameters. The calculation formula is as follows: ; k is the keypoint index; Let be the 2D coordinates of the k-th keypoint detected in the real image from the v-th viewpoint; Let be the 2D coordinates of the k-th keypoint obtained by projecting the model under the v-th camera pose.

[0044] Symbol explanation: and These are the weight coefficients for the shape prior and the keypoint loss term, respectively; V is the total number of views; v is the view index; The projected image of model M rendered by combining surface texture T at the v-th camera pose; α is the fish mask for the v-th viewpoint; α is the shape parameter vector in the current iteration; μ is the mean vector of the statistical shape model; is the inverse of the covariance matrix; k is the keypoint index; Let be the 2D coordinates of the k-th keypoint detected in the real image from the v-th viewpoint; Let be the 2D coordinates of the k-th keypoint obtained by projecting the model under the v-th camera pose.

[0045] By minimizing the aforementioned composite function E, the system can effectively fill in data gaps caused by surface reflection or weak texture of the fish, and improve the accuracy of local fine structure through keypoint constraints, ultimately generating a high-precision, complete, and biologically plausible 3D mesh model of the fish. This method fully utilizes the global constraints of statistical shape priors and the local anchoring of keypoint loss to achieve robust reconstruction in areas of weak texture and high reflectivity in fish, while providing a solid foundation for the automated and repeatable extraction of subsequent phenotypic parameters.

[0046] S6: Based on the predefined shape parameters and key feature points of the optimized 3D mesh model, the system automatically extracts the corresponding phenotypic parameters of the detected object. The system automatically extracts multiple phenotypic parameters of the fish body: calculating key morphological dimensions such as body length, body height, body width, and head length from the predefined key points of the optimal model; calculating parameters such as volume and surface area through geometric operations on the 3D mesh; and combining the weight data obtained in S1 to calculate composite indicators such as body fat percentage. All phenotypic parameters are output as a structured dataset.

[0047] In step S6, the phenotypic parameters include the linear size parameters, surface area, and volume corresponding to the detected object. Predefined key feature points include: for fish, head, trunk, fin, and tail feature points based on fish anatomy, specifically including the snout tip, eye center, posterior margin of the gill cover, origin of the dorsal fin, end of the dorsal fin, origin of the anal fin, midpoint of the caudal fork, origin of the pectoral fin, and origin of the pelvic fin; for shrimp, cephalothorax, abdominal segments, and telson feature points based on shrimp anatomy, specifically including the tip of the rostrum, midpoint of the posterior margin of the cephalothorax, anterior margin of the first abdominal segment, dorsal midline of the third abdominal segment, posterior margin of the sixth abdominal segment, and telson tip; for mollusks, shell apex feature points based on the shell geometry, shell... The anterior end feature point, posterior end feature point, ventral margin feature point, and ventral margin feature point are specifically included: the apex, the furthest point of the anterior end, the furthest point of the posterior end, the furthest point of the ventral margin, and the furthest point of the dorsal margin. The anterior end, posterior end, ventral margin, and dorsal margin are defined based on the principal axis and vertical direction of the shell of the mollusk. The shape parameters include: for fish, the first 10 principal component coefficients of the corresponding preset parametric model; for shrimp, the joint angle vectors of the skeletal model in the corresponding preset parametric model, as well as the shape principal component coefficients that control the overall size and body segment ratio; for mollusks, the spherical harmonic function expansion coefficients, whose lower-order terms characterize the global scale and symmetry of the shell, and higher-order terms characterize the local undulations and textures of the shell. The extraction of multiple phenotypic parameters of the detection object in step S6 specifically includes: the position information of predefined key feature points on a high-precision 3D mesh calculated by the shape parameters of the optimized 3D mesh model, and the linear size parameters automatically calculated based on the Euclidean distance between the key feature points. The linear size parameters are for different detection objects, including one or more of the following for fish samples: body length, body height, body width, and head length; one or more of the following for shrimp samples: total length, carapace length, abdominal segment width, and carapace height; and one or more of the following for shell samples: shell length, shell width, shell height, and shell thickness; and the surface area and volume of the detection object obtained by geometric integration of the optimized 3D mesh model.

[0048] More specifically, based on the high-precision 3D mesh model generated by S5, the system automatically extracts multiple phenotypic parameters of the fish body. This step fully utilizes the low-dimensional shape parameter α of the parametric fish body model (SSM) and predefined key points in the template to achieve efficient, objective, and consistent phenotypic extraction without manual intervention or additional image processing.

[0049] The SSM template mesh predefines several anatomical key points, such as the snout tip, caudal fork, base of the dorsal fin, lowest point of the abdomen, and posterior edge of the gill cover. The indices of these key points remain unchanged before and after optimization. The system calculates the final mesh vertex coordinates using the optimized shape parameter α. : ; in The average grid vertex coordinates, Let be the shape basis vector matrix.

[0050] Based on these predetermined keypoint coordinates, linear phenotypic parameters are automatically calculated using the Euclidean distance between the corresponding keypoints, including but not limited to key dimensions such as body length, body height, body width, and head length. Volume and surface area parameters are obtained through geometric integration of the optimized 3D mesh. Combining the weight data obtained from the weighing module in S1, composite indices are further calculated, such as Fulton's condition factor, which is calculated as the ratio of fish weight or volume to the cube of body length.

[0051] All phenotypic parameters are automatically generated through predefined keypoint indices, grid vertex coordinates, and geometric operations, ensuring that the extraction process is objective, consistent, and highly repeatable. This step fully utilizes the low-dimensional representation of the parametric model and the predetermined feature point mapping to achieve efficient and standardized phenotypic data output, facilitating subsequent breeding analysis, germplasm evaluation, or database storage.

Claims

1. A three-dimensional phenotypic detection device for fish, shrimp, and shellfish based on multi-view stereo vision, characterized in that: It includes a rigid outer frame (1), a control module (4), a weighing module (5), a sample support frame (6), a turntable module, a status monitoring camera (9), and an image acquisition module (10). The rigid outer frame (1) has a cavity inside, the weighing module (5) is arranged inside the cavity, the sample support frame (6) is used to support the test object arranged on the upper end of the weighing module (5), and the corresponding sample support frame (6) is replaced according to the type of test object; the two ends of the turntable module are respectively fixed on the inner walls of the corresponding two sides of the cavity, the image acquisition module (10) is installed on the turntable module and rotates with the turntable module, and the status monitoring camera (9) is arranged inside the cavity; the cavity also integrates a control module (4), the control module (4) is electrically connected to the weighing module (5), the turntable module, the image acquisition module (10), and the status monitoring camera (9) respectively, so as to realize the acquisition of data from the weighing module (5), control of the movement of the turntable module and receiving the motion information fed back by the turntable module, control of the image acquisition module (10) to acquire the image of the test object, and monitoring of the breathing state of the test object; The samples to be tested are live fish, shrimp, or shellfish, and different collection methods are used depending on the type of sample. When the object of detection is a live fish sample, the gill cover status of the live fish sample is monitored by the status monitoring camera (9), and the image acquisition module (10) is used to acquire images only when the gill cover is closed, thereby obtaining a multi-view image sequence of the live fish sample. When the object of detection is a live shrimp or shellfish sample, since it is not necessary to monitor it through the status monitoring camera (9), the image acquisition module (10) performs synchronous image acquisition at a predetermined angle to obtain a multi-view image sequence of the live shrimp or shellfish sample.

2. The three-dimensional phenotypic detection device for fish, shrimp, and shellfish based on multi-view stereo vision according to claim 1, characterized in that: The sample support frame (6) can be replaced with a corresponding structure depending on the object being tested: When the test object is fish, the sample support frame (6) consists of three independent support members that are parallel to the length of the fish body. Two of the independent support members are symmetrically arranged to support the head and body of the fish, and the other independent support member is arranged at the rear to support the tail of the fish. When the test object is shrimp, the sample support frame (6) is a horizontal platform used to place the shrimp; When the object of the test is a shellfish, the sample support frame (6) consists of two symmetrical clamping components (16) on both sides, and each clamping component (16) is provided with a flexible pin structure (17) for clamping the shellfish.

3. The three-dimensional phenotypic detection device for fish, shrimp, and shellfish based on multi-view stereo vision according to claim 2, characterized in that: The turntable module includes a first turntable (7) and a second turntable (8). The image acquisition module (10) includes a rigid mounting beam and multiple cameras. The first turntable (7) and the second turntable (8) are symmetrically fixed on the inner walls of the corresponding sides of the cavity. The first turntable (7) and the second turntable (8) are connected by a rigid mounting beam with a U-shaped central protrusion. The first turntable (7) and the second turntable (8) are synchronously rotating electric turntables and are both electrically connected to the control module (4), so that the rigid mounting beam rotates around the horizontal axis. All the cameras are arranged at intervals along the length of the rigid mounting beam on the central protrusion and are all facing the sample support frame (6). All the cameras are electrically connected to the control module (4). The height of the sample support frame (6) is lower than the horizontal axis formed by the first turntable (7) and the second turntable (8), so that the detection object placed on the sample support frame (6) is on the horizontal axis.

4. The three-dimensional phenotypic detection device for fish, shrimp, and shellfish based on multi-view stereo vision according to claim 3, characterized in that: The status monitoring camera (9) is fixedly connected to the weighing module (5) via an arc-shaped connecting rod (18), and the status monitoring camera (9) faces the sample support frame (6) for fish to support the head of the fish. The installation of the status monitoring camera (9) does not interfere with the rotation of the turntable module and the image acquisition module (10). The cavity inside the rigid outer frame (1) is also equipped with a lighting module to provide a light source.

5. A method for detecting the three-dimensional phenotype of fish, shrimp, and shellfish, applied to the apparatus described in any one of claims 1-4, characterized in that, The detection method is as follows: S1: Place the object to be tested on the corresponding sample support frame (6); S2: The detection object is monitored by the status monitoring camera (9). According to the category of the detection object and the detection result of the status monitoring camera (9), the turntable module is selectively controlled to rotate to drive the image acquisition module (10) to acquire images of the detection object and obtain a multi-view image sequence composed of multiple multi-view images. S3: Determine the camera pose corresponding to each of the multi-view images based on the rotation angle information of the image acquisition module (10) and the fixed relative pose parameters of each camera pre-calibrated in the image acquisition module (10); S4: Perform foreground segmentation on each multi-view image in the multi-view image sequence and extract the sample pixel region mask corresponding to each multi-view image; S5: Using the multi-view images, the corresponding camera poses and the corresponding sample pixel region masks, three-dimensional reconstruction is performed by fusing the preset parameterized model of the corresponding detection object to generate an optimized three-dimensional mesh model. S6: Extract the corresponding phenotypic parameters of the detected object based on the predefined shape parameters and feature points of the optimized 3D mesh model.

6. The method for detecting three-dimensional phenotypes of fish, shrimp, and shellfish according to claim 5, characterized in that: The specific method for determining the camera pose in step S3 is as follows: a world coordinate system is established with the intersection of the horizontal axis formed by the first turntable (7) and the second turntable (8) in the turntable module and the center of the sample support frame (6) as the origin of the world coordinate system; the rigid transformation matrix of the image acquisition module (10) in the world coordinate system is calculated according to the rotation angle of the first turntable (7) and the second turntable (8); the camera pose of each camera in each multi-view image is calculated according to the preset extrinsic parameters of the rigid mounting beam of the image acquisition module (10) relative to each camera.

7. The method for detecting three-dimensional phenotypes of fish, shrimp, and shellfish according to claim 5, characterized in that, Step S5 specifically involves: Starting with the pre-defined parametric model of the corresponding detection object, initial registration is performed using a contour-based fitting method. The projected contour of the pre-defined parametric model under each camera pose is rendered, and the difference between the pre-defined parametric model and the corresponding sample pixel region mask is minimized to obtain a preliminary 3D mesh model. The preliminary 3D mesh model is then simultaneously optimized using a joint optimization algorithm to obtain an optimized 3D mesh model.

8. The method for detecting three-dimensional phenotypes of fish, shrimp, and shellfish according to claim 7, characterized in that: The joint optimization algorithm obtains the optimal 3D mesh model by minimizing a composite objective function, which includes at least one photometric consistency loss based on the multi-view image and one shape prior error term based on the preliminary 3D mesh model. The preset parameterized model is a statistical shape model, which is trained using three-dimensional sample data of fish, shrimp, and shellfish. The three-dimensional sample data of fish, shrimp, and shellfish is obtained by processing the fish, shrimp, and shellfish samples using a multi-view three-dimensional reconstruction method.

9. The method for detecting three-dimensional phenotypes of fish, shrimp, and shellfish according to claim 5, characterized in that: In step S6, the phenotypic parameters include the linear size parameters, surface area, and volume corresponding to the detected object. The predefined feature points include: for fish, head feature points, trunk feature points, fin feature points, and tail feature points based on fish anatomy; for shrimp, cephalothorax feature points, abdominal segment feature points, and telson feature points based on shrimp anatomy; for shellfish, apex feature points, anterior end feature points, posterior end feature points, ventral margin feature points, and ventral margin feature points based on shell geometry. The predefined shape parameters are: for fish, principal component coefficients of the corresponding preset parametric model; for shrimp, joint angle vectors and shape principal component coefficients based on the corresponding preset parametric model; and for shellfish, spherical harmonic function expansion coefficients. The extraction of multiple phenotypic parameters of the detection object in step S6 specifically includes: the position information of predefined feature points calculated by predefined shape parameters of the optimized three-dimensional mesh model, and linear size parameters calculated based on the Euclidean distance between the feature points. The linear size parameters are for different detection objects, including one or more of the following for fish samples: body length, body height, body width, and head length; one or more of the following for shrimp samples: total length, carapace length, abdominal segment width, and carapace height; and one or more of the following for shell samples: shell length, shell width, shell height, and shell thickness. The surface area and volume of the detection object are obtained by geometric integration of the optimized three-dimensional mesh model.

Images