Method and device for measuring phenotype of cultured fish based on AR intelligent glasses

Abstract

The invention provides a cultured fish phenotype measurement method and device based on AR intelligent glasses. The method comprises the steps that a to-be-observed fish image is acquired through image acquisition equipment of the AR intelligent glasses; transmitting the fish image to an edge computing terminal through a data transmission device of AR intelligent glasses; analyzing the fish image through the edge computing terminal to obtain fish phenotype data; and returning the fish phenotype data to the AR intelligent glasses, and superposing the fish phenotype data to the observed fish in an augmented reality mode. The problem that the phenotype of the existing cultured fish is difficult to dynamically and accurately measure in real time is solved.

Description

Technical Field

[0001] The present invention relates to the field of aquaculture technology, and in particular to a method and device for measuring the phenotype of farmed fish based on AR smart glasses. Background Art

[0002] As the aquaculture industry evolves toward smarter, more efficient methods, non-contact fish phenotyping technologies are playing an increasingly important role in improving health assessment efficiency and promoting precision aquaculture. Current mainstream technologies rely primarily on fixed measurement systems, such as surveillance cameras installed near the aquaculture pond, binocular stereo vision systems, and underwater cameras. These systems are often cumbersome to deploy and lack flexibility. Limited by fixed locations, camera viewing angles, and environmental interference, they struggle to meet the practical needs of aquaculture sites for real-time, dynamic, and high-throughput measurements.

[0003] In recent years, the rapid development of augmented reality (AR) technology and wearable devices has provided new opportunities for mobile, real-time measurement. In particular, AR smart glasses (such as the XREAL One) integrate spatial perception, image acquisition, wireless communication, and AR-enhanced display, and support embedded basic image processing capabilities, demonstrating the potential for intelligent assisted perception in aquaculture inspections.

[0004] While some research has attempted to use mobile devices combined with cameras for simple measurements, there are no publicly available reports of a systematic approach that deeply integrates image acquisition, edge computing, and real-time AR visualization with AR glasses for estimating fish phenotypic data (such as length, width, and weight). Existing methods lack system optimization for individual identification and 3D reconstruction in dynamic aquaculture scenarios, and are unable to meet the needs of high-frequency, non-interventional measurement. Summary of the Invention

[0005] The present invention provides a method and device for measuring the phenotype of farmed fish based on AR smart glasses, which is used to solve the problem that the phenotype of farmed fish is difficult to measure accurately in real time and dynamically.

[0006] The present invention provides a method for measuring the phenotype of farmed fish based on AR smart glasses, which is applied to AR smart glasses and includes: Collect fish images; Sending the fish image to an edge computing terminal; Receive fish phenotypic data generated by edge computing terminal processing and overlay the fish phenotypic data onto the fish image.

[0007] According to a method for measuring farmed fish phenotypes based on AR smart glasses provided by the present invention, collecting fish images includes: The fish images are obtained by collecting images through a stereoscopic vision acquisition system constructed by two RGB cameras of AR smart glasses.

[0008] According to a method for measuring farmed fish phenotypes based on AR smart glasses provided by the present invention, the method includes receiving fish phenotype data generated by edge computing terminal processing and superimposing the fish phenotype data onto a fish image, including: Receiving fish phenotypic data, the fish phenotypic data being obtained by processing fish images through edge computing equipment; The fish phenotypic data is superimposed on the fish image through augmented reality.

[0009] The present invention provides a method for measuring the phenotype of farmed fish based on AR smart glasses, which is applied to edge computing terminals, including: Receive fish images sent by AR smart glasses; Analyzing the fish image to generate fish phenotypic data; The fish phenotypic data is sent to AR smart glasses.

[0010] According to a method for measuring farmed fish phenotypes based on AR smart glasses provided by the present invention, analyzing the fish image to generate fish phenotype data includes: performing image recognition analysis on the fish picture based on a pre-trained image recognition model to obtain fish phenotypic data; The image recognition model is obtained by training a preset neural network model using pre-acquired fish images with labeled information.

[0011] According to a method for measuring farmed fish phenotypes based on AR smart glasses provided by the present invention, the method performs image recognition analysis on the fish picture based on a pre-trained image recognition model to obtain fish phenotypic data, including: performing mask segmentation on the fish image; Estimate fish body length, body width, total length and volume based on mask segmentation results and combined with calibration size; The body length, body width, total length and volume of fish as well as the fish weight estimated based on the volume were used as fish phenotypic data.

[0012] The present invention also provides a farmed fish phenotype measurement device based on AR smart glasses, the device comprising: An image acquisition module, used for acquiring fish images; An image sending module, configured to send the fish image to an edge computing terminal; The data display module is used to receive the fish phenotypic data generated by the edge computing terminal and overlay the fish phenotypic data onto the fish image.

[0013] The present invention also provides a farmed fish phenotype measurement device based on AR smart glasses, the device comprising: an image receiving module for receiving fish images sent by the AR smart glasses; An image analysis module, configured to analyze the fish image and generate fish phenotypic data; The data return module is used to send the fish phenotypic data to the AR smart glasses.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for measuring the phenotype of farmed fish based on AR smart glasses as described above is implemented.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for measuring the phenotype of farmed fish based on AR smart glasses.

[0016] This invention provides a method and device for measuring farmed fish phenotypic data using AR smart glasses. The AR smart glasses capture fish images, wirelessly transmit the images to an edge computing terminal for recognition, and obtain fish phenotypic data. The AR smart glasses then overlay the fish phenotypic data onto the fish images using augmented reality. This enables real-time dynamic measurement of fish phenotypic data, and the edge computing device uses image recognition to obtain fish phenotypic data, improving measurement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 Schematic diagram of the process of measuring the phenotype of farmed fish based on AR smart glasses provided by the present invention.

[0019] Figure 2 Schematic diagram of the process of measuring the phenotype of farmed fish based on AR smart glasses provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the module connection of the farmed fish phenotype measurement system based on AR smart glasses provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the module connection of the farmed fish phenotype measurement system based on AR smart glasses provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention.

[0023] Reference numerals: 110: Image acquisition module; 120: Image sending module; 130: Data display module; 210: Image receiving module; 220: Image analysis module; 230: Data return module; 510: Processor; 520: Communication interface; 530: Memory; 540: Communication bus. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0025] The following combination Figure 1 The present invention describes a method for measuring farmed fish phenotypes based on AR smart glasses, which is applied to AR smart glasses and includes: step 101, collecting fish images.

[0026] The fish images are obtained by collecting images through a stereoscopic vision acquisition system constructed by two RGB cameras of AR smart glasses.

[0027] In the present invention, two high-resolution RGB cameras are installed on the left and right sides of the front end of the AR smart glasses to form a stereoscopic vision system to capture dynamic images of fish in the water in real time. The system ensures that the captured image is consistent with the wearer's perspective, which is convenient for natural inspection. The camera is equipped with a wide-angle lens with a frame rate of 30fps, which can support continuous dynamic capture of single or multiple fish images. After acquiring the dynamic image, a single-frame static image of the fish to be observed is extracted from the dynamic image, and phenotypic data is acquired based on the image of the fish to be observed. AR smart glasses can use XREAL One, and can also be adapted to other AR devices with spatial perception and image processing capabilities, such as the Meta Quest series and Rokid Max. Wearable and portable measurement is realized to improve adaptability to the breeding site. The wearer can complete information acquisition during the inspection process without additional operation, thereby improving efficiency.

[0028] Step 102: Send the fish image to the edge computing terminal.

[0029] The collected fish images are sent to the edge computing terminal through the wireless transmission device of the AR smart glasses.

[0030] To reduce the amount of transmitted data, fish images are compressed. This involves color space conversion (RGB to YCbCr), block processing (such as JPEG's 8x8 block division), transform coding (using DCT or DWT to convert the image to the frequency domain, concentrating energy on low frequencies), quantization (lossy quantization of transform coefficients (retaining low frequencies and discarding high frequencies), and entropy coding (using Huffman coding, arithmetic coding, etc.) to eliminate statistical redundancy in the quantized data, resulting in a compressed fish image.

[0031] In the present invention, the compressed fish images are sent in real time to an edge computing device (such as Jetson Orin) via Wi-Fi or Bluetooth, establishing a stable wireless communication link and providing efficient data input for subsequent computing links.

[0032] Step 103: Receive the fish phenotypic data generated by the edge computing terminal and superimpose the fish phenotypic data onto the fish image.

[0033] Specifically, receiving fish phenotypic data, wherein the fish phenotypic data is obtained by processing fish images through edge computing equipment; The fish phenotypic data is superimposed onto the fish image using augmented reality. In this invention, fish phenotypic data is transmitted to AR smart glasses via wireless transmission equipment. The AR smart glasses overlay the phenotypic data in the wearer's field of view in real time using spatial augmented reality. The wearer can observe fish measurements in real time during the inspection without stopping or operating the device. The measurement data can also be recorded locally or uploaded to a cloud database, enabling individual tracking and group assessment.

[0034] refer to Figure 2 , discloses a method for measuring the phenotype of farmed fish based on AR smart glasses, which is applied to an edge computing terminal, including: step 201, receiving a fish image sent by the AR smart glasses.

[0035] In the present invention, the edge computing terminal receives the compressed fish image via a Wi-Fi wireless network or Bluetooth, and decompresses it to obtain the original fish image.

[0036] Step 202: Analyze the fish image to generate fish phenotypic data.

[0037] performing image recognition analysis on the fish picture based on a pre-trained image recognition model to obtain fish phenotypic data; The image recognition model is obtained by training a preset neural network model using pre-acquired fish images with labeled information.

[0038] In the present invention, the edge computing device runs the YOLOv5s deep learning model to complete fish body recognition and mask segmentation, and estimates the volume in combination with the calibration scale.

[0039] Specifically, fish images were annotated using LabelMe or CVAT, with bounding boxes (YOLO format) and pixel-level masks (COCO format). The instance segmentation dataset must contain the precise outline of each fish. Images were resized to 640×640 (the default input for YOLOv5s), maintaining their aspect ratio and filling edges with gray bars to prevent distortion. Pixel values ​​were normalized to the range [0, 1] to accelerate model convergence. Data augmentation was performed using mosaics (four-image stitching), random rotations (±30°), HSV color perturbations (hue ±0.1, saturation / value ±0.5), and blurring (Gaussian kernel σ = 0.5) to simulate complex scenes.

[0040] The YOLOv5s model structure includes: Backbone: CSPDarknet53 (Cross-Stage Local Network), which uses the CSP structure to reduce computational complexity and extract multi-scale features.

[0041] Neck: PANet (Path Aggregation Network), which fuses deep semantic features with shallow details to enhance small object detection.

[0042] Head: Three detection heads (20×20, 40×40, 80×80 grids), which predict targets of different scales and output bounding box coordinates (cx, cy, w, h), confidence and category probability.

[0043] A segmentation branch is added after the detection head. Each detection box corresponds to a 28×28 mask prototype, and the final mask is generated through matrix multiplication. RoI Align is used to extract features within the detection box, retain spatial information, and avoid quantization errors.

[0044] An adaptive threshold (e.g., 0.5) is applied to the mask output to distinguish foreground from background. A closing operation (3×3 kernel) is used to fill holes, and an opening operation is used to remove isolated noise points. GrabCut or a conditional random field (CRF) is used to refine edges and improve segmentation accuracy. This high-precision segmentation of the fish body and background reduces volume estimation errors.

[0045] Based on the segmentation results, the stereo calibration method is used to restore the camera parameters and establish a three-dimensional coordinate system. The monocular depth estimation (such as the MiDaS model) predicts the depth map, which is combined with the mask to generate a point cloud, and the Poisson surface reconstruction is used to calculate the precise volume.

[0046] In the phenotypic data, body length refers to the length of the central axis from the head to the caudal peduncle of the fish body, which is converted into physical distance based on the midline pixel length in the mask image; body length refers to the length of the central axis from the head to the caudal peduncle of the fish body, which is converted into physical distance based on the midline pixel length in the mask image; total length refers to the maximum straight-line distance from the head to the end of the caudal fin; body weight is estimated based on the 2D image mask through volume reconstruction method, and combined with the trained volume-weight regression model to complete weight prediction.

[0047] Among them, the volume-weight regression model structure used is a lightweight convolutional neural network, which has been compressed and optimized for the AR edge deployment environment. It has the advantages of low computing resource usage and strong real-time performance.

[0048] Specifically, the volume of each fish is received from the volume estimation module V and morphological parameters L , W , H , (L, W, H are length, width, and height respectively) to obtain the species label from the classification module (such as the classification head output of YOLOv5 "Salmo salar").

[0049] Single species scenario: call the pre-trained linear regression coefficient and directly calculate: W = e log(a)+b⋅log(V) (Power law model restoration).

[0050] Multi-species scenario: Select the corresponding model file (such as model_rainbow_trout.pkl) based on the species label and load the Scikit-learn pipeline for inference.

[0051] A dynamic calibration mechanism regularly samples actual weight measurements (e.g., one per 100 fish) and calculates the prediction error. If the error persists >8%, model retraining is triggered. Streaming learning (River library) is used to incrementally update model parameters with new samples, avoiding full retraining.

[0052] Weight results are generated for each fish and output in a pre-set format. Through a volume-weight regression model, the system automatically converts non-contact images into biomass indicators, improving weight monitoring efficiency and reducing costs.

[0053] Step 203: Send the fish phenotypic data to the AR smart glasses.

[0054] After the fish phenotypic data is sent to the AR smart glasses, the fish phenotypic data is superimposed on the fish image on the AR smart glasses, and the fish phenotypic data is superimposed in the wearer's field of view in real time using spatial augmented reality.

[0055] The present invention provides a method for measuring farmed fish phenotypic data using AR smart glasses. The AR glasses capture fish images, wirelessly transmit the images to an edge computing terminal for recognition, and obtain fish phenotypic data. The AR glasses then overlay the fish phenotypic data onto the fish images using augmented reality. This enables real-time dynamic measurement of fish phenotypic data, and the edge computing device uses image recognition to obtain fish phenotypic data, improving measurement accuracy.

[0056] refer to Figure 3 The present invention also discloses a farmed fish phenotype measurement device based on AR smart glasses, comprising: An image acquisition module 110 is used to acquire fish images; An image sending module 120 is configured to send the fish image to an edge computing terminal; The data display module 130 is used to receive the fish phenotypic data generated by the edge computing terminal and superimpose the fish phenotypic data onto the fish image.

[0057] The collection of fish images includes: The fish images are obtained by collecting images through a stereoscopic vision acquisition system constructed by two RGB cameras of AR smart glasses.

[0058] Receive fish phenotypic data generated by edge computing terminals and overlay the fish phenotypic data onto fish images, including: Receiving fish phenotypic data, the fish phenotypic data being obtained by processing fish images through edge computing equipment; The fish phenotypic data is superimposed on the fish image through augmented reality.

[0059] refer to Figure 4 , the present invention also discloses a farmed fish phenotype measurement device based on AR smart glasses, comprising: an image receiving module 210 for receiving fish images sent by the AR smart glasses; An image analysis module 220 is used to analyze the fish image and generate fish phenotypic data; The data return module 230 is used to send the fish phenotypic data to the AR smart glasses.

[0060] The step of analyzing the fish image to generate fish phenotypic data includes: performing image recognition analysis on the fish picture based on a pre-trained image recognition model to obtain fish phenotypic data; The image recognition model is obtained by training a preset neural network model using pre-acquired fish images with labeled information.

[0061] Performing image recognition analysis on the fish image based on a pre-trained image recognition model to obtain fish phenotypic data, including: performing mask segmentation on the fish image; Estimate fish body length, body width, total length and volume based on mask segmentation results and combined with calibration size; The body length, body width, total length and volume of fish as well as the fish weight estimated based on the volume were used as fish phenotypic data.

[0062] The present invention provides a farmed fish phenotypic measurement device based on AR smart glasses. The AR glasses collect fish images, which are wirelessly transmitted to an edge computing terminal for recognition and acquisition of fish phenotypic data. The AR glasses then overlay the fish phenotypic data onto the fish images through augmented reality. This allows for real-time dynamic measurement of fish phenotypic data, and the edge computing device uses image recognition to obtain fish phenotypic data, improving measurement accuracy. Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call logic instructions in the memory 530 to execute the farmed fish phenotyping method based on AR smart glasses.

[0063] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0064] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the farmed fish phenotypic measurement method based on AR smart glasses provided by the above methods.

[0065] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the farmed fish phenotypic measurement method based on AR smart glasses provided by the above methods.

[0066] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0067] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for measuring the phenotype of farmed fish based on AR smart glasses, characterized in that: Applied to AR smart glasses, including: Collect fish images; Sending the fish image to an edge computing terminal; Receive fish phenotypic data generated by edge computing terminal processing and overlay the fish phenotypic data onto the fish image.

2. The method for measuring the phenotype of farmed fish based on AR smart glasses according to claim 1, characterized in that: The collecting of fish images comprises: The fish images are obtained by collecting images through a stereoscopic vision acquisition system constructed by two RGB cameras of AR smart glasses.

3. The method for measuring the phenotype of farmed fish based on AR smart glasses according to claim 1, characterized in that: The receiving of fish phenotypic data generated by processing at the edge computing terminal and superimposing the fish phenotypic data onto the fish image includes: Receiving fish phenotypic data, the fish phenotypic data being obtained by processing fish images through edge computing equipment; The fish phenotypic data is superimposed on the fish image through augmented reality.

4. A method for measuring the phenotype of farmed fish based on AR smart glasses, characterized in that: Applied to edge computing terminals, including: Receive fish images sent by AR smart glasses; Analyzing the fish image to generate fish phenotypic data; The fish phenotypic data is sent to AR smart glasses.

5. The method for measuring the phenotype of farmed fish based on AR smart glasses according to claim 4, characterized in that: The analyzing the fish image to generate fish phenotypic data includes: performing image recognition analysis on the fish picture based on a pre-trained image recognition model to obtain fish phenotypic data; The image recognition model is obtained by training a preset neural network model using pre-acquired fish images with labeled information.

6. The method for measuring the phenotype of farmed fish based on AR smart glasses according to claim 5, characterized in that: The performing image recognition analysis on the fish picture based on the pre-trained image recognition model to obtain fish phenotypic data includes: performing mask segmentation on the fish image; Estimate fish body length, body width, total length and volume based on mask segmentation results and combined with calibrated dimensions; The body length, body width, total length and volume of fish as well as the fish weight estimated based on the volume were used as fish phenotypic data.

7. A phenotypic measurement device for farmed fish based on AR smart glasses, characterized in that: include: An image acquisition module, used for acquiring fish images; An image sending module, configured to send the fish image to an edge computing terminal; The data display module is used to receive the fish phenotypic data generated by the edge computing terminal and overlay the fish phenotypic data onto the fish image.

8. A phenotypic measurement device for farmed fish based on AR smart glasses, characterized in that: include: An image receiving module, used to receive fish images sent by AR smart glasses; An image analysis module, configured to analyze the fish image and generate fish phenotypic data; The data return module is used to send the fish phenotypic data to the AR smart glasses.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the farmed fish phenotype measurement method based on AR smart glasses is implemented as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the farmed fish phenotype measurement method based on AR smart glasses is implemented as described in any one of claims 1 to 6.

Images