A shrimp information detection method and system based on computer binocular vision

Abstract

The present application discloses a shrimp information detection method and system based on computer binocular vision. The method uses a target detection model and feature matching technology to accurately measure the length and weight of shrimps, uses a binocular camera to obtain a video stream, identifies the key point coordinates of shrimps, and obtains the three-dimensional coordinates of shrimps through key point matching, and then calculates the length and weight of shrimps. The present invention improves detection accuracy, reduces manpower and cost input, and provides an efficient and economical solution for aquaculture and ecosystem management.

Description

Technical Field

[0001] The invention relates to the technical field of computer binocular vision and pattern recognition, and in particular to a shrimp information detection method and system based on computer binocular vision. Background Art

[0002] Shrimp farming is an important economic activity. Regularly measuring the length and weight of shrimp can effectively monitor the growth rate and development of shrimp. This is crucial for evaluating feed efficiency, adjusting feeding strategies, and predicting harvest time, which has a direct effect on improving farming efficiency and economic benefits.

[0003] Traditional methods for detecting shrimp growth usually include manual observation and regular sampling analysis. Manual observation is easily interfered by subjective factors and usually relies on regular sampling, so it cannot provide continuous and real-time data and misses important information about the shrimp growth process. Moreover, regular sampling requires the shrimp to be salvaged ashore and then measured, which causes the death of some shrimp. Manual observation and sampling analysis require a lot of human resources and time, and are costly.

[0004] In addition, in underwater environments, due to the refraction, scattering, brightness and the influence of suspended matter in the water, the measurement accuracy of computer vision technology is often greatly challenged. Manual observation may result in inaccurate information detection. Summary of the invention

[0005] Based on this, the embodiments of the present application provide a shrimp information detection method and system based on computer binocular vision, which uses computer vision technology to directly measure shrimp information underwater to solve the problems of manual measurement consuming manpower and material resources, and inaccurate shrimp information detection.

[0006] In a first aspect, a method for detecting shrimp information based on computer binocular vision is provided, the method comprising:

[0007] The video stream of underwater shrimp is obtained through a binocular camera, and the left and right binocular images are extracted frame by frame. The image annotation rules are formulated to annotate the key points of the shrimp to train the target detection model; the key points include the eyes, head, tail and the coordinates of each abdominal segment;

[0008] Using the trained object detection model, the input left and right binocular images are used for object detection to identify the position and key points of the shrimp. Then, the key points are extracted through the feature extractor to generate feature descriptors.

[0009] Use graph neural network to process feature descriptors, and obtain feature matching results through self-attention and cross-attention mechanisms;

[0010] The shrimp information detection result is determined according to the feature matching result; wherein the shrimp information detection result at least includes the length and weight of the shrimp.

[0011] Optionally, annotate the shrimp key points to train an object detection model, including:

[0012] The left and right binocular images are preprocessed, and the target detection model is trained using the YOLO V8 algorithm so that the target detection model can identify the object bounding box and feature points of the shrimp and generate descriptors of the feature points.

[0013] Optionally, feature descriptors are processed using graph neural networks, and after feature matching through self-attention and cross-attention mechanisms, the following is also included:

[0014] The matching scores between feature points are calculated, and the final matching point allocation matrix is ​​generated based on the scores and thresholds. Then the key point disparity is calculated, and the depth information is calculated based on the camera parameters and disparity to obtain the three-dimensional coordinate points in space.

[0015] Optionally, in feature matching through self-attention and cross-attention mechanisms, the loss function of the matching model is specifically:

[0016]

[0017] Where N is the number of key point pairs; L ij represents the true label, which is used to indicate whether the i-th key point matches the j-th key point; D ij represents the feature distance between the i-th key point and the j-th key point; m represents the threshold, which is used to control the distance between mismatched key points.

[0018] Optionally, determining the shrimp information detection result according to the feature matching result includes:

[0019] The obtained three-dimensional coordinate points are used to calculate the distance between two points in the three-dimensional space using the Euclidean distance formula to obtain the length of the shrimp, and the weight of the shrimp is determined based on the corresponding relationship between the shrimp body length and weight.

[0020] Optionally, the correspondence between the shrimp body length and weight includes:

[0021]

[0022] Among them, y represents weight in g; x represents body length in mm.

[0023] In a second aspect, a shrimp information detection system based on computer binocular vision is provided, the system comprising:

[0024] The training module is used to obtain the video stream of underwater shrimps through a binocular camera, extract the left and right binocular images frame by frame, formulate image annotation rules, annotate the key points of shrimps to train the target detection model; the key points include the eyes, head, tail and the coordinates of each abdominal segment;

[0025] The feature extraction module is used to use the trained target detection model to perform target detection on the input left and right binocular images, identify the location and key points of the shrimp, and then use the feature extractor to extract features of the key points and generate feature descriptors;

[0026] The matching module is used to process feature descriptors using graph neural networks and obtain feature matching results through self-attention and cross-attention mechanisms;

[0027] The determination module is used to determine the shrimp information detection result according to the feature matching result; wherein the shrimp information detection result at least includes the length and weight of the shrimp.

[0028] In a third aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the shrimp information detection method described in any one of the first aspects is implemented.

[0029] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the shrimp information detection method described in any one of the first aspects is implemented.

[0030] In a fifth aspect, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the shrimp information detection method described in any one of the first aspects is implemented.

[0031] The beneficial effects of the technical solution provided by the embodiment of the present application include: through the computer binocular vision algorithm, the length, width and other data of shrimps can be accurately measured in water, and the weight of shrimps can be predicted. Compared with traditional manual observation and sampling analysis, this method reduces the dependence on manual operation, reduces human errors, and improves the accuracy and efficiency of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0033] Figure 1A flow chart of the steps of the shrimp information detection method provided in the embodiment of the present application;

[0034] Figure 2 This is a network structure diagram of high-precision shrimp information detection based on binocular vision in an embodiment of the present application;

[0035] Figure 3 This is a style diagram of shrimp key points marked in the embodiment of this application;

[0036] Figure 4 This is a mathematical model diagram of shrimp body length and weight in the embodiment of this application;

[0037] Figure 5 This is a block diagram of a shrimp information detection system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.

[0040] The present application provides a method for detecting the growth of shrimps underwater. By adopting a computer binocular vision algorithm, the length, width and other data of shrimps can be measured in water, and the weight of shrimps can be predicted using the data. Traditional methods for detecting the growth of shrimps usually include manual observation and regular sampling analysis. Manual observation is easily interfered by subjective factors and usually relies on regular sampling. Therefore, it is impossible to provide continuous and real-time data, and important information about the growth process of shrimps is missed. Manual observation and sampling analysis require a lot of human resources and time, and the cost is high. Therefore, underwater intelligent detection of shrimp growth is crucial for aquaculture and ecosystem management. However, in underwater environments, due to the refraction, scattering, brightness of light and the influence of suspended matter in the water, the underwater measurement accuracy of computer vision technology is often greatly challenged. In response to the above problems and challenges, this paper invents a high-precision detection method for shrimp information based on computer binocular vision. This method uses a target detection model, takes the video stream obtained from the binocular camera as input, identifies the shrimp, and outputs the coordinates of the shrimp's key points and the descriptors of the coordinates. It uses the descriptors of the key points in the left and right binocular views to match the key points, obtain the parallax, depth and spatial three-dimensional coordinates of the key points, and then uses the Euclidean distance formula to obtain the length of the shrimp. The length is input into the mathematical model to obtain information such as the weight of the shrimp. This innovative method helps to accurately obtain shrimp growth data in practical applications. By combining computer binocular vision technology with shrimp growth data detection, it not only improves detection accuracy, but also reduces manpower input and cost overhead.

[0041] In order to avoid the influence of refraction and scattering of underwater light on measurement accuracy, this application does not adopt the commonly used land-based length measurement method based on point cloud, disparity map or depth map. It innovatively introduces a network structure of dual-target detection based on key point matching to measure the length of shrimp. It not only has higher accuracy and stronger underwater anti-interference ability, but also the network runs faster.

[0042] Please refer to Figure 1 , which shows a flow chart of a shrimp information detection method based on computer binocular vision in one embodiment, the method may include the following steps:

[0043] S1, obtains the video stream of underwater shrimps through a binocular camera, extracts left and right binocular images frame by frame, and formulates image annotation rules to annotate the key points of shrimps to train the target detection model.

[0044] Among them, the key points include the eyes, head, tail and the coordinates of each abdominal segment.

[0045] In this step, we first capture the video stream of underwater shrimps through a binocular camera, and extract the left and right binocular images frame by frame. Then, according to the pre-established image annotation rules, we annotate the key points of the shrimps, including the eyes, head, tail, and the coordinates of each abdominal segment. These annotated data will be used to train the improved target detection model based on the YOLO V8 algorithm, so that it can identify and locate the key parts of shrimps.

[0046] S2 uses the trained target detection model to perform target detection on the input left and right binocular images, identify the position and key points of the shrimp, and then extract features of the key points through the feature extractor and generate feature descriptors.

[0047] Using the object detection model trained in step S1, the input left and right binocular images are subjected to object detection to identify the position and key points of the shrimp. Then, the feature extractor extracts features from these key points and generates feature descriptors.

[0048] S3, uses graph neural network to process feature descriptors, and performs feature matching through self-attention and cross-attention mechanisms to obtain feature matching results.

[0049] The feature descriptors generated in step S2 are processed using a graph neural network. Through the self-attention and cross-attention mechanisms, the graph neural network is able to perform feature matching, calculate the matching scores between feature points, and generate the final matching point allocation matrix based on the matching scores and thresholds.

[0050] S4, determining the shrimp information detection result according to the feature matching result.

[0051] Among them, the shrimp information detection result includes at least the length and weight of the shrimp. According to the feature matching result obtained in step S3, the parallax of the key points is calculated, the depth information is calculated according to the camera parameters and the parallax, and the three-dimensional coordinate points in space are calculated using the depth information. The Euclidean distance formula is used to calculate the distance between two points in three-dimensional space, so as to obtain the length of the shrimp. Then, the weight, chest carapace length, head carapace length, abdominal segment length, abdominal segment height and other information of the shrimp are estimated in combination with the mathematical model, and the shrimp information detection result is finally determined, including at least the length and weight of the shrimp.

[0052] like Figure 2The network structure of this application includes three modules. The first algorithm module is an improved target detection module based on the YOLO V8 algorithm. The algorithm inputs two left and right eye pictures obtained from the binocular camera, and the output picture is marked with the object boundary box and feature points detected by the algorithm. The module for generating feature points using the YOLO V8 method has been modified, and the descriptor of the feature points is output through the feature extraction network branch for the subsequent feature matching module. The second module is the feature matching module, which is used to match the feature points in the left and right pictures with feature descriptors output in the first module. The third module is the data prediction module, which calculates the length of the shrimp based on the key point disparity calculated by the previous module, and predicts the weight of the shrimp and other information through other mathematical models.

[0053] 1. Shrimp detection module:

[0054] The detection module is trained by manually annotating the dataset, and the annotation style is as follows: Figure 3 ,After the training is completed, the object detection is performed on the ,left and left images respectively to identify the shrimp, the position of the shrimp, and the key points of the shrimp, and the ,results are passed to the feature extractor to obtain the ,feature descriptors of the feature points in the left and right images.

[0055] 2. Key point matching model:

[0056] Use graph neural network to process feature descriptors, perform feature matching through self-attention and cross-attention mechanisms, calculate the matching scores between feature points, and generate the final matching point allocation matrix based on the matching scores and thresholds. Then, get the matching results, including the coordinates and confidence of the matching points, and display the matching point pairs in the left and right eye images of the original input. Matching model loss function:

[0057]

[0058] Where N is the number of key point pairs; L ij represents the true label, that is, Figure 2 The i-th key point in Figure 3 Whether the jth key point in D matches (1 means match, 0 means no match). N is the number of key point pairs. ij Indicates Figure 2 The i-th key point in Figure 3 The characteristic distance (Euclidean distance) between the jth key points in ; m represents the threshold, which is used to control the distance between mismatched key points.

[0059] 3. Calculate the length of the shrimp and other information based on the matching results:

[0060] Calculate the disparity of the matched key points in the left and right images, obtain the depth information of the corresponding key points through the disparity, and obtain the spatial three-dimensional coordinates of the key points from the depth and the internal and external parameters of the camera. According to the distance formula between two points in space, calculate the length of the shrimp, and thus predict the weight of the shrimp and other information. Figure 4 The mathematical model diagram of shrimp body length and weight is given. Among them:

[0061]

[0062] Among them, y represents weight in g; x represents body length in mm.

[0063] In another embodiment of the present application, the above method can be expanded into the following steps:

[0064] Step 1: Obtain a video of underwater shrimp from a binocular camera, and obtain left and right binocular images frame by frame.

[0065] Step 2: Establish image annotation rules, annotate the shrimp eyes, head, tail and the coordinates of each abdominal segment to train the target detection model.

[0066] Step 3: Extract several consecutive frames from the video obtained by the binocular camera as input, and obtain the shrimp position information and the coordinates of the key points of the eyes, head, tail, and each abdominal segment through the trained target detection model.

[0067] Step 4: The feature extractor extracts features from the left and right binocular images of the shrimp containing key point information, and generates feature descriptors for each feature point in the left and right binocular images.

[0068] Step 5: Use graph neural network to process feature descriptors and perform feature matching through self-attention and cross-attention mechanisms.

[0069] Step 6: Calculate the matching scores between feature points.

[0070] Step 7: Generate the final matching point allocation matrix based on the matching score and threshold.

[0071] Step 8: Output the final matching results, including the coordinates and confidence of the matching points.

[0072] Step 9: Calculate the parallax of key points, calculate the depth information based on the camera parameters and parallax, use the depth information to calculate the three-dimensional coordinate points in space, and use the Euclidean distance formula to calculate the distance between two points in three-dimensional space to obtain the length of the shrimp.

[0073] Step 10: Estimate the shrimp's weight, thorax length, head carapace length, abdominal segment length, abdominal segment height and other information by combining the shrimp's length with a mathematical function model.

[0074] Step 11: Return the shrimp length, weight and other information to the mobile terminal.

[0075] In one embodiment of the present application, Figure 5 , and also provides a shrimp information detection system based on computer binocular vision, the system comprising:

[0076] The training module is used to obtain the video stream of underwater shrimps through a binocular camera, extract the left and right binocular images frame by frame, formulate image annotation rules, annotate the key points of shrimps to train the target detection model; the key points include the eyes, head, tail and the coordinates of each abdominal segment;

[0077] The feature extraction module is used to use the trained target detection model to perform target detection on the input left and right binocular images, identify the location and key points of the shrimp, and then use the feature extractor to extract features of the key points and generate feature descriptors;

[0078] The matching module is used to process feature descriptors using graph neural networks and obtain feature matching results through self-attention and cross-attention mechanisms;

[0079] The determination module is used to determine the shrimp information detection result according to the feature matching result; wherein the shrimp information detection result at least includes the length and weight of the shrimp.

[0080] The specific definition of the shrimp information detection system based on computer binocular vision can be found in the above definition of the shrimp information detection method based on computer binocular vision, which will not be repeated here. Each module in the above shrimp information detection system based on computer binocular vision can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0081] In one embodiment, an electronic device is provided, which may be a computer. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for shrimp information detection data based on computer binocular vision. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a shrimp information detection method based on computer binocular vision is implemented.

[0082] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above shrimp information detection method based on computer binocular vision are implemented.

[0083] In one embodiment of the present application, a computer program product is provided, including a computer program / instruction, which implements the steps of the above shrimp information detection method based on computer binocular vision when executed by a processor.

[0084] The computer-readable storage medium and computer program product provided in this embodiment have similar implementation principles and technical effects to those of the above method embodiments, and are not described in detail here.

[0085] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0086] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A shrimp information detection method based on computer binocular vision, characterized in that: The method comprises: The video stream of underwater shrimp is obtained through a binocular camera, and the left and right binocular images are extracted frame by frame. The image annotation rules are formulated to annotate the key points of the shrimp to train the target detection model; the key points include the eyes, head, tail and the coordinates of each abdominal segment; Using the trained object detection model, the input left and right binocular images are used for object detection to identify the position and key points of the shrimp. Then, the key points are extracted through the feature extractor to generate feature descriptors. Use graph neural network to process feature descriptors, and obtain feature matching results through self-attention and cross-attention mechanisms; Determine the shrimp information detection result according to the feature matching result; wherein the shrimp information detection result at least includes the length and weight of the shrimp; Using graph neural networks to process feature descriptors, after feature matching through self-attention and cross-attention mechanisms, it also includes: Calculate the matching score between feature points, and generate the final matching point allocation matrix based on the score and threshold. Then calculate the key point disparity, and calculate the depth information based on the camera parameters and disparity to obtain the three-dimensional coordinate points in space. In feature matching through self-attention and cross-attention mechanisms, the loss function of the matching model is specifically: ， in, N is the number of keypoint pairs; L ij Represents the true label, which is used to characterize whether the i-th key point matches the j-th key point; D ij Represents the feature distance between the i-th key point and the j-th key point; m Represents the threshold value, which is used to control the distance between mismatched keypoints.

2. The shrimp information detection method according to claim 1, characterized in that: Label the key points of shrimp to train the object detection model, including: The left and right binocular images are preprocessed, and the target detection model is trained using the YOLO V8 algorithm so that the target detection model can identify the object bounding box and feature points of the shrimp and generate descriptors of the feature points.

3. The shrimp information detection method according to claim 1, characterized in that: The shrimp information detection results are determined based on the feature matching results, including: The obtained three-dimensional coordinate points are used to calculate the distance between two points in the three-dimensional space using the Euclidean distance formula to obtain the length of the shrimp, and the weight of the shrimp is determined based on the corresponding relationship between the shrimp body length and weight.

4. The shrimp information detection method according to claim 3, characterized in that: The corresponding relationship between shrimp body length and weight includes: ， in, y Indicates weight in g; x Indicates body length in mm.

5. A shrimp information detection system based on computer binocular vision, characterized in that: The system comprises: The training module is used to obtain the video stream of underwater shrimps through a binocular camera, extract the left and right binocular images frame by frame, formulate image annotation rules, annotate the key points of shrimps to train the target detection model; the key points include the eyes, head, tail and the coordinates of each abdominal segment; The feature extraction module is used to use the trained target detection model to perform target detection on the input left and right binocular images, identify the location and key points of the shrimp, and then use the feature extractor to extract features of the key points and generate feature descriptors; The matching module is used to process feature descriptors using graph neural networks and obtain feature matching results through self-attention and cross-attention mechanisms; A determination module, used to determine the shrimp information detection result according to the feature matching result; wherein the shrimp information detection result at least includes the length and weight of the shrimp; Using graph neural networks to process feature descriptors, after feature matching through self-attention and cross-attention mechanisms, it also includes: Calculate the matching score between feature points, and generate the final matching point allocation matrix based on the score and threshold. Then calculate the key point disparity, and calculate the depth information based on the camera parameters and disparity to obtain the three-dimensional coordinate points in space. In feature matching through self-attention and cross-attention mechanisms, the loss function of the matching model is specifically: ， in, N is the number of keypoint pairs; L ij Represents the true label, which is used to characterize whether the i-th key point matches the j-th key point; D ij Represents the feature distance between the i-th key point and the j-th key point; m Represents the threshold value, which is used to control the distance between mismatched keypoints.

6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the shrimp information detection method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the shrimp information detection method according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the shrimp information detection method according to any one of claims 1 to 4 is implemented.

Images