Bimodal peacock sex detection method and system
By combining the dual-modal information of RGB cameras and event cameras, guppies gender detection is performed, and the problems of inefficient and poor accuracy of traditional manual detection are solved, achieving higher detection accuracy and efficiency.
Patent Information
- Application Number
- CN202510112205.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-24
- Publication Date
- 2025-05-16
AI Technical Summary
Traditional fish gender identification methods rely on manual observation, and there are problems such as large subjective judgment errors and inefficiency, making it difficult to accurately detect the gender of guppies.
The dual-modal guppies gender detection method is used to obtain RGB videos through the RGB camera, and the video is converted into grayscale superimposed videos through the event camera. The information is fused with the two modal information and input it into the gender detection model for detection.
It improves the accuracy of guppies' gender detection, saves a lot of manpower and time, and can better capture the contours and color characteristics of guppies.
Smart Images
Figure CN120014520A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of aquaculture, and in particular to a dual-mode guppy gender detection method and system. Background Art
[0002] In the field of aquaculture, fish sex detection can be used to control the sex ratio and stocking density, improve feed utilization and aquaculture output. Traditional fish sex identification methods mainly rely on manual observation, which not only consumes a lot of manpower and time, but also highly depends on the experience and professional knowledge of the identification personnel. There are many disadvantages that are difficult to overcome, such as large subjective judgment errors and low efficiency.
[0003] Guppies are often used as research subjects in laboratories because of their short reproductive cycle and remarkable reproductive capacity. However, manual observation is subjective, prone to errors, and too time-consuming to distinguish. In recent years, some scientists have noticed that biological vision systems are much better than traditional digital vision systems in terms of quality, power consumption, and response speed. Event cameras are a type of biologically inspired silicon retinal visual sensor. Event cameras can better capture the contour information of active guppies, but the grayscale superposition images obtained by event cameras do not have obvious color information. RGB cameras can only capture the color characteristics of guppies of different genders, but cannot capture the behavior and contour characteristics of guppies. Male fish are usually more active than female fish, swim faster, and behave more agilely. Female fish are usually gentler, calmer, and behave more restrained than male fish. The accuracy of detecting the gender of guppies using event cameras or RGB cameras alone cannot meet the needs of experiments. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a dual-modal guppy gender detection method and system, which improves the accuracy of guppy gender detection and saves a lot of manpower and time.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] A dual-mode guppy gender detection method comprises the following steps:
[0007] Acquire an RGB video of the guppy through an RGB camera, and convert the RGB video into a grayscale overlay video through an event camera;
[0008] Extract frames from the RGB video and the grayscale overlay video to obtain an RGB image and a grayscale overlay image of the same time frame, and perform perspective transformation on the grayscale overlay image to obtain an aligned RGB image and grayscale overlay image;
[0009] Inputting the aligned RGB image and the grayscale superposition image as feature tensors into a gender detection model for information fusion to obtain a fused feature tensor with highlighted features, and detecting the gender of the guppy based on the fused feature tensor with highlighted features;
[0010] Among them, the information fusion step includes: splitting the input feature tensor into two parts, left original features and right original features according to the channel dimension, numerically scaling and mask weighting the left original features and the right original features to obtain left mask weighted features and right mask weighted features, adding the left mask weighted features and the left original features, adding the right mask weighted features and the right original features, further extracting features through corresponding bottleneck convolution layers to obtain grayscale superimposed image processing features and RGB image processing features, splicing the grayscale superimposed image processing features and the RGB image processing features on the channel dimension, and adjusting the weights to obtain a feature tensor with highlighted features after fusion.
[0011] Furthermore, the specific steps of converting the RGB video into a grayscale overlay video through an event camera include: obtaining a binary file according to the RGB video, and converting the binary file into a grayscale overlay video through an event camera.
[0012] Furthermore, the specific step of obtaining a binary file according to the RGB video includes: collecting FPN in the RGB video by an event camera, and generating a binary file of the event stream.
[0013] Furthermore, if the pixel brightness of the FPN exceeds a set threshold, it is recorded as an event and a binary file of the event stream is generated.
[0014] Furthermore, the event includes an on event and an off event. When the brightness of the pixel increases, it is recorded as an on event, and when the brightness of the pixel decreases, it is recorded as an off event.
[0015] Furthermore, the fused highlighted feature tensor includes color information of the RGB image and contour information of the grayscale overlay image.
[0016] Furthermore, the gender detection model is an improved yolov8 model, and its training steps specifically include: using labelimg to perform data annotation on the aligned RGB image, dividing the RGB image and grayscale overlay image with completed data annotation into a training set and a test set, the training set is used to be put into the gender detection model for training, and the test set is used to test whether the gender detection model can detect the gender of guppies, and the accuracy of the gender detection model is evaluated by map50.
[0017] Furthermore, the specific steps of using the labelimg to perform data annotation include: according to the aligned RGB image, the guppies are divided into two categories of labels, male fish and female fish, for annotation, the male fish has bright colors, and the female fish has a single color.
[0018] Furthermore, the mask is obtained by convolving a 1×1 convolution kernel with a stride of 1 and no padding, combined with a learnable bias term.
[0019] According to another aspect of the present invention, there is provided a dual-mode guppy gender detection system, comprising:
[0020] A video acquisition module, used for acquiring an RGB video of the guppy through an RGB camera, and converting the RGB video into a grayscale overlay video through an event camera;
[0021] An image alignment module is used to extract frames from the RGB video and the grayscale overlay video to obtain an RGB image and a grayscale overlay image of the same time frame, and to perform perspective transformation on the grayscale overlay image to obtain an aligned RGB image and grayscale overlay image;
[0022] A gender detection module, used for inputting the aligned RGB image and the grayscale superposition image as feature tensors into a gender detection model for information fusion, obtaining a fused feature tensor with highlighted features, and detecting the gender of the guppy according to the fused feature tensor with highlighted features;
[0023] Among them, the information fusion step includes: splitting the input feature tensor into two parts, left original features and right original features according to the channel dimension, numerically scaling and mask weighting the left original features and the right original features to obtain left mask weighted features and right mask weighted features, adding the left mask weighted features and the left original features, adding the right mask weighted features and the right original features, further extracting features through corresponding bottleneck convolution layers to obtain grayscale superimposed image processing features and RGB image processing features, splicing the grayscale superimposed image processing features and the RGB image processing features on the channel dimension, and adjusting the weights to obtain a feature tensor with highlighted features after fusion.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] 1. The present invention adopts a front-end fusion method to fuse the information of the two modalities of the event camera and the RGB camera into one input, splits the input feature tensor into two parts according to the channel dimension, performs numerical scaling and mask weighting, adds the mask-weighted feature tensor and the original split feature tensor, further extracts features through corresponding bottleneck convolutional layers, obtains feature tensors after two different branch processing, splices the two feature tensors in the channel dimension, and adjusts the weights to obtain a fused feature tensor, and detects the gender of the guppy according to the fused feature tensor, thereby improving the accuracy of the gender detection of the guppy.
[0026] 2. The present invention adopts a dual-modal fusion method of an event camera and an RGB camera, combines the advantages of an event camera, and can better capture the contour information of active guppies. It also combines the advantages of an RGB camera and can better capture the color information of guppies of different genders. The color of female fish is relatively simple, while the color of male fish is bright. The information of guppies can be comprehensively obtained, saving a lot of manpower and time. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 A schematic diagram of a dual-mode guppy gender detection method proposed by the present invention;
[0028] Figure 2 are the aligned images obtained by perspective transformation, where (2a) is the aligned grayscale overlay image and (2b) is the aligned RGB image;
[0029] Figure 3 are the detection result images, where (3a) is the grayscale overlay image detection result image, and (3b) is the RGB image detection result image. DETAILED DESCRIPTION
[0030] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0031] Abbreviations involved:
[0032] Fixed Pattern Noise: Fix Pattern Noise, FPN
[0033] Squeeze-and-Excitation Networks, SENet
[0034] Example 1
[0035] This embodiment provides a dual-mode guppy gender detection method, such as Figure 1 As shown, the following steps are included:
[0036] S1. Obtain RGB video of guppies through an RGB camera, and convert the RGB video into a grayscale overlay video through an event camera.
[0037] Use an RGB camera to record a video of the guppy breeding tank. Use an event camera to record the captured guppy RGB video to generate a binary file. First, collect fixed pattern noise (FPN) is a term for a specific noise pattern on a digital image sensor, often visible in longer exposure shots, where specific pixels tend to provide brighter intensities above the general background noise. If FPN is not subtracted from the image, the image may show a high level of background noise and therefore become rough. Then record a binary file of the grayscale overlay image in Fixed_event_intensity mode. The event camera uses pixels as units. If the brightness change of a pixel exceeds the set threshold, it will be recorded as an event. The increase in brightness is recorded as an on event and the decrease in brightness is recorded as an off event. After the acquisition is completed, a binary file of the event stream is generated based on the recorded on events and off events. Run the event camera CeleX-5Demo GUI, set the specific frame number to 60, and convert the generated binary file into a grayscale overlay video in Fixed_event_intensity mode.
[0038] S2. Extract frames from the RGB video and the grayscale overlay video to obtain an RGB image and a grayscale overlay image of the same time frame, and perform perspective transformation on the grayscale overlay image to obtain an aligned RGB image and grayscale overlay image.
[0039] Extract frames from the RGB video and grayscale overlay video to get corresponding frames with the same time; read the coordinate array of corresponding point coordinates used for perspective transformation in the RGB video and grayscale overlay video, and calculate the transformation matrix; extract frames from the RGB video and grayscale overlay video to get corresponding frame images, and perform perspective transformation according to the transformation matrix to get aligned grayscale overlay image and RGB image. The aligned image obtained by perspective transformation is as follows: Figure 2 As shown, Figure 2 (2a) is the grayscale overlay image after alignment. Figure 2 (2b) is the aligned RGB image.
[0040] S3. The aligned RGB image and the grayscale overlay image are input as feature tensors into the gender detection model for information fusion to obtain the highlighted feature tensor after fusion, and the gender of the guppy is detected based on the highlighted feature tensor after fusion.
[0041] The gender detection model is an improved yolov8 model, which adopts the front-end fusion method. The steps of information fusion include: first, the information of the two modalities is fused into a feature tensor input, the input feature tensor is split into two parts according to the channel dimension, and the feature information is divided into left original features and right original features. Then the two parts of the features are numerically scaled, and the features after numerical scaling are weighted by the mask generated by convolution (the convolution layer input is 3 channels, and the convolution operation is performed by 1×1 convolution kernel with a step size of 1 and no padding, combined with the learnable bias term, and finally the single-channel mask is output) to obtain the left mask weighted feature and the right mask weighted feature, and the left mask weighted feature is added to the left original feature, and the right mask weighted feature is added to the right original feature, and the features are further extracted through the corresponding bottleneck convolution layer to obtain the grayscale superposition image processing features and RGB image processing features, which each carry the image information obtained by the event camera and the image information obtained by the RGB image. Finally, the features after grayscale overlay image processing and RGB image processing are spliced together in the channel dimension, and then the weights are adjusted through the SEBlock module in SENet to finally output a fused and highlighted feature tensor.
[0042] The gender detection model training steps specifically include:
[0043] The aligned RGB images are labeled using labelimg. The RGB images include the color information of guppies. The male fish has bright colors, while the female fish has a single color. The guppies are divided into two categories: male and female. The labeled RGB images and grayscale overlay images are divided into training sets and test sets. The training set is used to train the gender detection model, and the test set is used to test whether the gender detection model can detect the gender of guppies. The accuracy of the gender detection model is evaluated by map50, and the gender detection of guppies reaches 94.5%.
[0044] The RGB image and grayscale overlay image of the guppy are passed into the trained gender detection model to detect the gender of the guppy. The detection results of the gender detection model are as follows: Figure 3 As shown, Figure 3 (3a) is the detection result of the grayscale overlay image, and (3b) is the detection result of the RGB image.
[0045] Example 2
[0046] This embodiment provides a dual-mode guppy gender detection system, including:
[0047] A video acquisition module is used to acquire RGB video of guppies through an RGB camera and convert the RGB video into a grayscale overlay video through an event camera;
[0048] An image alignment module is used to extract frames from the RGB video and the grayscale overlay video to obtain an RGB image and a grayscale overlay image of the same time frame, and to perform perspective transformation on the grayscale overlay image to obtain an aligned RGB image and grayscale overlay image;
[0049] The gender detection module is used to input the aligned RGB image and the grayscale superposition image as feature tensors into the gender detection model for information fusion, so as to obtain the highlighted feature tensors after fusion, and to detect the gender of the guppy according to the highlighted feature tensors after fusion;
[0050] Among them, the information fusion step includes: splitting the input feature tensor into two parts, left original features and right original features according to the channel dimension, numerically scaling and mask weighting the left original features and the right original features to obtain left mask weighted features and right mask weighted features, adding the left mask weighted features and the left original features, adding the right mask weighted features and the right original features, further extracting features through corresponding bottleneck convolution layers to obtain grayscale superimposed image processing features and RGB image processing features, splicing the grayscale superimposed image processing features and the RGB image processing features on the channel dimension, and adjusting the weights to obtain a feature tensor with highlighted features after fusion.
[0051] The rest is the same as in Example 1.
[0052] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A dual-modal guppy gender detection method, characterized in that: The following steps are involved: Acquire an RGB video of the guppy through an RGB camera, and convert the RGB video into a grayscale overlay video through an event camera; Extract frames from the RGB video and the grayscale overlay video to obtain an RGB image and a grayscale overlay image of the same time frame, and perform perspective transformation on the grayscale overlay image to obtain an aligned RGB image and grayscale overlay image; Inputting the aligned RGB image and the grayscale superposition image as feature tensors into a gender detection model for information fusion to obtain a fused feature tensor with highlighted features, and detecting the gender of the guppy based on the fused feature tensor with highlighted features; Among them, the information fusion step includes: splitting the input feature tensor into two parts, left original features and right original features according to the channel dimension, numerically scaling and mask weighting the left original features and the right original features to obtain left mask weighted features and right mask weighted features, adding the left mask weighted features and the left original features, adding the right mask weighted features and the right original features, further extracting features through corresponding bottleneck convolution layers to obtain grayscale superimposed image processing features and RGB image processing features, splicing the grayscale superimposed image processing features and the RGB image processing features on the channel dimension, and adjusting the weights to obtain a feature tensor with highlighted features after fusion.
2. The dual-mode guppy gender detection method according to claim 1, characterized in that: The specific steps of converting the RGB video into a grayscale overlay video through an event camera include: obtaining a binary file according to the RGB video, and converting the binary file into a grayscale overlay video through an event camera.
3. The dual-mode guppy gender detection method according to claim 2, characterized in that: The specific steps of obtaining the binary file according to the RGB video include: collecting FPN in the RGB video by an event camera, and generating a binary file of the event stream.
4. The dual-mode guppy gender detection method according to claim 3, characterized in that: If the pixel brightness of the FPN exceeds a set threshold, it is recorded as an event and a binary file of the event stream is generated.
5. The dual-mode guppy gender detection method according to claim 4, characterized in that: The events include on events and off events. When the brightness of the pixel increases, it is recorded as an on event, and when the brightness of the pixel decreases, it is recorded as an off event.
6. The dual-mode guppy gender detection method according to claim 1, characterized in that: The fused highlighted feature tensor includes color information of the RGB image and contour information of the grayscale overlay image.
7. The dual-mode guppy gender detection method according to claim 1, characterized in that: The gender detection model is an improved yolov8 model, and its training steps specifically include: using labelimg to annotate the aligned RGB image, dividing the annotated RGB image and the grayscale overlay image into a training set and a test set, the training set is used to put it into the gender detection model for training, and the test set is used to test whether the gender detection model can detect the gender of guppies, and the accuracy of the gender detection model is evaluated by map50.
8. The dual-mode guppy gender detection method according to claim 7, characterized in that: The specific steps of using the labelimg to label data include: according to the aligned RGB image, the guppies are divided into two types of labels, male fish and female fish, and labeled, wherein the male fish has bright colors and the female fish has a single color.
9. The dual-mode guppy gender detection method according to claim 1, characterized in that: The mask is obtained by convolving a 1×1 convolution kernel with a stride of 1 and no padding, combined with a learnable bias.
10. A dual-mode guppy gender detection system, characterized in that: include: A video acquisition module, used for acquiring an RGB video of the guppy through an RGB camera, and converting the RGB video into a grayscale overlay video through an event camera; An image alignment module is used to extract frames from the RGB video and the grayscale overlay video to obtain an RGB image and a grayscale overlay image of the same time frame, and to perform perspective transformation on the grayscale overlay image to obtain an aligned RGB image and grayscale overlay image; A gender detection module, used for inputting the aligned RGB image and the grayscale superposition image as feature tensors into a gender detection model for information fusion, obtaining a fused feature tensor with highlighted features, and detecting the gender of the guppy according to the fused feature tensor with highlighted features; Among them, the information fusion step includes: splitting the input feature tensor into two parts, left original features and right original features according to the channel dimension, numerically scaling and mask weighting the left original features and the right original features to obtain left mask weighted features and right mask weighted features, adding the left mask weighted features and the left original features, adding the right mask weighted features and the right original features, further extracting features through corresponding bottleneck convolution layers to obtain grayscale superimposed image processing features and RGB image processing features, splicing the grayscale superimposed image processing features and the RGB image processing features on the channel dimension, and adjusting the weights to obtain a feature tensor with highlighted features after fusion.