Intelligent algae identification system and method based on hyperspectral-color bimodal microscopic imaging

Abstract

The invention belongs to the field of water quality monitoring, and discloses an intelligent algae identification system and method based on hyperspectral-color bimodal microscopic imaging, and the system comprises a data collection module, a data analysis module and a cloud database module. The data acquisition module is used for performing hyperspectral and RGB color bimodal microscopic image data acquisition on the algae sample slide to obtain hyperspectral-color bimodal algae microscopic image data; the data analysis module is used for calling the trained hyperspectral-color bimodal fusion algae recognition deep learning model according to the hyperspectral-color bimodal algae microscopic image data to obtain a recognition result of the hyperspectral-color bimodal algae microscopic image data; and the cloud database module is used for providing related information of the identified algae and a historical identification result. According to the invention, a hyperspectral-color bimodal fusion algae identification deep learning model is designed to identify a full-slide algae microscopic image, and a high-precision algae identification result is obtained.

Description

Technical Field

[0001] The present invention relates to the field of water quality monitoring, and in particular to an algae intelligent identification system and method based on hyperspectral-color dual-modal microscopic imaging. Background Art

[0002] Algae play an important role in water bodies. As primary producers in water bodies, they have many species, wide distribution, and are sensitive to water quality changes, which makes them play an important role in water quality monitoring. However, the large-scale reproduction of certain algae in water bodies will cause algal blooms, destroy the quality and structure of water bodies, threaten the survival of aquatic organisms, and also cause harm to human health.

[0003] At present, the identification and counting of algae species using optical microscopes mainly relies on trained and experienced professionals, which is costly and generally time-consuming. This traditional manual microscopic inspection method requires high professional ability of the inspectors, otherwise the identification accuracy is low and it is difficult to achieve large-scale monitoring of water bodies.

[0004] Hyperspectral imaging is different from traditional optical imaging. It contains spatial information and spectral information. It captures the response of samples through incident light of different wavelengths and constructs a three-dimensional data cube. For each pixel in the two-dimensional space, it corresponds to a spectral curve in the third dimension. Therefore, hyperspectral images have rich sample feature information. At present, hyperspectral imaging technology is widely used in agricultural monitoring, mineral exploration, medical diagnosis and other fields.

[0005] With the rapid development of computer science, people have begun to use deep learning methods to classify and identify algae, but most of them are currently based on microscopic RGB color images, while hyperspectral images are mostly used in the field of remote sensing in algae work. One method is to obtain information in the RGB channel at the microscopic level, and the other method is to obtain spectral information at the macroscopic level. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides an intelligent algae identification system and method based on hyperspectral-color dual-modal microscopy imaging, which can obtain more comprehensive algae microscopic image information and save manpower and time costs. At the same time, it can realize the automated dual-modal microscopic image data collection and intelligent algae sample analysis, obtain RGB and hyperspectral dual-modal detection results, and export them in an integrated manner.

[0007] The present invention provides an algae intelligent identification system based on hyperspectral-color dual-modal microscopic imaging, the system comprising: a data acquisition module, a data analysis module and a cloud database module;

[0008] The data acquisition module is used to collect the dual-modal microscopic image data of the algae sample slide, i.e., the hyperspectral and RGB color, to obtain the hyperspectral-color dual-modal microscopic image data of the algae;

[0009] The data analysis module is used to call the trained hyperspectral-color dual-modal fusion algae recognition deep learning model according to the hyperspectral-color dual-modal algae microscopic image data to obtain the recognition result of the hyperspectral-color dual-modal algae microscopic image data;

[0010] The cloud database module is used to provide relevant information of the identified algae and historical identification results.

[0011] Preferably, the hyperspectral-color dual-modal algae microscopic image acquisition device in the data acquisition module includes: an optical microscope, a grayscale industrial camera, an RGB industrial camera, a preview camera, an acousto-optic tunable filter, a driver for the acousto-optic tunable filter, a light source, a three-axis electric stage, and an integrated controller for the three-axis electric stage;

[0012] The optical microscope is used to magnify the algae sample;

[0013] The grayscale industrial camera is used to collect hyperspectral image data;

[0014] The RGB industrial camera is used to collect RGB color image data;

[0015] The preview camera is used to provide a preview image before scanning the image, so as to select a pre-focus point;

[0016] The acousto-optic tunable filter is used to adjust the wavelength range of light;

[0017] The driver of the acousto-optic tunable filter is used to output an electrical signal to adjust the acousto-optic tunable filter;

[0018] The light source includes a reflection light source and a transmission light source, which are respectively located at different positions of the algae slide sample, so as to obtain different spectral information;

[0019] The three-axis electric stage is used to move the position so as to collect a clear image of the corresponding position;

[0020] The integrated controller of the three-axis electric stage is used to control the three-axis electric stage and the light source through the computer of the control device, wherein the computer of the control device is used to set the operating parameters of the hyperspectral-color dual-modal algae microscopic image acquisition device and stitch the acquired images; the operating parameters include the number of bands, wavelength range, and magnification of the hyperspectral microscopic image.

[0021] Preferably, the data analysis module includes: an acquisition module, a call module and a display module;

[0022] The acquisition module is used to acquire detection information;

[0023] The calling module is used to call the hyperspectral-color dual-modal fusion algae recognition deep learning model, and obtain the detection result according to the detection information;

[0024] The display module is used to display the detection results.

[0025] Preferably, the detection information includes: counting method, sampling volume and fixed volume, and hyperspectral-color dual-modal algae microscopic image data path;

[0026] The counting methods include full-sheet counting method, row-grid counting method, diagonal counting method, and random field of view method;

[0027] The sampling volume and the fixed volume are the sampling capacity and the fixed volume capacity of the algae sample during the processing, which are used for the subsequent calculation of the algae density;

[0028] The dual-modal algae microscopic image data path provides a storage location for the hyperspectral-color dual-modal algae microscopic image data collected by the data collection module.

[0029] Preferably, the hyperspectral-color dual-modal fusion algae recognition deep learning model is an improved dual-modal fusion algae recognition model based on YOLOLv8, including a dual-modal feature extraction and fusion module, a multi-scale feature enhancement module, a multi-scale feature fusion module, and a detection result output module;

[0030] The dual-modal feature extraction and fusion module is used to extract and fuse features of RGB algae microscopic images and hyperspectral algae microscopic images;

[0031] The multi-scale feature enhancement module is used to fuse feature information of different sizes by means of spatial pyramid pooling;

[0032] The multi-scale feature fusion module is used to fuse features extracted from different levels;

[0033] The detection result output module is used to generate the target category, bounding box position and confidence score based on the features output by each module, and complete the final detection and classification tasks through the detection head and classification head; it is also used to receive feature maps of multiple scales, and predict the feature maps of each scale, and filter the output results through the post-processing operation of non-maximum suppression.

[0034] Preferably, the detection results include: data reports, charts, RGB algae microscopic image recognition results, hyperspectral algae microscopic image recognition results, and algae hyperspectral curve results;

[0035] The data report includes the types and numbers of algae detected, and the calculated density, dominance, biomass, diversity index, and richness index of the algae detected;

[0036] The charts include a bar chart and a pie chart result display of the detected algae;

[0037] The RGB algae microscopic image recognition result marks the location and type of the detected algae on the basis of the original image, and supports manual correction of the detection result;

[0038] The hyperspectral algae microscopic image recognition result is to mark the location and type of the detected algae in the false color image generated by the combined bands, and support manual correction of the detection result;

[0039] The algae hyperspectral curve result shows the spectral curve of any pixel in the algae microscopic image, which is convenient for observing its spectral characteristics. At the same time, the algae sample area of ​​interest is selected, and its spectral curve is extracted and displayed. By analyzing the spectral curve, the characteristic bands are displayed and identified, and the statistical information of the spectral curve is calculated, including the mean, standard deviation, maximum value, and minimum value, the spectral characteristic distribution of a specific area or pixel can be understood.

[0040] Preferably, the cloud database module stores relevant information of algae, including phylum, class, order, family, genus, species, Latin name, and historical detection results; after the back end calls the hyperspectral-color dual-modal fusion algae recognition deep learning model to obtain the detection result, it obtains the phylum, class, order, family, genus, species, and Latin name of the detected algae from the cloud database module, and then passes it to the front end for result display. At the same time, the cloud database module stores the historical detection results, supports the back end to query the cloud database module, and then allows the front end to display the historical results.

[0041] The present invention also provides an algae intelligent identification method based on hyperspectral-color dual-modal microscopic imaging, which is implemented by using any of the algae intelligent identification systems based on hyperspectral-color dual-modal microscopic imaging, and the method comprises:

[0042] Building a hyperspectral-color dual-modal algae microscopic image acquisition device in a data acquisition module, and acquiring hyperspectral-color dual-modal algae microscopic image data through the hyperspectral-color dual-modal algae microscopic image acquisition device;

[0043] The hyperspectral-color dual-modal algae microscopic image data is input into a data analysis module, and a trained hyperspectral-color dual-modal fusion algae recognition deep learning model is called to obtain a recognition result of the hyperspectral-color dual-modal algae microscopic image data;

[0044] The cloud database module stores the information of phylum, class, order, family, genus, species, and Latin name of algae, as well as saves historical test results.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention can collect dual-modal algae microscopic images of hyperspectral microscopic images and RGB color microscopic images, obtain richer information, provide a hyperspectral-color dual-modal fusion algae recognition deep learning model that can integrate dual-modal features, more accurately identify and classify algae in samples, and provide different counting methods. It displays dual-modal detection result information and provides detection result graphs corresponding to the two modes. It can view spectral curve result information, select different result statistical forms for export, and support secondary averaging function. The cloud database provided stores algae-related information and historical detection results for query and display. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 This is a workflow diagram of an algae intelligent identification system based on hyperspectral-color dual-modal microscopic imaging according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the operation flow of an algae intelligent identification system based on hyperspectral-color dual-modal microscopic imaging according to an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the working process of a data analysis module according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of a hyperspectral-color dual-modality fusion algae recognition deep learning model according to an embodiment of the present invention;

[0052] Figure 5 Schematic diagram of a dual-modal feature extraction and fusion unit in a hyperspectral-color dual-modal fusion algae recognition deep learning model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Embodiment 1

[0057] The present invention provides an algae intelligent identification system based on hyperspectral-color dual-modal microscopic imaging, the system comprising: a data acquisition module, a data analysis module and a cloud database module;

[0058] The data acquisition module is used to collect the dual-modal microscopic image data of the algae sample slide, i.e., the hyperspectral and RGB color, to obtain the hyperspectral-color dual-modal microscopic image data of the algae;

[0059] The data analysis module is used to call the trained hyperspectral-color dual-modal fusion algae recognition deep learning model according to the hyperspectral-color dual-modal algae microscopic image data to obtain the recognition result of the hyperspectral-color dual-modal algae microscopic image data;

[0060] The cloud database module is used to provide relevant information of the identified algae and historical identification results.

[0061] In this embodiment, the hyperspectral-color dual-modal algae microscopic image acquisition device in the data acquisition module includes: an optical microscope, a grayscale industrial camera, an RGB industrial camera, a preview camera, an acousto-optic tunable filter, a driver of the acousto-optic tunable filter, a light source, a three-axis electric stage, and an integrated controller of the three-axis electric stage;

[0062] An optical microscope can magnify algae samples;

[0063] Grayscale industrial cameras can be used to collect hyperspectral image data;

[0064] RGB industrial cameras can be used to collect RGB color image data;

[0065] The preview camera can be used to provide a preview image before scanning the image, so as to select the pre-focus point;

[0066] Acousto-optic tunable filters can be used to tune the wavelength range of light;

[0067] The driver of the acousto-optic tunable filter can be used to output an electrical signal to control the acousto-optic tunable filter;

[0068] The light source includes a reflected light source and a transmitted light source, which are located at different positions of the algae slide sample, so that different spectral information can be obtained;

[0069] The three-axis motorized stage can be moved to facilitate the acquisition of clear images at the corresponding position;

[0070] The integrated controller of the three-axis electric stage can control the three-axis electric stage and the light source through the computer of the control device.

[0071] The dual-modal algae microscopic image acquisition device can obtain hyperspectral microscopic images and RGB color images of algae slide samples. This dual-modal data provides a data set and sufficient feature information for the deep learning model.

[0072] In this embodiment, the data analysis module includes: an acquisition module, a call module and a display module;

[0073] The acquisition module is used to obtain detection information;

[0074] The calling module is used to call the hyperspectral-color dual-modal fusion algae recognition deep learning model, and obtain the detection result according to the detection information; the hyperspectral-color dual-modal fusion algae recognition deep learning model fuses the dual-modal information of the RGB algae microscopic image data and the hyperspectral algae microscopic image data, providing a richer amount of information for the deep learning network, thereby learning more features and providing sufficient data basis for algae detection;

[0075] The display module is used to display the test results.

[0076] The detection information includes: counting method, sampling volume and fixed volume, and hyperspectral-color dual-modal algae microscopic image data path;

[0077] The counting methods include full-sheet counting, row counting, diagonal counting, and random field of view counting, which can be selected according to actual conditions and needs;

[0078] The sampling volume and the fixed volume are the sampling capacity and the fixed volume capacity of the algae sample during the processing, which are used for the subsequent calculation of the algae density;

[0079] The hyperspectral-color dual-modal algae microscopic image data path provides a storage location for the dual-modal algae microscopic image data collected by the data collection module;

[0080] Among them, the hyperspectral-color dual-modal fusion algae recognition deep learning model is an improved dual-modal fusion algae recognition model based on YOLOLv8, including dual-modal feature extraction and fusion module (DFEFM), convolution module, splicing module, multi-scale feature enhancement module (MFEM), multi-scale feature fusion module (MFFM), and detection result output module (DROM);

[0081] The dual-modal feature extraction and fusion module is used to extract and fuse features of RGB algae microscopic images and hyperspectral algae microscopic images;

[0082] The multi-scale feature enhancement module is used to fuse feature information of different sizes through spatial pyramid pooling;

[0083] The multi-scale feature fusion module is used to fuse features extracted from different levels;

[0084] The detection result output module is used to generate the target category, bounding box position and confidence score based on the features output by the previous module, and complete the final detection and classification tasks through the detection head and classification head; it can also receive feature maps of multiple scales and make predictions for the feature maps of each scale. This multi-scale prediction mechanism can better adapt to targets of different sizes, so that the model has higher accuracy when detecting small and large targets, and through the post-processing operation of non-maximum suppression, the output results are screened to remove redundant boxes with high overlap and only retain prediction boxes with higher confidence, mainly based on the intersection-union ratio and confidence score.

[0085] Specifically, the hyperspectral-color dual-modal fusion algae recognition deep learning model needs to input the RGB algae microscopic image and the hyperspectral algae microscopic image, and input the RGB algae microscopic image and the hyperspectral algae microscopic image into the dual-modal feature extraction and fusion module;

[0086] The dual-modal feature extraction and fusion module consists of four cascaded dual-modal feature extraction and fusion units. The dual-modal feature extraction and fusion unit can be used with DFEFU express;

[0087] The outputs of the subsequent three cascaded bimodal feature extraction and fusion units are respectively input into the convolution module, and the three outputs are DO1, DO2, and DO3, respectively, and the formula is as follows;

[0088] DO1=DFEFU(DFEFU(RGB,HSI))

[0089] DO2=DFEFU(DFEFU(DFEFU(RGB,HSI)))

[0090] DO3=DFEFU(DFEFU(DFEFU(DFEFU(RGB,HSI))))

[0091] The dual-modal feature extraction and fusion unit mainly passes through the convolution module (CONV) and the feature extraction module (FEM). After passing through the next convolution module, the features on the two branches are input into the concatenation module (CCM) for jump concatenation, and the RGB algae microscopic image features and the hyperspectral algae microscopic image features represented by the two branches are fused. The formula is as follows:

[0092] (OUT1,OUT2)=(CCM(CONV(FEM(CONV(IN1)))),

[0093] CONV(FEM(CONV(IN2)))),CCM(CONV(FEM(CONV(IN2))),

[0094] CONV(FEM(CONV(IN1)))))

[0095] In the dual-modal feature extraction and fusion module, the outputs of the dual-modal feature extraction and fusion units of the last three layers are input into the multi-scale fusion module after passing through the convolution module and the splicing module. The outputs of the dual-modal feature extraction and fusion units of the last layer are input into the multi-scale feature enhancement module after passing through the convolution module and the splicing module. The outputs are then input into the multi-scale fusion module and finally into the detection result output module. The formula is as follows;

[0096] OUT3=MFFM(CCM(CONV(DO1)),CCM(CONV(DO2)),

[0097] MFEM(CCM(CONV(DO3))))

[0098] The dual-modal feature extraction and fusion module extracts and fuses features from the RGB algae microscopic image and the hyperspectral algae microscopic image. The feature extraction module optimizes the transmission and calculation of features, provides a lightweight but efficient way to enhance the feature extraction capability of the model, reduces redundant calculations, and strengthens the gradient flow, thereby improving the calculation efficiency while maintaining good detection accuracy, and providing a more efficient feature extraction and information transmission mechanism; the splicing module is a key step in fusing the two modal information, so that the two branches have dual-modal feature maps at the same time;

[0099] The multi-scale feature enhancement module can significantly improve the model's ability to detect multi-scale objects through spatial pyramid pooling, adapt to the detection of algae of different sizes, and fuse information of different sizes through multi-scale pooling, enhancing the model's understanding of global and local features while maintaining efficient computing speed. In practical applications, it can achieve a better balance between accuracy and speed.

[0100] The multi-scale feature fusion module is mainly used for feature fusion, optimization and multi-scale processing. It is responsible for further processing the features, enhancing the multi-scale information and improving the overall detection performance of the model. Its goal is to effectively fuse the features extracted from different levels to better support the algae detection task.

[0101] The detection result output module includes a detection head and a classification head. Therefore, its loss function has two branches: classification and regression. The loss function of the classification branch is L VEL , the formula is as follows

[0102]

[0103] q is the intersection-over-union ratio of the predicted box and the true box, and p is the score;

[0104] The loss function of the regression branch is L DFL and L CIOU The composition formula is as follows

[0105] L DFL =-((y i+1 -y)log(S i )+(yy i )log(S i+1 ))

[0106]

[0107] S i is the predicted value output by the network, S i+1 is the immediate prediction value output by the network, y, y i ,y i+1is the actual value of the label, the integral value of the label, and the integral value of the adjacent labels;

[0108] v is a parameter to measure the consistency of aspect ratio, w g is the width of the real frame, h g is the height of the real frame, w p is the width of the prediction box, h p is the height of the prediction box;

[0109] The detection results include: the detected algae species and number obtained by calling the hyperspectral-color dual-modal fusion algae recognition deep learning model, the detected algae density, dominance, biomass, diversity index, and richness index obtained by using the sampling volume and fixed volume in the monitoring information and calculating through the back-end program;

[0110] After the front-end program obtains the detection results, it displays the results in the form of data reports and charts. At the same time, it supports exporting reports and performing secondary averaging of the results of two tests to make the results more convincing. At the same time, it can display image results, including RGB algae microscopic image recognition results, hyperspectral algae microscopic image recognition results, and algae hyperspectral curve results. The RGB algae microscopic image recognition results can mark the position and type of the detected algae on the basis of the original image, and support manual correction of the detection results, such as correcting the size and position of the mark box and correcting the detection type. The hyperspectral algae microscopic image recognition results are The position and type of the detected algae are marked in the false color image generated by the combined bands, and manual correction of the detection results is supported, such as correcting the size and position of the marking box and correcting the detected type; the algae hyperspectral curve result can display the spectral curve of any pixel in the algae microscopic image, which is convenient for observing its spectral characteristics. At the same time, it also supports the selection of algae sample areas of interest, extraction and display of their spectral curves, and calculation of statistical information of the spectral curves, including mean, standard deviation, maximum and minimum values, by analyzing the spectral curves, displaying and identifying characteristic bands, and calculating the statistical information of the spectral curves, so as to understand the spectral characteristic distribution of specific areas or pixels, and the result information data that needs to be retained can be selected for export.

[0111] In this embodiment, the cloud database module stores relevant information of algae, including phylum, class, order, family, genus, species, Latin name, and historical detection results; after the back end calls the hyperspectral-color dual-modal fusion algae recognition deep learning model to obtain the detection result, it obtains the phylum, class, order, family, genus, species, and Latin name of the detected algae from the cloud database module, and then passes it to the front end for result display. At the same time, the cloud database module stores the historical detection results, supports the back end to query the cloud database module, and then allows the front end to display the historical results.

[0112] Embodiment 2

[0113] like Figure 1 As shown, this embodiment proposes a workflow of an algae intelligent identification system based on hyperspectral-color dual-modal microscopic imaging, which mainly includes:

[0114] The dual-modal algae microscopic image data are acquired using a data acquisition module;

[0115] The dual-modal algae microscopic image acquisition device in the data acquisition module includes an optical microscope, a grayscale industrial camera, an RGB industrial camera, a preview camera, an acousto-optic tunable filter and the driver, a light source, a three-axis electric stage and the integrated controller;

[0116] The computer of the control device is mainly used to set some operating parameters of the dual-modal algae microscopic image acquisition device and to stitch the acquired images;

[0117] The operating parameters include the number of bands, wavelength range, magnification, etc. of the hyperspectral microscopic image;

[0118] The dual-modal algae microscopic image data includes RGB algae microscopic image data and hyperspectral algae microscopic image data;

[0119] Inputting the dual-modal algae microscopic image data into a data analysis module;

[0120] Acquiring detection information, the detection information including counting method, sampling volume and fixed volume, and the dual-modal algae microscopic image data path;

[0121] The counting methods include full-sheet counting method, row counting method, diagonal counting method, and random field of view method, which can be selected according to actual conditions and needs;

[0122] The sampling volume and the fixed volume are the sampling capacity and the fixed volume capacity of the algae sample during the processing, which are used for the subsequent calculation of the algae density;

[0123] The hyperspectral-color dual-modal algae microscopic image data path provides a storage location for the hyperspectral-color dual-modal algae microscopic image data collected by the data collection module;

[0124] Call the hyperspectral-color dual-modal fusion algae recognition deep learning model to obtain the detection results;

[0125] The detection results include the type and number of the detected algae, the calculated density, dominance, biomass, diversity index, and richness index of the detected algae;

[0126] Display results, including data reports, charts, RGB algae microscopic image recognition results, hyperspectral algae microscopic image recognition results, and algae hyperspectral curve results;

[0127] The data report includes the type and number of the detected algae, and calculates the density, dominance, biomass, diversity index, and richness index of the detected algae;

[0128] The chart includes a bar chart and a pie chart showing the detected algae, which is clear and intuitive;

[0129] The RGB algae microscopic image recognition result can mark the position and type of the detected algae on the basis of the original image, and support manual correction of the detection result, such as correcting the size and position of the mark box and correcting the detection type;

[0130] The hyperspectral algae microscopic image recognition result is to mark the position and type of the detected algae in the false color image generated by the combined band, and also supports manual correction of the detection result, such as correcting the size and position of the marking box and correcting the detection type;

[0131] The algae hyperspectral curve result can display the spectral curve of any pixel in the algae microscopic image, which is convenient for observing its spectral characteristics. At the same time, the algae sample area of ​​interest can be selected, and its spectral curve can be extracted and displayed. By analyzing the spectral curve, the characteristic bands can be displayed and identified, and the statistical information of the spectral curve can be calculated, including the mean, standard deviation, maximum value, and minimum value, so as to understand the spectral characteristic distribution of a specific area or pixel, and the result information data that needs to be retained can be selected for export;

[0132] The cloud database module stores relevant information about algae, including phylum, class, order, family, genus, species, Latin name, and historical test results;

[0133] After the back-end program calls the hyperspectral-color dual-modal fusion algae recognition deep learning model to obtain the detection results, the phylum, class, order, family, genus, species, and Latin name of the detected algae can be obtained from the cloud database module, and then passed to the front-end program for result display. At the same time, the cloud database module can store historical detection results, support the back-end program to query the cloud database module, and then let the front-end program display historical results;

[0134] like Figure 2 As shown, this embodiment provides an algae intelligent identification system based on hyperspectral-color dual-modal microscopic imaging, and its operation process includes:

[0135] Step 201, making a slide sample, making the processed algae sample into an algae slide to be tested according to standard steps.

[0136] Step 202, placing the algae glass slide to be detected on the stage and adjusting the position to ensure that the hyperspectral-color dual-modal algae microscopic imaging acquisition device can accurately acquire and scan.

[0137] Step 203 , the acquisition band and the number of bands of the hyperspectral algae microscopic image data can be set by setting software in the computer of the control device.

[0138] Step 204, after setting various acquisition parameters, the dual-modal algae microscopic image acquisition device starts data acquisition, wherein the grayscale industrial camera and the RGB industrial camera are controlled by the computer of the control device to respectively acquire grayscale algae image data and color algae image data, wherein the grayscale algae image data is used to constitute the hyperspectral algae microscopic image data.

[0139] Step 205 , after completing the dual-modal algae microscopic image data acquisition, the computer of the control device performs a splicing operation on the obtained dual-modal algae microscopic image data.

[0140] Step 206, after the stitching operation is completed, the hyperspectral-color dual-modal algae microscopic image data can be obtained from the image data and the band information data, including a hyperspectral algae microscopic image and an RGB color algae microscopic image.

[0141] Step 207 , in the analysis software in the computer of the control device, select the counting method to be adopted and the algae image to be detected, and start the analysis.

[0142] During the analysis process, the analysis software in the computer of the control device will call the trained hyperspectral-color dual-modal fusion algae recognition deep learning model, and pass the processed algae image to be detected to the trained deep learning model. After the model is run, the analysis software program in the computer of the control device will process and calculate other data based on the model operation results, and the other data include but are not limited to the phylum, class, order, family, genus, species, Latin name, density, dominance, biomass, diversity index, and richness index of the algae.

[0143] Step 208, after the hyperspectral-color dual-modal fusion algae recognition deep learning model is called and the other data are processed and calculated, the results will be displayed on the interface through the analysis software in the computer of the control device, and the reports, statistical charts and spectral analysis results in the required format can be downloaded. The two analysis result reports can be averaged twice as needed.

[0144] like Figure 3As shown, this embodiment provides a system workflow for analyzing and displaying hyperspectral-color dual-mode algae microscopic image data, including:

[0145] Step 301, after selecting the counting method and the hyperspectral-color dual-modal algae microscopic image data path, start the analysis.

[0146] Step 302, taking the counting method and the hyperspectral-color dual-modal algae microscopic image data path as parameters, calling the hyperspectral-color dual-modal fusion algae recognition deep learning model, after the model runs, the recognized visualization results will be stored in another corresponding position in the computer of the control device, and at the same time, the recognized algae species and number are also obtained.

[0147] Step 303: perform subsequent processing on the obtained hyperspectral-color dual-modal detection result data.

[0148] Step 3031, according to the selected counting method, call the formula conversion module to perform formula conversion on the number of algae. If the selected counting method is not the whole slide method, it is necessary to multiply the counted number of algae by the corresponding multiple according to the proportion to obtain the estimated number of algae in the whole slide. If the selected counting method is the whole slide method, there is no need to perform the formula conversion.

[0149] Step 3032, calling the density calculation function module, and performing density calculation on the number of algae in the whole glass slide obtained by conversion according to the formula. At the same time, the density calculation function module requires the sampling volume and the fixed volume as parameters, and calculates the initial density according to whether the sample is diluted or concentrated during the previous processing.

[0150] Step 3033, calling the dominance calculation function module to calculate the dominance, which can be calculated based on the number of algae.

[0151] Step 3034, calling the index calculation function module to perform index calculation, including diversity index, richness index, etc. Each index has a corresponding formula, and the data obtained in the early stage can be substituted into the formula for calculation.

[0152] Step 3035, reading the hyperspectral algae microscopic image data, which includes the spectral value of each pixel in different bands, so as to display the hyperspectral algae microscopic image recognition result.

[0153] Step 3036, reading the pixel spectral values ​​of the hyperspectral algae microscopic image, and extracting the spectral values ​​of these pixels in each band.

[0154] Step 3037, the spectral value of the selected pixel in each band is matched with the wavelength of the band, and a spectral curve is drawn. This curve reflects the spectral response characteristics of the pixel in different bands and can reveal the spectral characteristics of the algae to be detected.

[0155] Step 3038, calculate the statistical information of the spectral curve, including the mean, standard deviation, maximum value, and minimum value, so as to understand the spectral feature distribution of a specific area or pixel.

[0156] Step 3039, identifying characteristic bands of the spectral curve. If absorption peaks or reflection peaks appear in the spectral curve, these bands can be identified through sharp changes in the curve.

[0157] Step 304: display the visualization results of the recognition and the results of the calculation.

[0158] Step 3041, calling the species summary module to summarize and display the identified algae species.

[0159] Step 3042, calling a chart display function module, using the algae density as a parameter, and presenting the identified number of algae in the form of a bar chart and a pie chart.

[0160] Step 3043, calling the report display module to display the algae density, index, etc. obtained by the previous calculation in the form of a report. At the same time, you can choose to call the report export module to save the report information locally, and you can choose the content and format of the required report.

[0161] Step 3044, the RGB algae microscopic image recognition result can mark the position and type of the detected algae on the basis of the original image, and support manual correction of the detection result, such as correcting the size and position of the mark box and correcting the detection type;

[0162] Step 3045, generating a false color image by combining the bands;

[0163] Step 3046, the hyperspectral algae microscopic image recognition result is to mark the position and type of the detected algae in the false color image generated by the combined band, and then display it, and at the same time support manual correction of the detection result, such as correcting the size and position of the mark box, and correcting the detection type.

[0164] Step 3047, displaying the spectrum curve and analysis results obtained in step 303.

[0165] Step 305, performing a second average on the reports derived from the two tests.

[0166] Step 3051, uploading the two test results that need to be averaged twice.

[0167] Step 3052, calling the secondary averaging function module, performing an averaging operation on the files that need to be secondary averaged, and displaying the averaged results. At the same time, it also supports the option of calling the report export module to save the report information after the secondary averaging locally.

[0168] like Figure 4 As shown, this embodiment provides a hyperspectral-color dual-modal fusion algae recognition deep learning model.

[0169] The hyperspectral-color dual-modal fusion algae recognition deep learning model fuses the dual-modal information of the RGB algae microscopic image data and the hyperspectral algae microscopic image data, providing a richer amount of information for the deep learning network, thereby learning more features and providing sufficient data basis for algae detection;

[0170] The hyperspectral-color dual-modal fusion algae recognition deep learning model is an improved dual-modal fusion algae recognition algorithm based on YOLOLv8, which mainly includes a dual-modal feature extraction and fusion module, a multi-scale feature enhancement module, a multi-scale feature fusion module, and a detection result output module;

[0171] The hyperspectral-color dual-modal fusion algae recognition deep learning model needs to input the RGB algae microscopic image and the hyperspectral algae microscopic image, and input the RGB algae microscopic image and the hyperspectral algae microscopic image into the dual-modal feature extraction and fusion module;

[0172] The bimodal feature extraction and fusion module is composed of bimodal feature extraction and fusion units;

[0173] like Figure 5 As shown, this embodiment provides a schematic diagram of a dual-modal feature extraction and fusion unit in a hyperspectral-color dual-modal fusion algae recognition deep learning model.

[0174] The dual-modal feature extraction and fusion unit mainly passes through a convolution module and a feature extraction module. After passing through the convolution module, the features on the two branches are jump-joined, and the two branches represent the RGB algae microscopic image features and the hyperspectral algae microscopic image features respectively;

[0175] In the bimodal feature extraction and fusion module, the outputs of the bimodal feature extraction and fusion units of different layers are input into the multi-scale fusion module after passing through the convolution module and the splicing module, and the outputs of the bimodal feature extraction and fusion units of the last layer are input into the multi-scale feature enhancement module after passing through the convolution module and the splicing module, and then the outputs are input into the multi-scale fusion module, and finally into the detection result output module;

[0176] The dual-modal feature extraction and fusion module extracts and fuses features from the RGB algae microscopic image and the hyperspectral algae microscopic image. The feature extraction provides a lightweight but efficient way to enhance the feature extraction capability of the model by optimizing the transmission and calculation of features, reduces redundant calculations, and strengthens the gradient flow, thereby improving the calculation efficiency while maintaining good detection accuracy, and providing a more efficient feature extraction and information transmission mechanism; the feature fusion is mainly completed by each convolution module and splicing module in the feature extraction process, and the splicing module is a key step in fusing the two modal information, so that the two branches have dual-modal feature maps at the same time;

[0177] The multi-scale feature enhancement module can significantly improve the model's ability to detect multi-scale objects through spatial pyramid pooling, adapt to algae detection of different sizes, and fuse information of different sizes through multi-scale pooling, thereby enhancing the model's understanding of global and local features while maintaining efficient computing speed, so as to achieve a better balance between accuracy and speed in practical applications;

[0178] The multi-scale feature fusion module is mainly used for feature fusion, optimization and multi-scale processing. It is responsible for further processing the features, enhancing the multi-scale information and improving the overall detection performance of the model. Its goal is to effectively fuse the features extracted from different levels to better support the algae detection task.

[0179] The detection result output module can generate the target category, bounding box position and confidence score based on the features output by the previous module, and complete the final detection and classification tasks through the detection head and classification head; it can also receive feature maps of multiple scales and make predictions for the feature maps of each scale. This multi-scale prediction mechanism can better adapt to targets of different sizes, so that the model has higher accuracy when detecting small and large targets, and through the post-processing operation of non-maximum suppression, the output results are screened to remove redundant frames with high overlap, and only retain prediction frames with higher confidence, mainly based on the intersection-union ratio and the confidence score.

[0180] The present embodiment provides an intelligent recognition system for the hyperspectral-color dual-modal microscopic imaging of algae slide samples. The slide is scanned by a hyperspectral-color dual-modal microscopic imaging system in a data acquisition module that can control a grayscale camera and an RGB color camera to obtain a hyperspectral-color dual-modal algae microscopic image. The data is analyzed and the results are displayed by calling a hyperspectral-color dual-modal fusion algae recognition deep learning model through communication between the front-end and back-end programs in the data analysis module. The hyperspectral-color dual-modal fusion algae recognition deep learning model extracts features and performs jump splicing. The feature fusion integrates the information of both hyperspectral and color modes, and has a higher recognition accuracy. Through the recognition of the model and the calculation of the back-end program, the algae species, number, density, dominance, biomass, diversity index, and richness index can be obtained. The displayed results include data reports, charts, RGB algae microscopic image recognition results, hyperspectral algae microscopic image recognition results, and algae hyperspectral curve results. At the same time, the cloud database module can provide the algae’s phylum, class, order, family, genus, species, and Latin name information for the data reports in the result display, and can record historical detection results.

[0181] The technical solution of the present invention,

[0182] The present invention provides an algae intelligent identification system based on hyperspectral-color dual-modal microscopic imaging, which is used for full-slide collection of microscopic hyperspectral images and RGB color images of algae in water bodies, and identification statistics and determination of related indexes of algae in samples. The present invention collects hyperspectral and RGB color full-slide images of algae slide samples made in a standardized manner through a dual-modal algae microscopic imaging acquisition device in a data acquisition module. The hyperspectral-color dual-modal algae microscopic image data is uploaded to the data analysis module, and a hyperspectral-color dual-modal fusion algae identification deep learning model is called to obtain dual-modal detection results. The hyperspectral-color dual-modal fusion algae identification deep learning model fuses the dual-modal information of RGB algae microscopic image data and hyperspectral algae microscopic image data, providing a richer amount of information for the deep learning network, thereby learning more features. Then, the image recognition results and data statistics results are displayed through the back-end program calculation processing and the front-end program operation. The cloud database module provides the data analysis module with relevant information about the detected algae. At the same time, the data analysis module saves historical detection results in the cloud database module and supports query. The present invention provides a systematic method from algae data collection to detection result display, designs a hyperspectral-color dual-modal fusion algae recognition deep learning model to identify whole-slide algae microscopic images, obtains high-precision algae recognition results, and displays the detection results in various forms.

[0183] Embodiment 3

[0184] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present invention discloses an algae intelligent identification method based on hyperspectral-color dual-modal microscopy imaging, which is implemented by using any of the algae intelligent identification systems based on hyperspectral-color dual-modal microscopy imaging, and the method includes:

[0185] Building a hyperspectral-color dual-modal algae microscopic image acquisition device in a data acquisition module, and acquiring hyperspectral-color dual-modal algae microscopic image data through the hyperspectral-color dual-modal algae microscopic image acquisition device;

[0186] The hyperspectral-color dual-modal algae microscopic image data is input into a data analysis module, and a trained hyperspectral-color dual-modal fusion algae recognition deep learning model is called to obtain a recognition result of the hyperspectral-color dual-modal algae microscopic image data;

[0187] The cloud database module stores the information of phylum, class, order, family, genus, species, and Latin name of algae, as well as saves historical test results.

[0188] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An intelligent algae identification system based on hyperspectral-color dual-modal microscopic imaging, characterized in that: The system includes: a data acquisition module, a data analysis module and a cloud database module; The data acquisition module is used to collect hyperspectral and RGB color dual-modal microscopic image data of the algae sample slide to obtain hyperspectral-color dual-modal algae microscopic image data; The data analysis module is used to call the trained hyperspectral-color dual-modal fusion algae recognition deep learning model according to the hyperspectral-color dual-modal algae microscopic image data to obtain the recognition result of the hyperspectral-color dual-modal algae microscopic image data; The cloud database module is used to provide relevant information of the identified algae and historical identification results.

2. The system according to claim 1, characterized in that The hyperspectral-color dual-modal algae microscopic image acquisition device in the data acquisition module includes: an optical microscope, a grayscale industrial camera, an RGB industrial camera, a preview camera, an acousto-optic tunable filter, a driver of the acousto-optic tunable filter, a light source, a three-axis electric stage, and an integrated controller of the three-axis electric stage; The optical microscope is used to magnify the algae sample; The grayscale industrial camera is used to collect hyperspectral image data; The RGB industrial camera is used to collect RGB color image data; The preview camera is used to provide a preview image before scanning the image, so as to select a pre-focus point; The acousto-optic tunable filter is used to adjust the wavelength range of light; The driver of the acousto-optic tunable filter is used to output an electrical signal to adjust the acousto-optic tunable filter; The light source includes a reflection light source and a transmission light source, which are respectively located at different positions of the algae slide sample, so as to obtain different spectral information; The three-axis electric stage is used to move the position so as to collect a clear image of the corresponding position; The integrated controller of the three-axis electric stage is used to control the three-axis electric stage and the light source through the computer of the control device, wherein the computer of the control device is used to set the operating parameters of the hyperspectral-color dual-modal algae microscopic image acquisition device and stitch the acquired images; the operating parameters include the number of bands, wavelength range, and magnification of the hyperspectral microscopic image.

3. The system according to claim 2, characterized in that The data analysis module includes: an acquisition module, a call module and a display module; The acquisition module is used to acquire detection information; The calling module is used to call the hyperspectral-color dual-modal fusion algae recognition deep learning model, and obtain the detection result according to the detection information; The display module is used to display the detection results.

4. The system according to claim 3, characterized in that The detection information includes: counting method, sampling volume and fixed volume, and hyperspectral-color dual-modal algae microscopic image data path; The counting methods include full-sheet counting method, row-grid counting method, diagonal counting method, and random field of view method; The sampling volume and the fixed volume are the sampling capacity and the fixed volume capacity of the algae sample during the processing, which are used for the subsequent calculation of the algae density; The dual-modal algae microscopic image data path provides a storage location for the hyperspectral-color dual-modal algae microscopic image data collected by the data collection module.

5. The system according to claim 3, characterized in that The hyperspectral-color dual-modal fusion algae recognition deep learning model is an improved dual-modal fusion algae recognition model based on YOLOLv8, including a dual-modal feature extraction and fusion module, a multi-scale feature enhancement module, a multi-scale feature fusion module, and a detection result output module; The dual-modal feature extraction and fusion module is used to extract and fuse features of RGB algae microscopic images and hyperspectral algae microscopic images; The multi-scale feature enhancement module is used to fuse feature information of different sizes by means of spatial pyramid pooling; The multi-scale feature fusion module is used to fuse features extracted from different levels; The detection result output module is used to generate the category, bounding box position and confidence score of the target based on the features output by each module, and complete the final detection and classification tasks through the detection head and classification head; It is also used to receive feature maps of multiple scales, make predictions for the feature maps of each scale, and filter the output results through the post-processing operation of non-maximum suppression.

6. The system according to claim 3, characterized in that The detection results include: data reports, charts, RGB algae microscopic image recognition results, hyperspectral algae microscopic image recognition results, and algae hyperspectral curve results; The data report includes the types and numbers of algae detected, the density, dominance, biomass, diversity index, and richness index of the algae detected; The charts include a bar chart and a pie chart result display of the detected algae; The RGB algae microscopic image recognition result marks the location and type of the detected algae on the basis of the original image, and supports manual correction of the detection result; The hyperspectral algae microscopic image recognition result is to mark the location and type of the detected algae in the false color image generated by the combined bands, and support manual correction of the detection result; The algae hyperspectral curve result shows the spectral curve of any pixel in the algae microscopic image, which is convenient for observing its spectral characteristics. At the same time, the algae sample area of ​​interest is selected, and its spectral curve is extracted and displayed. By analyzing the spectral curve, the characteristic bands are displayed and identified, and the statistical information of the spectral curve is calculated, including the mean, standard deviation, maximum value, and minimum value, the spectral characteristic distribution of a specific area or pixel can be understood.

7. The system according to claim 1, characterized in that The cloud database module stores relevant information of algae, including phylum, class, order, family, genus, species, Latin name, and historical detection results; after the back end calls the hyperspectral-color dual-modal fusion algae recognition deep learning model to obtain the detection result, it obtains the phylum, class, order, family, genus, species, and Latin name of the detected algae from the cloud database module, and then passes it to the front end for result display. At the same time, the cloud database module stores the historical detection results, supports the back end to query the cloud database module, and then allows the front end to display the historical results.

8. A method for intelligent identification of algae based on hyperspectral-color dual-modal microscopy imaging, which is implemented by using the intelligent identification system of algae based on hyperspectral-color dual-modal microscopy imaging according to any one of claims 1 to 7, characterized in that: The method comprises: Building a hyperspectral-color dual-modal algae microscopic image acquisition device in a data acquisition module, and acquiring hyperspectral-color dual-modal algae microscopic image data through the hyperspectral-color dual-modal algae microscopic image acquisition device; The hyperspectral-color dual-modal algae microscopic image data is input into a data analysis module, and a trained hyperspectral-color dual-modal fusion algae recognition deep learning model is called to obtain a recognition result of the hyperspectral-color dual-modal algae microscopic image data; The cloud database module stores the information of phylum, class, order, family, genus, species, and Latin name of algae, as well as saves historical test results.

Images