A real-time online probe imaging Raman coupling method, medium and system

Abstract

The present invention provides a real-time online probe imaging Raman coupling method, medium and system, belonging to the field of spectral imaging technology, including: real-time acquisition of particle images and Raman spectrum data of the substance to be tested obtained in sequence by the imaging Raman coupling probe; real-time processing of the particle images to complete the calculation of the morphological characteristics and particle size distribution of the particles, the morphological characteristics at least including particle size, aspect ratio, roundness, bluntness; synchronous processing of Raman spectrum data to extract characteristic peaks of the substance to be tested; using a preset Raman spectrum analysis model to input the characteristic peaks and output the particle content of the substance to be tested; and storing the particle images, morphological characteristics, particle size distribution, Raman spectrum and particle content data in real time in a database. The present invention realizes the real-time acquisition and analysis of particle morphological characteristics and composition information by imaging Raman coupling, solving the technical problem that the existing means are all single-mode detection and are not accurate enough.

Description

Technical Field

[0001] The present invention belongs to the technical field of spectral imaging, and in particular, relates to a real-time online probe-type imaging Raman coupling method, medium and system. Background Art

[0002] Particle analysis is widely used in many fields such as industrial production and environmental monitoring. Particles of various types and sizes are produced in different processes and environments. Real-time online monitoring of the morphological characteristics, particle size distribution, and component content of these particles is of great significance for optimizing process parameters, assessing the degree of pollution, and preventing environmental risks.

[0003] The morphological characteristics of particulate matter (including particle size, aspect ratio, roundness, etc.) play an important role in the use, packaging, and transportation of particulate matter. At the same time, the physical properties of particulate matter (including chemical composition, crystal form, etc.) are also extremely critical to particulate matter. In the production process of particles, the physical properties and morphological characteristics of particles are not static, but are very dependent on external conditions. For example, in the process of drug crystallization, the morphology and size of crystal particles will change with the change of crystallization conditions (temperature, stirring, etc.). Similarly, in the process of drug crystallization, the crystal form of crystal particles will also change with the change of crystallization conditions. In chemical synthesis, the morphology of the synthetic chemical is an important result. At present, there are many methods for measuring the morphology of particles, such as microscopy, SEM, etc.; there are also many methods for measuring the physical properties of particles, such as infrared, Raman spectroscopy, etc. However, the existing detection methods are often single-mode detection, and there are technical problems of insufficient accuracy. Summary of the invention

[0004] In view of this, the present invention provides a real-time online probe-type imaging Raman coupling method, medium and system, which can solve the technical problem that existing detection means are often single-mode detection and have insufficient accuracy.

[0005] The present invention is achieved in that:

[0006] The present invention provides a real-time online probe type imaging Raman coupling method, comprising the following steps:

[0007] S10, real-time acquisition of particle images and Raman spectrum data of the substance to be tested, which are successively acquired by the imaging Raman coupling probe;

[0008] S20, processing the particle image in real time, completing the calculation of the particle morphological characteristics and particle size distribution within 0.1 second, wherein the morphological characteristics at least include particle size, aspect ratio, roundness, and bluntness; synchronously processing Raman spectrum data to extract characteristic peaks of the substance to be tested;

[0009] S30, using a preset Raman spectroscopy analysis model to input the characteristic peak and output the particle content of the substance to be tested;

[0010] S40, storing the particle image, morphological features, particle size distribution, Raman spectrum and particle content data in a database in real time.

[0011] Wherein, the imaging Raman coupling probe includes an imaging light source, a camera, and a Raman laser device.

[0012] Furthermore, the imaging Raman coupling probe is used to sequentially acquire the following information:

[0013] First, enter the image acquisition mode, turn on the imaging light source, and synchronously trigger the camera to capture the image. The capture time is controlled within 0.01ms to achieve image acquisition and obtain the particle image.

[0014] After the image acquisition is completed, turn off the imaging light source and camera, turn on the Raman spectrum acquisition mode, turn on the Raman laser, and perform Raman spectrum acquisition;

[0015] After the Raman spectrum acquisition is completed, switch back to the image acquisition mode and repeat the above steps.

[0016] Furthermore, the step of calculating the particle size distribution is specifically:

[0017] Binarize the image and separate the particle outline;

[0018] Calculate the equivalent circular diameter of each particle;

[0019] All particles are grouped according to the size of equivalent circle diameter;

[0020] Calculate the percentage of each group of particles to the total number of particles;

[0021] Plot the particle size distribution curve, with the horizontal axis being the equivalent circle diameter of the particle outline and the vertical axis being the cumulative percentage of the particle number to the total particle number.

[0022] The Raman spectroscopy analysis model adopts an improved multi-branch deep learning structure to reduce the impact of noise and baseline drift, including a large characteristic peak sub-network, a small characteristic peak sub-network and a fusion sub-network.

[0023] Furthermore, the large characteristic peak sub-network is used to process the main characteristic peaks in the spectrum; the input is the pre-processed Raman spectrum data; the output is the position and intensity information of the large characteristic peak; the structure is a multi-layer 1D convolutional network, including residual connections and attention mechanisms;

[0024] The small characteristic peak sub-network is used to process the minor characteristic peaks in the spectrum; the input is the preprocessed Raman spectrum data; the output is the position and intensity information of the small characteristic peak; the structure is a multi-layer 1D convolutional network, including dilated convolution and channel attention mechanism;

[0025] The fusion subnetwork is used to integrate the information of large characteristic peaks and small characteristic peaks, and output the final particle content prediction; the input is the output of the large characteristic peak subnetwork and the small characteristic peak subnetwork; the output is the particle content prediction of the substance to be tested; the structure is a multi-layer fully connected network, including a dropout layer and batch normalization.

[0026] This model is an improvement on the traditional single network structure. It improves the model's ability to identify features of different intensities by processing large characteristic peaks and small characteristic peaks separately. The fusion sub-network comprehensively considers the information of the two characteristic peaks to make a more accurate prediction of particle content.

[0027] Furthermore, the threshold ranges of large characteristic peaks and small characteristic peaks are specifically: a large characteristic peak is a characteristic peak with a relative intensity greater than or equal to 30% of the baseline intensity; a small characteristic peak is a characteristic peak with a relative intensity between 5% and 30% of the baseline intensity.

[0028] This multi-branch structure allows the model to better adapt to the characteristics of Raman spectral data, taking into account characteristic peaks of different intensities, improving the accuracy and robustness of the analysis. The large characteristic peak sub-network focuses on capturing the main spectral features, while the small characteristic peak sub-network is able to identify and process those minor features that may be overlooked but are equally important. The fusion network integrates these two types of information and makes a comprehensive judgment. This improved model structure should be able to better handle the various characteristic peaks in the Raman spectrum, thereby improving the accuracy of particle content prediction.

[0029] The calculation of the morphological features adopts a preset morphological feature model, and the morphological feature model adopts a YOLOX model or an improved YOLOX model.

[0030] The improved YOLOX model specifically makes the following improvements on the basis of the original YOLOX model:

[0031] Add CBAM attention mechanism before multiple residual structures of dark3, dark4, and dark5;

[0032] Add an adaptive feature fusion (ASFF) module between Neck and Head;

[0033] Introducing deformable convolution: The last convolution layer of dark3, dark4, and dark5 is replaced with a deformable convolution to enhance the model's adaptability to different particle sizes and shapes;

[0034] Add Feature Pyramid Network (FPN): Introduce FPN structure in the Neck part to better integrate the features of different scales of granularity, aspect ratio, roundness, and bluntness;

[0035] Add Spatial Pyramid Pooling (SPP): Add the SPP module between Backbone and Neck to enhance the model's perception of objects of different granularities;

[0036] Introducing Ghost Convolution: Some standard convolution layers are replaced with Ghost convolution to reduce the amount of calculation while maintaining model performance.

[0037] Optionally, the Mish activation function is also used: the optional part of the ReLU activation function is replaced by the Mish activation function to improve the nonlinear expression ability of the model.

[0038] Wherein, the step S10 specifically includes: firstly, entering the image acquisition mode, turning on the imaging light source, and synchronously triggering the camera to capture the image. In order to achieve real-time performance, the exposure time of the camera is controlled within 0.01 milliseconds, so that the static image of the particles can be quickly captured, and the image will not be blurred due to the movement of the particles. In this way, a high-definition particle image of the substance to be tested can be obtained. After the image acquisition is completed, the imaging light source and the camera are turned off, the Raman spectrum acquisition mode is turned on, and the Raman laser is turned on to collect Raman spectrum data. After the Raman spectrum acquisition is completed, the image acquisition mode is switched back to the image acquisition mode again, and the above steps are executed cyclically. In this way, the morphological image and Raman spectrum data of the particles can be obtained alternately, so as to realize real-time and efficient data acquisition.

[0039] The specific steps of step S20 include: real-time processing of the acquired particle image, and calculation of particle morphological features within 0.1 seconds. The morphological features include parameters such as particle size, aspect ratio, roundness, and bluntness. This requires the use of efficient calculation algorithms, such as the improved YOLOX model, which can extract and calculate these features within 0.1 seconds. At the same time, the collected Raman spectral data needs to be preprocessed to remove interference such as noise and baseline drift, and extract usable spectral feature data. This lays the foundation for subsequent spectral analysis.

[0040] Wherein, the step S30 adopts an improved multi-branch deep learning model. The model includes three parts: a large characteristic peak sub-network, a small characteristic peak sub-network and a fusion sub-network. The large characteristic peak sub-network is used to process the main characteristic peaks in the spectrum whose relative intensity is greater than or equal to 30% of the baseline intensity, and the small characteristic peak sub-network processes the secondary characteristic peaks whose relative intensity is between 5% and 30%. The fusion sub-network comprehensively considers the information of the two characteristic peaks with different intensities to make the final particle content prediction. This multi-branch structure can better adapt to the identification of characteristic peaks with different intensities and improve the accuracy and robustness of the model. The pre-processed Raman spectrum data is input into the above-mentioned improved Raman spectrum analysis model, and the model can output the particle content of the substance to be tested.

[0041] Wherein, the step S40 includes: storing the above-obtained particle images, morphological characteristic parameters (particle size, aspect ratio, roundness, bluntness), particle size distribution data, as well as Raman spectrum data and predicted particle content in a database in real time. This can provide support for subsequent data analysis and application. The entire process is carried out in real time, from data acquisition, feature calculation, spectrum analysis to data storage, all of which can be completed in a very short time. This requires the system to have efficient data processing capabilities, and to be able to complete all calculation and storage operations within 0.1 seconds to meet the real-time requirements.

[0042] Furthermore, the imaging Raman coupling probe includes an imaging light source, a camera, and a Raman laser device. By turning on different devices, particle images and Raman spectrum data can be acquired alternately, thereby achieving real-time and efficient multimodal data acquisition. The imaging light source is used to provide illumination, the camera is used to capture static images of particles, and the Raman laser device is used to excite the Raman spectrum of particles. The three work together to ensure the integrity and high time resolution of data acquisition.

[0043] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned real-time online probe imaging Raman coupling method.

[0044] A third aspect of the present invention provides a real-time online probe-type imaging Raman coupling system, which includes the above-mentioned computer-readable storage medium.

[0045] Compared with the prior art, the beneficial effects of the real-time online probe-type imaging Raman coupling method, medium and system provided by the present invention are: adopting the imaging-Raman alternating acquisition mode, and obtaining high-definition images and Raman spectrum data of the particles to be tested in real time through the imaging Raman coupling probe.

[0046] In terms of image processing, the present invention uses an improved target detection algorithm YOLOX, which can quickly complete the extraction and calculation of particle size, aspect ratio, roundness, bluntness and other morphological features within 0.1 seconds. This greatly improves the analysis efficiency and meets the needs of real-time monitoring.

[0047] In terms of Raman spectroscopy analysis, the present invention proposes a multi-branch deep learning model that can better identify and process the main and secondary characteristic peaks in the spectrum, thereby improving the accuracy of particle composition prediction. The fusion sub-network takes the two characteristic information into consideration and makes the final particle content judgment.

[0048] In summary, the present invention solves the technical problem that the existing detection means are often single-mode detection and have insufficient accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flow chart of the method provided by the present invention;

[0050] Figure 2 This is a schematic diagram of the structure of the equipment in Example 2;

[0051] Figure 3 Schematic diagram of the imaging Raman coupling probe of Example 2. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0053] like Figure 1 As shown, it is a flow chart of a real-time online probe imaging Raman coupling method provided by the present invention, and the method comprises the following steps:

[0054] S10, real-time acquisition of particle images and Raman spectrum data of the substance to be tested, which are successively acquired by the imaging Raman coupling probe;

[0055] S20, processing the particle image in real time, completing the calculation of the particle morphological characteristics and particle size distribution within 0.1 seconds, the morphological characteristics at least including particle size, aspect ratio, roundness, and bluntness; synchronously processing Raman spectrum data to extract characteristic peaks of the substance to be tested;

[0056] S30, using a preset Raman spectroscopy analysis model to input characteristic peaks and output the particle content of the substance to be tested;

[0057] S40, storing the particle image, morphological features, particle size distribution, Raman spectrum and particle content data in a database in real time.

[0058] Wherein, the imaging Raman coupling probe includes an imaging light source, a camera, and a Raman laser device.

[0059] The imaging light source can be an LED light source, a halogen lamp, a laser, etc. The camera includes an industrial CMOS camera or a CCD camera, and the sensor size can be 2 / 3 inch, 1 / 1.8 inch, 1 / 2.3 inch, 1 / 2.5 inch, 1 / 2.7 inch, 1 / 3.2 inch, etc. The global shutter can be used for rapid exposure, with a minimum pixel size of up to 2.7 microns or 3.45 microns, and the optical resolution can reach 500 nanometers in combination with the object square telecentric lens group. The excitation wavelength of the Raman laser device is usually 532nm or 785nm, and it is also equipped with a Raman spectroscopy analysis system, which can adopt: Raman spectroscopy analysis technology of single-channel detection, Raman spectroscopy analysis technology of multi-channel detectors represented by CCD, FT-Raman spectroscopy analysis technology using Fourier transform technology, resonance Raman spectroscopy analysis technology, surface enhanced Raman effect analysis technology, etc.

[0060] The specific implementation methods of the above steps are described in detail below:

[0061] First, step S10 is mainly divided into two sub-steps:

[0062] 1) Image acquisition sub-step. First, enter the image acquisition mode, turn on the imaging light source, and synchronously trigger the camera to capture the image. In order to achieve real-time performance, the camera exposure time is controlled within 0.01 milliseconds, which can quickly capture the static image of the particles without causing image blur due to the movement of the particles. In this way, high-definition particle images of the substance to be tested can be obtained.

[0063] 2) Raman spectrum acquisition sub-step. After the image acquisition is completed, turn off the imaging light source and camera, turn on the Raman spectrum acquisition mode, and turn on the Raman laser to collect Raman spectrum data. After the Raman spectrum acquisition is completed, switch back to the image acquisition mode again and repeat the above steps. In this way, the morphological image and Raman spectrum data of the particles can be obtained alternately, realizing real-time and efficient data acquisition.

[0064] Next is step S20 which mainly includes the following sub-steps:

[0065] 1) Real-time processing of particle images. The acquired particle images are processed in real time to complete the calculation of particle morphological features within 0.1 seconds. Morphological features include parameters such as particle size, aspect ratio, roundness, and bluntness. This requires the use of efficient calculation algorithms, such as the improved YOLOX model, which can complete the extraction and calculation of these features within 0.1 seconds.

[0066] 2) Raman spectrum preprocessing sub-step. At the same time, the collected Raman spectrum data also needs to be preprocessed to remove interference such as noise and baseline drift, and extract usable spectral feature data. This lays the foundation for subsequent spectral analysis.

[0067] Next is step S30 which mainly includes the following sub-steps:

[0068] 1) Raman spectroscopy analysis model construction sub-step. An improved multi-branch deep learning model is used here. The model consists of three parts: a large characteristic peak sub-network, a small characteristic peak sub-network, and a fusion sub-network. The large characteristic peak sub-network is used to process the main characteristic peaks in the spectrum with a relative intensity greater than or equal to 30% of the baseline intensity, and the small characteristic peak sub-network processes the secondary characteristic peaks with a relative intensity between 5% and 30%. The fusion sub-network comprehensively considers the information of the two different intensity characteristic peaks to make the final particle content prediction. This multi-branch structure can better adapt to the identification of characteristic peaks of different intensities and improve the accuracy and robustness of the model.

[0069] 2) Particle content prediction sub-step. The pre-processed Raman spectral data is input into the above-mentioned improved Raman spectral analysis model, and the model can output the particle content of the substance to be tested. This step uses deep learning technology. Through model training, the complex mapping relationship between Raman spectra and particle content can be learned, so that the particle content of the target substance can be accurately predicted.

[0070] Finally, step S40 mainly includes the following sub-steps:

[0071] 1) Data storage sub-step. The particle images, morphological characteristic parameters (particle size, aspect ratio, roundness, bluntness), particle size distribution data, Raman spectrum data and predicted particle content acquired above are stored in the database in real time. This can provide support for subsequent data analysis and application.

[0072] 2) Real-time processing sub-step. The entire process is carried out in real time, from data acquisition, feature calculation, spectral analysis to data storage, all of which can be completed in a very short time. This requires the system to have efficient data processing capabilities and be able to complete all calculation and storage operations within 0.1 seconds to meet real-time requirements.

[0073] In general, this invention adopts the imaging Raman coupling method, which can quickly and accurately extract the morphological characteristics and particle size distribution of particles and predict the particle content of the substance to be tested by real-time acquisition of particle images and Raman spectrum data, combined with efficient image processing algorithms and improved Raman spectrum analysis models. The whole process is carried out in real time, and the data is stored in the database in real time, providing basic data support for subsequent applications.

[0074] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned real-time online probe imaging Raman coupling method.

[0075] A third aspect of the present invention provides a real-time online probe-type imaging Raman coupling system, which includes the above-mentioned computer-readable storage medium.

[0076] Specifically, the principle of the present invention is to achieve real-time acquisition and analysis of particle morphology and composition information through imaging Raman coupling. Among them, the imaging-Raman alternating acquisition method can efficiently obtain static images and Raman spectrum data of particles. The high-speed exposure of the camera ensures clear images and avoids blurring caused by particle movement. The excitation of Raman laser can obtain chemical information of particles.

[0077] In order to complete the morphological feature extraction within 0.1 seconds, the present invention adopts an improved YOLOX model. Compared with the traditional target detection model, YOLOX has higher computational efficiency. By further optimizing the network structure, such as adding attention mechanism, deformable convolution, feature fusion, etc., the model's perception ability of particles of different sizes and shapes is greatly improved, meeting the real-time requirements.

[0078] In terms of Raman spectroscopy analysis, the present invention proposes a multi-branch deep learning model. The model consists of three parts: a large characteristic peak sub-network, a small characteristic peak sub-network, and a fusion sub-network. The large characteristic peak sub-network focuses on extracting the main characteristic peaks in the spectrum, while the small characteristic peak sub-network can identify and process those neglected but equally important secondary characteristic peaks. The fusion sub-network comprehensively considers the two characteristic peak information to make a more accurate prediction of particle content.

[0079] This multi-branch structure design makes full use of the rich information in Raman spectroscopy data and can better adapt to the identification of characteristic peaks of different intensities. Compared with a single network structure, this method also shows stronger robustness under interference such as noise and baseline drift, thereby improving the accuracy of particle composition analysis.

[0080] In order to facilitate understanding and implementation of the present invention, a specific embodiment 1 of the present invention is provided below. The steps of this embodiment 1 are specifically described as follows:

[0081] Figure 2This is a schematic diagram of the structure of the equipment composition of this embodiment 2, including an imaging Raman coupling probe, a light source, a Raman spectrometer, an industrial camera and a host computer. The imaging Raman coupling probe can be directly inserted into the sample under the protection of the probe housing to obtain the image and Raman spectrum of the particles in real time. In the intelligent real-time data processing software, the particle size, aspect ratio, roundness, bluntness and other information will be analyzed, displayed and recorded by the software in real time, and the delay time does not exceed 0.1 seconds. At the same time, in the intelligent real-time data processing software, the characteristic peaks in the Raman spectrum signal will also be intelligently selected, and the content will be predicted according to the existing model, analyzed, displayed and recorded in real time, and the delay time will not exceed 0.1 seconds. A readable storage medium is provided in the host computer, and program instructions are stored in the readable storage medium. When the program instructions are run, they are used to execute a real-time online probe imaging Raman coupling method provided by the present invention. The method is presented on the host computer in the form of software.

[0082] Figure 3 Schematic diagram of the imaging Raman coupling probe of Example 2, Figure 3 As shown, the camera 1 is an industrial CMOS camera or a CCD camera, which can be exposed quickly by using a global shutter.

[0083] The eyepiece 2 is a part of the imaging microscope system, which can magnify the image obtained by the objective lens again.

[0084] The imaging Raman coupling structure 3 is the core component of the imaging Raman coupling probe, which spatially couples the laser entry and return optical paths used for Raman and the light source entry and return optical paths used for imaging to obtain higher resolution and transmission efficiency.

[0085] The objective lens 4 is a part of the imaging microscope system. It is a combination of a group of lenses. The quality of the objective lens directly affects the image quality of the microscope. It is the main component that determines the resolution and imaging clarity of the microscope.

[0086] Raman optical fiber interface 5, the interface can be SMA, FC, SC, LC, ST and other optical fiber interfaces, among which SMA interface is preferred. The two interfaces are respectively a laser interface and an optical fiber interface leading to the Raman spectrometer.

[0087] Axial fine-tuning structure 6, the fine-tuning structure has three buttons, the upper button is the micro-focus fine-tuning button, which is used to control the imaging clarity. The lower two buttons are the imaging light path and Raman light path fine-tuning buttons, which are used to control the position of the imaging area and the Raman area. The imaging area and the Raman area can be overlapped by these two buttons.

[0088] Imaging light source interface 7, the imaging light source can be LED light source, halogen lamp, laser, etc. The light source is programmed and controlled by the light source controller for intensity, strobe, exposure time, etc.

[0089] Let’s start with the following steps:

[0090] First, immerse the imaging Raman coupling probe into the sample to be tested. Check the hardware and connection status. After confirming that the hardware status is normal, turn on the switches of each device and the main switch.

[0091] Open the real-time online probe imaging Raman coupling software. The self-check status of the main hardware parts will be displayed on the software waiting interface. After confirming that there is no problem with the hardware, enter the software main page. If there is a problem with the self-check, wait until the problem is eliminated before entering the software main interface; the software performs the following 4 steps:

[0092] First, step S10 can be divided into two sub-steps: image acquisition and Raman spectrum acquisition.

[0093] For the image acquisition sub-step, a function I(t) can be defined to represent the particle image acquired at time t. In order to achieve real-time performance, the camera exposure time T exp Controlled within 0.01 milliseconds, that is:

[0094] T exp ≤0.01ms

[0095] This allows for the rapid capture of static images of particles, avoiding blurring caused by particle motion. The image I(t) acquired in this way has high definition and can be used for subsequent morphological feature analysis.

[0096] For the Raman spectrum acquisition sub-step, S(t) can be defined to represent the Raman spectrum data acquired at time t. After completing the image acquisition, turn off the imaging light source and camera, turn on the Raman spectrum acquisition mode, and turn on the Raman laser for spectrum acquisition. After the Raman spectrum acquisition is completed, switch back to the image acquisition mode and repeat the above steps. This alternating acquisition method can achieve high time resolution acquisition of particle images and Raman spectrum data, reflecting real-time performance.

[0097] Next is step S20 which can be divided into two sub-steps: particle image processing and Raman spectrum preprocessing.

[0098] For the particle image processing sub-step, a function F(I(t)) can be defined to represent real-time processing of the image I(t) and complete the extraction of particle morphological features within 0.1 seconds. This requires the use of efficient image processing algorithms, such as the improved YOLOX model. The YOLOX model is a target detection algorithm based on deep learning that can quickly identify and locate targets from images. A series of improvements have been made to the original YOLOX model, including:

[0099] 1. Add CBAM attention mechanism before multiple residual structures of dark3, dark4, and dark5 to enhance the model's ability to capture key features. CBAM can adaptively adjust feature channels and spatial weights to improve the model's perception ability.

[0100] 2. Add the ASFF module between Neck and Head to better integrate features of different scales and enhance the model's ability to recognize particles of different sizes.

[0101] 3. The last convolution layer of dark3, dark4, and dark5 is replaced with a deformable convolution to enhance the model’s adaptability to particles of different shapes.

[0102] 4. Introduce the FPN structure in the Neck part to better integrate features such as particle size, aspect ratio, roundness, bluntness, etc. of different scales.

[0103] 5. Add the SPP module between Backbone and Neck to enhance the model's perception of targets of different granularities.

[0104] 6. Replace some standard convolutional layers with Ghost convolution to reduce the amount of calculation while maintaining model performance.

[0105] 7. Replace some ReLU activation functions with Mish activation functions to improve the nonlinear expression ability of the model.

[0106] Through these improvements, the improved YOLOX model can extract the morphological characteristic parameters such as particle size, aspect ratio, roundness, and bluntness from the image I(t) within 0.1 seconds. These characteristic parameters are recorded as P(t) = {p1(t), p2(t), p3(t), p4(t)}, where p1 represents particle size, p2 represents aspect ratio, p3 represents roundness, and p4 represents bluntness.

[0107] At the same time, it is also necessary to preprocess the Raman spectral data S(t) to remove interference such as noise and baseline drift and extract usable spectral features. A function G(S(t)) can be defined to represent this preprocessing process and output the preprocessed spectral data This lays the foundation for subsequent spectral analysis.

[0108] Next, step S30 involves an improved multi-branch deep learning model, which can be expressed as

[0109] The model consists of three sub-networks: large feature peak sub-network, small feature peak sub-network and fusion sub-network.

[0110] The large feature peak sub-network is denoted as It is used to process the main characteristic peaks in the spectrum whose relative intensity is greater than or equal to 30% of the baseline intensity. It adopts a multi-layer 1D convolutional network structure and contains residual connections and attention mechanisms, which can effectively extract the position and intensity information of large characteristic peaks.

[0111] The small feature peak sub-network is denoted as It is used to process minor feature peaks with relative intensity between 5% and 30%. It also uses a multi-layer 1D convolutional network, but uses dilated convolution and channel attention mechanisms to enhance the perception of weak features.

[0112] The fusion sub-network is denoted as It is used to comprehensively consider the information of large characteristic peaks and small characteristic peaks to make the final particle content prediction. It consists of a multi-layer fully connected network, including dropout layers and batch normalization to improve the generalization ability of the model.

[0113] Through this multi-branch structure, the model can better adapt to the characteristics of Raman spectral data, make full use of the information of characteristic peaks with different intensities, and improve the accuracy and robustness of the analysis. Input into the model to get the particle content prediction result of the substance to be tested.

[0114] Finally, step S40 mainly includes two sub-steps: data storage and real-time processing.

[0115] For the data storage sub-step, you can define a function The acquired particle image I(t), morphological features P(t), and preprocessed Raman spectrum and the predicted particle content Stored in the database in real time, this can provide basic data support for subsequent data analysis and application.

[0116] In addition, before the actual test, the imaging and Raman test parameters can be set according to the conditions of the test sample. For imaging, the main parameters include exposure time, pulse light intensity and frequency, storage time interval, image processing parameters, etc. For Raman, the main parameters include laser power, Raman spectrum integration time, etc.

[0117] During real-time measurement, in order to avoid interference of the imaging light source and the Raman laser on their respective signals, the imaging light source and the Raman laser are switched to each other for continuous detection in the present invention. First, the imaging light source turns on the synchronized camera to capture the image. Due to the use of a pulsed light source, the entire image capture process can be controlled within 0.01ms at the fastest. After the image acquisition is completed, the software automatically controls the switch to Raman spectrum acquisition. For different material systems, the Raman integration time is different, and the fastest can be up to 200ms. After that, it switches to image acquisition to cycle these two processes.

[0118] For the real-time processing sub-step, the entire process from data acquisition, feature calculation, spectral analysis to data storage needs to be completed in a very short time. Define a time threshold T total Represents the total time of the entire process, which must meet the following requirements:

[0119] T total ≤0.1s

[0120] This requires the system to have efficient data processing capabilities and be able to complete all calculations and storage operations within 0.1 seconds to meet real-time requirements.

[0121] In general, this invention uses imaging Raman coupling to collect particle images and Raman spectrum data in real time, combined with efficient image processing algorithms and improved Raman spectrum analysis models, to quickly and accurately extract the morphological characteristics and particle size distribution of particles, and predict the particle content of the substance to be tested. The entire process is carried out in real time, and the data is stored in the database in real time, providing basic data support for subsequent applications.

[0122] In order to better understand and implement the present invention, the following is a specific application scenario of the present invention, Example 2: A chemical plant will produce a large amount of fine particles during the production process. The particle size distribution, morphology and chemical composition of these particles directly affect product quality and environmental emissions. In order to monitor these key parameters in real time, the chemical plant adopts the real-time online probe imaging Raman coupling method proposed in the present invention.

[0123] First, the factory installed several imaging Raman coupling probes at key locations in the production workshop. These probes include imaging light sources, high-speed cameras, and Raman laser equipment, which can realize the alternating acquisition of particle morphology images and Raman spectral data. The camera's exposure time is controlled within 0.01 milliseconds, which can quickly capture static images of particles and avoid blurring caused by particle movement.

[0124] Every few seconds, the imaging probe will automatically start data acquisition. First, enter the image acquisition mode, turn on the imaging light source, and use the camera to take high-speed pictures to acquire particle images. After the image acquisition is completed, turn off the imaging light source and camera, turn on the Raman spectrum acquisition mode, excite the Raman laser, and test the Raman spectrum of the particles. After the Raman spectrum acquisition is completed, switch back to the image acquisition mode and repeat the above steps. Through this alternating acquisition method, multimodal data of particles can be obtained in a short time.

[0125] These real-time collected particle images and Raman spectrum data are transmitted to the data processing system in the factory's central control room. In this system, the particle images are first processed in real time using the improved YOLOX target detection algorithm. The particle size, aspect ratio, roundness, bluntness and other morphological features are extracted and calculated within 0.1 seconds.

[0126] For example, for a particle image collected at a certain time, the YOLOX model detected 208 particle targets and calculated their morphological characteristic parameters as shown in Table 1:

[0127] Table 1 Morphological characteristic parameters

[0128] Characteristic parameters average value Standard Deviation Particle size (μm) 15.7 4.2 Aspect Ratio 1.38 0.26 Roundness 0.78 0.11 Bluntness 0.52 0.08

[0129] It can be seen that the average particle size of this batch of particles is 15.7 microns, the aspect ratio is relatively high, and the shape is relatively irregular.

[0130] At the same time, the Raman spectral data is also sent to the preprocessing module to remove interference such as noise and baseline drift and extract usable spectral features.

[0131] Then, the preprocessed Raman spectrum data is input into the improved multi-branch deep learning model proposed by the present invention for analysis. The model includes three parts: a large characteristic peak sub-network, a small characteristic peak sub-network and a fusion sub-network.

[0132] The large characteristic peak sub-network focuses on extracting the main characteristic peaks in the spectrum whose relative intensity is greater than or equal to 30% of the baseline intensity. Its output is shown in Table 2:

[0133] Table 2 Main characteristic peaks of the large characteristic peak sub-network

[0134]

[0135]

[0136] The small feature peak sub-network identifies the minor feature peaks with relative intensities between 5% and 30%, and its output is shown in Table 3:

[0137] Table 3 Main characteristic peaks of the small characteristic peak sub-network

[0138] <![CDATA[Characteristic peak position (cm -1 )]]> Relative strength (%) 1165 15.2 1435 21.7 2725 18.4

[0139] The fusion sub-network comprehensively considers the information of the above two characteristic peaks and finally gives the content prediction result of this batch of particles:

[0140] Silicon dioxide (SiO2): 65.4%

[0141] Calcium carbonate (CaCO3): 18.2%

[0142] Iron oxide (Fe2O3): 16.4%

[0143] The entire data processing process, from image acquisition, feature extraction to spectral analysis, is completed within 0.1 seconds, thanks to the high-performance computing capability of the system and the efficient algorithm adopted by the present invention.

[0144] The particle morphology, particle size distribution and chemical composition data obtained above are stored in the database in real time. The production management personnel of the factory can check the real-time changes of these key parameters at any time to provide a basis for the optimization of the production process.

[0145] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A real-time online probe imaging Raman coupling method, characterized in that: The following steps are involved: S10, real-time acquisition of particle images and Raman spectrum data of the substance to be tested, which are successively acquired by the imaging Raman coupling probe; S20, processing the particle image in real time, completing the calculation of the particle morphological characteristics and particle size distribution within 0.1 second, wherein the morphological characteristics at least include particle size, aspect ratio, roundness, and bluntness; synchronously processing Raman spectrum data to extract characteristic peaks of the substance to be tested; S30, using a preset Raman spectroscopy analysis model to input the characteristic peak and output the particle content of the substance to be tested; S40, storing the particle image, morphological features, particle size distribution, Raman spectrum and particle content data in a database in real time; The Raman spectroscopy analysis model adopts an improved multi-branch deep learning structure to reduce the impact of noise and baseline drift, including a large characteristic peak sub-network, a small characteristic peak sub-network and a fusion sub-network; wherein the large characteristic peak sub-network is used to process the main characteristic peaks in the spectrum; the input is the preprocessed Raman spectroscopy data; the output is the position and intensity information of the large characteristic peak; the structure is a multi-layer 1D convolutional network, including residual connections and attention mechanisms; the small characteristic peak sub-network is used to process the secondary characteristic peaks in the spectrum; the input is the preprocessed Raman spectroscopy data; the output is the position and intensity information of the small characteristic peak Intensity information; the structure is a multi-layer 1D convolutional network, including dilated convolution and channel attention mechanism; the fusion sub-network is used to integrate the information of large characteristic peaks and small characteristic peaks, and output the final particle content prediction; the input is the output of the large characteristic peak sub-network and the small characteristic peak sub-network; the output is the particle content prediction of the substance to be tested; the structure is a multi-layer fully connected network, including dropout layer and batch normalization; wherein, the threshold range of large characteristic peaks and small characteristic peaks is specifically: the large characteristic peak is a characteristic peak with a relative intensity greater than or equal to 30% of the baseline intensity; the small characteristic peak is a characteristic peak with a relative intensity between 5% and 30% of the baseline intensity.

2. A real-time online probe imaging Raman coupling method according to claim 1, characterized in that: The imaging Raman coupling probe comprises an imaging light source, a camera, and a Raman laser device.

3. A real-time online probe imaging Raman coupling method according to claim 2, characterized in that: The imaging Raman coupling probe is used to obtain the data sequentially, specifically: First, enter the image acquisition mode, turn on the imaging light source, and synchronously trigger the camera to capture the image. The capture time is controlled within 0.01ms to achieve image acquisition and obtain the particle image. After the image acquisition is completed, turn off the imaging light source and camera, turn on the Raman spectrum acquisition mode, turn on the Raman laser, and perform Raman spectrum acquisition; After the Raman spectrum acquisition is completed, switch back to the image acquisition mode and repeat the above steps.

4. A real-time online probe imaging Raman coupling method according to claim 3, characterized in that: The calculation steps of the particle size distribution are specifically: Binarize the image and separate the particle outline; Calculate the equivalent circular diameter of each particle; All particles are grouped according to the size of equivalent circle diameter; Calculate the percentage of each group of particles to the total number of particles; Plot the particle size distribution curve, with the horizontal axis being the equivalent circle diameter of the particle outline and the vertical axis being the cumulative percentage of the particle number to the total particle number.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the real-time online probe-type imaging Raman coupling method according to any one of claims 1 to 4.

6. A real-time online probe imaging Raman coupling system, characterized in that: Contains the computer-readable storage medium of claim 5.

Images