A hyperspectral foreign body detection method for liquid medicine integrating vision and spectral significance

Abstract

The present invention discloses a hyperspectral liquid foreign body detection method that integrates vision and spectral significance. The method comprises the following steps: capturing a hyperspectral original image of the liquid to be detected and constructing a hyperspectral image sample set; selecting a preset number of bands from the hyperspectral image sample using an information entropy-based sorting band selection method, and converting the bands into pseudo-color images; performing histogram contrast calculation and gradient feature extraction on the pseudo-color image, and performing feature fusion to obtain a visual-based significance feature map. After total variation filtering, a preliminary anomaly detection map is obtained; calculating the spectral angular distance and spectral Euclidean distance of the hyperspectral image of the liquid to be detected and fusing them to obtain a spectral significance feature map; using the spectral significance feature map to suppress background information of the preliminary anomaly detection map, a foreign body detection map is obtained, and foreign body detection of the liquid to be detected is realized. The method can quickly and accurately detect weak foreign bodies in finished liquid medicines, achieving non-destructive testing and quality control of the liquid medicines.

Description

Technical Field

[0001] The present invention relates to the field of medical hyperspectral intelligent detection and analysis, and in particular to a hyperspectral foreign body detection method for liquid medicine that integrates visual saliency and dual spectral saliency. The method couples visual saliency features and dual spectral attention mechanism technology and can be used for rapid and accurate detection of weak foreign bodies in liquid medicine. Background Art

[0002] Traditional Chinese medicine (TCM) liquids, with their advantages of high concentration, ease of administration, and easy storage, have become a major product segment in the pharmaceutical industry. However, the complex production process, including batching, cleaning, filling, and sterilization, makes them susceptible to the presence of minute foreign matter, such as drug residue, fibers, and glass fragments. These foreign matter pose a serious threat to consumers' safety. Therefore, strict quality control of foreign matter generated during the production of liquid medicinal preparations is urgently needed.

[0003] Currently, the pharmaceutical industry primarily uses manual visual inspection or machine vision-based inspection systems for quality control of liquid medicines. Manual inspection involves workers visually inspecting under strong light, which is not only time-consuming and labor-intensive, but also very harmful to the eyes. Machine vision-based inspection systems rotate the liquid medicine bottle to allow foreign matter to rise from the bottom of the bottle while simultaneously capturing and processing continuous images with a CCD camera to detect foreign matter. However, most liquid medicines are traditional Chinese medicine preparations, which are highly sensitive to light. Therefore, they are typically packaged in dark bottles, making it difficult for cameras to capture foreign matter that resembles the bottle's color. Furthermore, conventional RGB cameras cannot analyze the type and composition of foreign matter. Hyperspectral imaging technology can simultaneously acquire spectral and spatial information, accurately reflecting the physical and chemical properties of different substances, providing valuable information for foreign matter detection. Furthermore, hyperspectral anomaly detection requires no prior information and can distinguish different targets based on their spectral characteristics. Therefore, hyperspectral anomaly detection technology is applicable to a wide range of foreign matter detection in the pharmaceutical industry. However, the amount of hyperspectral data is large, the redundancy between data is high, and the types of foreign matter in the medicinal liquid are diverse and the shapes are complex, especially foreign matter such as fibers are very weak. Existing anomaly detection methods are difficult to extract effective feature information of foreign matter, and the accuracy of foreign matter detection is not high. Therefore, it is necessary to address the problem of weak foreign matter detection and develop a method that can effectively extract the significant features of foreign matter from hyperspectral images of traditional Chinese medicine liquid, detect foreign matter quickly and accurately, and realize non-destructive quality control of weak foreign matter in various types of traditional Chinese medicine liquid. Summary of the Invention

[0004] The present invention proposes a hyperspectral liquid medicine foreign body detection method that integrates visual saliency and dual spectral saliency. By converting hyperspectral images into color images to effectively extract visual saliency features, a dual spectral attention mechanism is proposed to extract spectral saliency features, thereby achieving accurate detection of foreign matter in liquid medicine.

[0005] A hyperspectral method for detecting foreign matter in liquid medicine by integrating vision and spectral significance comprises the following steps:

[0006] S1, capturing a hyperspectral original image of the drug solution to be tested, and constructing a hyperspectral image sample set of the drug solution to be tested based on the hyperspectral original image;

[0007] S2. randomly selecting a hyperspectral image sample from the hyperspectral image sample set, selecting a preset number of bands from the hyperspectral image sample using a sorting band selection method based on information entropy, and converting the preset number of bands into a pseudo-color image corresponding to the hyperspectral image sample;

[0008] S3. Perform histogram contrast calculation and gradient feature extraction on the pseudo-color image, and perform feature fusion to obtain a visually significant feature map. Perform total variation filtering on the visually significant feature map to obtain a preliminary anomaly detection map.

[0009] S4. Calculate the spectral angular distance and spectral Euclidean distance based on the hyperspectral image samples and fuse them to obtain a spectral saliency feature map based on the dual spectral attention mechanism. Use the spectral saliency feature map to suppress the background information of the preliminary anomaly detection map to obtain a foreign body detection map.

[0010] S5. Traverse all hyperspectral image samples in the hyperspectral image sample set, and process them through steps S2 to S4 to obtain a plurality of foreign matter detection images, thereby realizing foreign matter detection of the liquid medicine to be detected.

[0011] Preferably, in S1, a hyperspectral image sample set of the drug solution to be detected is constructed based on the hyperspectral original image. The hyperspectral image sample set can be expressed as:

[0012] D={(X1,Y1),(X2,Y2),…,(X i ,Y i ),…,(X m ,Y m )}

[0013] Where D is the sample set of hyperspectral images of the liquid to be tested, X i is the i-th hyperspectral image sample of the drug to be detected in the hyperspectral image sample set D of the drug to be detected, i=1,2,…,m, m is the number of samples in the hyperspectral image sample set of the drug to be detected, N is the number of pixels in the hyperspectral image of the liquid to be tested, B is the number of bands in the hyperspectral image of the liquid to be tested, and Y i is the foreign body label corresponding to the i-th hyperspectral image sample of the drug to be detected in the hyperspectral image sample set D of the drug to be detected.

[0014] Preferably, in S2, a sorting band selection method based on information entropy is used to select a preset number of bands from the hyperspectral image samples, and the preset number of bands are converted into pseudo-color images corresponding to the hyperspectral image samples, specifically including:

[0015] S21, calculating the probability density of each band in the hyperspectral image sample;

[0016] S22, calculating the information entropy of the corresponding band in the hyperspectral image sample according to the probability density;

[0017] S23. Arrange the bands in the hyperspectral image sample in descending order according to information entropy, select a preset number of bands as three primary color bands, and use the three primary color bands as pseudo-color images corresponding to the hyperspectral image sample.

[0018] Preferably, in S22, the information entropy of the corresponding band in the hyperspectral image sample is calculated according to the probability density, and the calculation formula of the information entropy is specifically as follows:

[0019]

[0020] Among them, p k (w)=h k (w) / N

[0021] Where H k is the information entropy of the kth band, p k (w) is the probability density of the kth band, w is the grayscale level of each band image pixel, Ω is the grayscale color space, h k (w) is the grayscale histogram of the kth band, k = 1, 2, ..., B, B is the number of bands of the hyperspectral image of the liquid to be detected, and N is the number of pixels of the hyperspectral image of the liquid to be detected.

[0022] Preferably, in S3, histogram contrast calculation and gradient feature extraction are performed on the pseudo-color image, and feature fusion is performed to obtain a visually significant feature map, which specifically includes:

[0023] S31, converting the pseudo color image from RGB space to Lab space, and converting the RGB component of each pixel in the pseudo color image into L, a, and b components accordingly;

[0024] S32, performing color histogram statistics on the pseudo-color image, calculating the significant features of pixels of the same color, and obtaining the color significant features of the corresponding pixels;

[0025] S33, calculating the horizontal gradient feature and the vertical gradient feature for each pixel in the pseudo-color image, and fusing the horizontal gradient feature and the vertical gradient feature to obtain the multi-scale gradient feature of the corresponding pixel;

[0026] S34, fusing the color saliency feature with the multi-scale gradient feature to obtain a visually-based saliency feature of the corresponding pixel;

[0027] S35 , traverse all pixels in the pseudo-color image, and process through steps S32 to S34 to obtain a visually significant feature map corresponding to the pseudo-color image.

[0028] Preferably, the color saliency feature of the corresponding pixel point is obtained in S32, and the specific formula is:

[0029]

[0030] Where S(x,y) is the color saliency feature of pixel (x,y), C u is the color value of the pixel (x, y), which is the u-th color value in the pseudo-color image, C v is the vth color value in the pseudo-color image, n is the total number of color values ​​in the pseudo-color image, u,v∈1,2,…,n, p v is the vth color value C in the pseudo-color image v Probability of occurrence.

[0031] Preferably, the multi-scale gradient feature of the corresponding pixel point is obtained in S33, and the specific formula is:

[0032]

[0033] in,

[0034]

[0035] Where G(x,y) is the multi-scale gradient feature of the pixel point in the xth row and yth column in the pseudo-color image, G h (x, y) is the horizontal gradient feature of the pixel at the xth row and yth column in the pseudo-color image, G v (x,y) is the vertical gradient feature G of the pixel at the xth row and yth column in the pseudo-color image v (x,y), I(x,y+1) refers to the pixel value of the pixel in the xth row and y+1th column in the pseudo-color image, I(x,y-1) refers to the pixel value of the pixel in the xth row and y-1th column in the pseudo-color image, I(x+1,y) refers to the pixel value of the pixel in the xth row and y-1th column in the pseudo-color image, and I(x-1,y) refers to the pixel value of the pixel in the x-1th row and yth column in the pseudo-color image.

[0036] Preferably, the visually significant feature of the corresponding pixel point is obtained in S34, and the specific formula is:

[0037]

[0038] Where V(x,y) is the visual saliency feature of the pixel in the x-th row and y-th column in the pseudo-color image, S(x,y) is the color saliency feature of the pixel (x,y), and G(x,y) is the multi-scale gradient feature of the pixel in the x-th row and y-th column in the pseudo-color image.

[0039] Preferably, in S4, the spectral angular distance and the spectral Euclidean distance are calculated and fused according to the hyperspectral image samples. The calculation formula of the spectral angular distance is specifically:

[0040]

[0041] Where S SAD (A p ,A q ) is the spectral angle distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, A p is the spectral vector of the pth pixel in the hyperspectral image sample of the liquid to be detected, A q is the spectral vector of the qth pixel in the hyperspectral image sample of the liquid to be detected, and N is the number of pixels in the hyperspectral image of the liquid to be detected.

[0042] Preferably, in S4, the spectral angular distance and the spectral Euclidean distance are calculated and fused according to the hyperspectral image samples. The calculation formula of the spectral Euclidean distance is specifically:

[0043]

[0044] Where S SED (A p ,A q ) is the spectral Euclidean distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, A p is the spectral vector of the pth pixel in the hyperspectral image sample of the liquid to be detected, A q is the spectral vector of the qth pixel in the hyperspectral image sample of the liquid to be detected, N is the number of pixels in the hyperspectral image of the liquid to be detected, B is the number of bands in the hyperspectral image of the liquid to be detected, and k = 1, 2, …, B.

[0045] The above-mentioned hyperspectral liquid foreign body detection method that integrates vision and spectral saliency extracts three bands with the largest information entropy from the hyperspectral image samples of the liquid to be tested and converts them into pseudo-color images. This method not only retains the most important spectral features, but also enables the application of saliency detection algorithms to hyperspectral data. By fusing the color saliency features and multi-scale gradient features extracted from the pseudo-color images, it not only effectively extracts the color, contour, and texture information in the pseudo-color images, but also weakens the influence of noise. The total variation filter is used to filter the preliminary anomaly detection map, effectively reducing the background interference of the liquid bottle. The spectral angular distance and spectral Euclidean distance are calculated and fused for the hyperspectral image samples of the liquid to be tested, effectively extracting dual spectral saliency features. The spectral saliency feature map is multiplicatively transformed and fused with the preliminary anomaly detection map, further suppressing the background and noise information in the preliminary anomaly detection map. At the same time, the influence of the bottleneck of the liquid bottle and the liquid level line of the liquid are eliminated. The method can quickly and accurately detect weak foreign bodies in the liquid to be tested, realizing non-destructive testing and quality control of the liquid. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a hyperspectral liquid foreign body detection method that integrates vision and spectral significance in one embodiment of the present invention;

[0047] Figure 2 Schematic diagram of a framework of a hyperspectral liquid foreign body detection method integrating vision and spectral significance in one embodiment of the present invention;

[0048] Figure 3 FIG. 1 is a schematic diagram of some samples of a hyperspectral image sample set in one embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0050] A hyperspectral method for detecting foreign matter in liquid medicine that integrates vision and spectral significance, specifically comprising:

[0051] S1, capturing a hyperspectral original image of the drug solution to be tested, and constructing a hyperspectral image sample set of the drug solution to be tested based on the hyperspectral original image;

[0052] S2. randomly selecting a hyperspectral image sample from the hyperspectral image sample set, selecting a preset number of bands from the hyperspectral image sample using a sorting band selection method based on information entropy, and converting the preset number of bands into a pseudo-color image corresponding to the hyperspectral image sample;

[0053] S3. Perform histogram contrast calculation and gradient feature extraction on the pseudo-color image, and perform feature fusion to obtain a visually significant feature map. Perform total variation filtering on the visually significant feature map to obtain a preliminary anomaly detection map.

[0054] S4. Calculate the spectral angular distance and spectral Euclidean distance based on the hyperspectral image samples and fuse them to obtain a spectral saliency feature map based on the dual spectral attention mechanism. Use the spectral saliency feature map to suppress the background information of the preliminary anomaly detection map to obtain a foreign body detection map.

[0055] S5. Traverse all hyperspectral image samples in the hyperspectral image sample set, and process them through steps S2 to S4 to obtain a plurality of foreign matter detection images, thereby realizing foreign matter detection of the liquid medicine to be detected.

[0056] Specifically, see Figure 1 and Figure 2 , Figure 1 This is a flow chart of a hyperspectral liquid medicine foreign body detection method that integrates vision and spectral significance in one embodiment of the present invention. Figure 2 Schematic diagram of a framework of a hyperspectral liquid medicine foreign body detection method integrating vision and spectral significance in one embodiment of the present invention.

[0057] First, a hyperspectral original image of the drug liquid to be tested is captured. A hyperspectral image sample set of the drug liquid to be tested is constructed based on the hyperspectral original image. A hyperspectral image sample is randomly selected from the hyperspectral image sample set, and three bands are selected and converted into a pseudo-color image. Then, histogram contrast calculation and gradient feature extraction are performed on the pseudo-color image to obtain the salient features and multi-scale gradient features with the largest color feature variance. The color salient features and multi-scale gradient features are then fused to obtain a preliminary anomaly detection map based on visual saliency, which is then subjected to total variation filtering to obtain a preliminary anomaly detection map. Then, the spectral angular distance and spectral Euclidean distance are calculated based on the hyperspectral image sample and the two are fused to obtain a spectral saliency feature map based on a dual spectral attention mechanism. Finally, the dual spectral saliency feature map is used to suppress the background information of the preliminary anomaly detection map to obtain a foreign body detection map. Another hyperspectral image sample is selected from the hyperspectral image sample set until all hyperspectral image samples are selected. The above method is used to process the results to obtain multiple foreign body detection maps, thereby realizing foreign body detection of the drug liquid to be tested.

[0058] In one embodiment, in S1, a hyperspectral image sample set of the to-be-detected liquid medicine is constructed based on the hyperspectral original image. The hyperspectral image sample set can be expressed as:

[0059] D={(X1,Y1),(X2,Y2),…,(X i ,Y i ),…,(Xm ,Y m )}

[0060] Where D is the sample set of hyperspectral images of the liquid to be tested, X i is the i-th hyperspectral image sample of the drug to be detected in the hyperspectral image sample set D of the drug to be detected, i=1,2,…,m, m is the number of samples in the hyperspectral image sample set of the drug to be detected, N is the number of pixels in the hyperspectral image of the liquid to be tested, B is the number of bands in the hyperspectral image of the liquid to be tested, and Y i is the foreign body label corresponding to the i-th hyperspectral image sample of the drug to be detected in the hyperspectral image sample set D of the drug to be detected.

[0061] Specifically, see Figure 3 , Figure 3 This is a schematic diagram of some samples of a hyperspectral image sample set according to an embodiment of the present invention, wherein: Figure 3 (a) is a schematic diagram of a sample containing glass foreign matter. Figure 3 (b) Schematic diagram of a sample containing fibrous foreign matter.

[0062] First, obtain a sample of the drug solution to be tested. Use a hyperspectral imaging system to collect the original hyperspectral image of the drug solution to be tested. Perform reflectance correction on the original hyperspectral image to obtain a corrected hyperspectral image. Use the corrected hyperspectral image to construct a sample set of hyperspectral images of the drug solution to be tested:

[0063] D={(X1,Y1),(X2,Y2),…,(X i ,Y i ),…,(X m ,Y m )}

[0064] Where D is the sample set of hyperspectral images of the liquid to be tested, X i is the i-th hyperspectral image sample of the drug to be detected in the hyperspectral image sample set D of the drug to be detected, i=1,2,…,m, m is the number of samples in the hyperspectral image sample set of the drug to be detected, is a set of real numbers, N is the number of pixels of the hyperspectral image of the liquid to be detected, that is, the total number of pixels of the hyperspectral image of the liquid to be detected, B is the number of bands of the hyperspectral image of the liquid to be detected, and Y i is the foreign body label corresponding to the hyperspectral image sample of the i-th liquid in the hyperspectral image sample set D of the liquid to be tested. For each hyperspectral image of the liquid to be tested, the number of bands is fixed, that is, B is 176 bands.

[0065] The hyperspectral imaging system preferably utilizes a Sunlight Hyperspectral Instrument (V10E) with a spectral range of 400-1000 nm. It uses transmission imaging to capture raw hyperspectral images of the drug solution to be tested, along with the reagent bottle. It should be noted that while this embodiment primarily uses glass and fiber as examples for foreign body detection, the types of drug solutions and foreign bodies in practice are not limited to these.

[0066] In one embodiment, S2 uses an information entropy-based sorting band selection method to select a preset number of bands from the hyperspectral image samples, and converts the preset number of bands into pseudo-color images corresponding to the hyperspectral image samples, specifically including:

[0067] S21, calculating the probability density of each band in the hyperspectral image sample;

[0068] S22, calculating the information entropy of the corresponding band in the hyperspectral image sample according to the probability density;

[0069] S23. Arrange the bands in the hyperspectral image sample in descending order according to information entropy, select a preset number of bands as three primary color bands, and use the three primary color bands as pseudo-color images corresponding to the hyperspectral image sample.

[0070] In one embodiment, in S22, the information entropy of the corresponding band in the hyperspectral image sample is calculated according to the probability density. The calculation formula of the information entropy is specifically as follows:

[0071]

[0072] Among them, p k (w)=h k (w) / N

[0073] Where H k is the information entropy of the kth band, p k (w) is the probability density of the kth band, w is the grayscale level of each band image pixel, Ω is the grayscale color space, h k (w) is the grayscale histogram of the kth band, k = 1, 2, ..., B, B is the number of bands of the hyperspectral image of the liquid to be detected, and N is the number of pixels of the hyperspectral image of the liquid to be detected.

[0074] Specifically, a preset number of bands are selected from the hyperspectral image sample of the drug solution to be detected and converted into corresponding pseudo-color images, including the following process:

[0075] 1) Calculate the probability density of each band in the hyperspectral image sample:

[0076] p k (w)=h k (w) / N

[0077] Where p k (w) is the probability density of the kth band, h k (w) is the grayscale histogram of the kth band, w is the grayscale level of each band image pixel, N is the number of pixels in the hyperspectral image of the drug to be detected, k = 1, 2, ..., B, B is the number of bands in the hyperspectral image of the drug to be detected.

[0078] 2) Calculate the information entropy of each band in the hyperspectral image sample:

[0079]

[0080] Where H k is the information entropy of the kth band, p k (w) is the probability density of the kth band, w is the grayscale level of each band image pixel, and Ω is the grayscale color space.

[0081] 3) Arrange the bands in the hyperspectral image sample in descending order according to the information entropy, and select the first three bands with large corresponding information entropy as the three primary color bands. Specifically, according to the range of the three primary color bands of R, G, and B, the first three corresponding bands are matched to the three primary color bands closest to them, and use them as the pseudo-color image corresponding to the hyperspectral image sample.

[0082] In one embodiment, S3 performs histogram contrast calculation and gradient feature extraction on the pseudo-color image, and performs feature fusion to obtain a visually significant feature map, specifically including:

[0083] S31, converting the pseudo color image from RGB space to Lab space, and converting the RGB component of each pixel in the pseudo color image into L, a, and b components accordingly;

[0084] S32, performing color histogram statistics on the pseudo-color image, calculating the significant features of pixels of the same color, and obtaining the color significant features of the corresponding pixels;

[0085] S33, calculating the horizontal gradient feature and the vertical gradient feature for each pixel in the pseudo-color image, and fusing the horizontal gradient feature and the vertical gradient feature to obtain the multi-scale gradient feature of the corresponding pixel;

[0086] S34, fusing the color saliency feature with the multi-scale gradient feature to obtain a visually-based saliency feature of the corresponding pixel;

[0087] S35 , traverse all pixels in the pseudo-color image, and process through steps S32 to S34 to obtain a visually significant feature map corresponding to the pseudo-color image.

[0088] In one embodiment, the color saliency feature of the corresponding pixel point is obtained in S32, and the specific formula is:

[0089]

[0090] Where S(x,y) is the color saliency feature of pixel (x,y), C u is the color value of the pixel (x, y), which is the u-th color value in the pseudo-color image, C v is the vth color value in the pseudo-color image, n is the total number of color values ​​in the pseudo-color image, u,v∈1,2,…,n, p v is the vth color value C in the pseudo-color image v Probability of occurrence.

[0091] In one embodiment, the multi-scale gradient feature of the corresponding pixel point is obtained in S33, and the specific formula is:

[0092]

[0093] in,

[0094]

[0095] Where G(x,y) is the multi-scale gradient feature of the pixel point in the xth row and yth column in the pseudo-color image, G h (x, y) is the horizontal gradient feature of the pixel at the xth row and yth column in the pseudo-color image, G v (x,y) is the vertical gradient feature of the pixel in the xth row and yth column in the pseudo-color image, I(x,y+1) refers to the pixel value of the pixel in the xth row and y+1th column in the pseudo-color image, I(x,y-1) refers to the pixel value of the pixel in the xth row and y-1th column in the pseudo-color image, I(x+1,y) refers to the pixel value of the pixel in the xth row and yth column in the pseudo-color image, and I(x-1,y) refers to the pixel value of the pixel in the x-1th row and yth column in the pseudo-color image.

[0096] In one embodiment, the visually based saliency feature map is obtained in S34, and the specific formula is:

[0097]

[0098] Where V(x,y) is the visual saliency feature map of the pixel in the x-th row and y-th column in the pseudo-color image, S(x,y) is the color saliency feature map of the pixel (x,y), and G(x,y) is the multi-scale gradient feature of the pixel in the x-th row and y-th column in the pseudo-color image.

[0099] Specifically, the pseudo-color image is subjected to histogram contrast calculation and gradient feature extraction, and feature map fusion is performed to obtain a visually significant feature map. The process is as follows:

[0100] 1) Convert the pseudo-color image I from RGB space to Lab space to obtain the pixel value corresponding to each pixel in the pseudo-color image. The pixel value of each pixel in the pseudo-color image I is represented by F = [L, a, b], where L represents brightness, a represents the component from green to red, and b represents the component from blue to yellow;

[0101] 2) Perform color histogram statistics on the pseudo-color image. Specifically, perform color statistics on each pixel (x, y) in the pseudo-color image I, and then calculate the saliency features of the pixels with the same color to obtain the color saliency features of the corresponding pixels:

[0102]

[0103] Where S(x,y) is the color saliency feature of pixel (x,y), C u is the color value of the pixel (x, y), which is the u-th color value in the pseudo-color image, C v is the vth color value in the pseudo-color image, n is the total number of color values ​​in the pseudo-color image, u,v∈1,2,…,n, p v is the vth color value C in the pseudo-color image v Probability of occurrence.

[0104] 3) Calculate the horizontal gradient feature G for each pixel (x, y) in the pseudo-color image I h (x,y), the specific formula is:

[0105]

[0106] Where G h (x,y) is the horizontal gradient feature of the pixel in the xth row and yth column in the pseudo-color image, I(x,y+1)

[0107] Refers to the pixel value of the pixel at the xth row and y+1th column in the pseudo-color image, and I(x,y-1) refers to the pixel value of the pixel at the xth row and y-1th column in the pseudo-color image. Specifically, the horizontal gradient feature is calculated by subtracting the pixel values ​​of the pixels before and after the current column with the current column as a reference. It should be noted that when the current column is the first column, the horizontal gradient feature of the current column is calculated by subtracting the pixel values ​​of the pixels in the first two columns (the first and second columns). When the current column is the last column, the horizontal gradient feature is calculated by subtracting the pixel values ​​of the pixels in the last two columns (the last column and the second-to-last column).

[0108] 4) Calculate the vertical gradient feature G for each pixel (x, y) in the pseudo-color image I v (x,y), the specific formula is:

[0109]

[0110] Where G v (x,y) is the vertical gradient feature of the pixel in the xth row and yth column of the pseudo-color image, I(x+1,y) refers to the pixel value of the pixel in the x+1th row and yth column of the pseudo-color image, and I(x-1,y) refers to the pixel value of the pixel in the x-1th row and yth column of the pseudo-color image. That is, when calculating the vertical gradient feature, the current row is used as a reference and the pixel values ​​of the pixels in the previous and next rows of the current row are subtracted. It should be noted that when the current row is the first row, the horizontal gradient feature of the current row is calculated by subtracting the pixel values ​​of the pixels in the first two rows (the first and second rows). When the current row is the last row, the pixel values ​​of the pixels in the last two rows (the last row and the second to last row) are subtracted.

[0111] 5) Fusion of horizontal gradient features G h (x,y) and vertical gradient features G v (x, y), and obtain the multi-scale gradient feature G(x, y). The specific formula is:

[0112]

[0113] Where G(x,y) is the multi-scale gradient feature of the pixel at the xth row and yth column in the pseudo-color image.

[0114] 6) The color saliency feature is integrated with the multi-scale gradient feature to obtain the visual saliency feature. The specific formula is:

[0115]

[0116] Where V(x,y) is the visual saliency feature of the pixel at the xth row and yth column in the pseudo-color image.

[0117] 7) Traverse all pixels in the pseudo-color image and process through steps S32 to S34 to obtain a visually significant feature map corresponding to the pseudo-color image.

[0118] In one embodiment, a total variation filter is performed on the visual salient feature map V to obtain a background image B. The visual salient feature map V is then differentiated from the obtained background image B to obtain a preliminary anomaly detection map based on visual saliency:

[0119] S visual =(VB) 2

[0120] Where S visual is the preliminary anomaly detection map based on visual saliency, and B is the background image.

[0121] In one embodiment, in S4, the spectral angular distance and the spectral Euclidean distance are calculated based on the hyperspectral image samples and then fused. The calculation formula of the spectral angular distance is specifically:

[0122]

[0123] Where S SAD (A p ,A q ) is the spectral angle distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, A p is the spectral vector of the pth pixel in the hyperspectral image sample of the liquid to be detected, A q is the spectral vector of the qth pixel in the hyperspectral image sample of the liquid to be detected, and N is the number of pixels in the hyperspectral image of the liquid to be detected.

[0124] In one embodiment, in S4, the spectral angular distance and the spectral Euclidean distance are calculated and fused according to the hyperspectral image samples. The calculation formula of the spectral Euclidean distance is specifically:

[0125]

[0126] Where S SED (A p ,A q ) is the spectral Euclidean distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, A p is the spectral vector of the pth pixel in the hyperspectral image sample of the liquid to be detected, A q is the spectral vector of the qth pixel in the hyperspectral image sample of the liquid to be detected, N is the number of pixels in the hyperspectral image of the liquid to be detected, B is the number of bands in the hyperspectral image of the liquid to be detected, and k = 1, 2, …, B.

[0127] Specifically, the spectral angular distance and spectral Euclidean distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the drug to be detected are calculated, and the spectral angular distance and spectral Euclidean distance are fused to obtain the spectral saliency feature map based on the dual spectral attention mechanism:

[0128]

[0129] in,

[0130]

[0131] Where S spectral is the spectral saliency feature map based on the dual spectral attention mechanism corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, S SAD (A p ,A q ) is the spectral angle distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, S SED (A p ,A q ) is the spectral Euclidean distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, A p is the spectral vector of the pth pixel in the hyperspectral image sample of the liquid to be detected, A q is the spectral vector of the qth pixel in the hyperspectral image sample of the liquid to be detected, N is the number of pixels in the hyperspectral image of the liquid to be detected, p,q∈1,2,…,N, B is the number of bands in the hyperspectral image of the liquid to be detected, k=1,2,…,B.

[0132] In one embodiment, the background information of the preliminary abnormality detection image is suppressed by using the spectral significant feature map, which can further suppress the background and noise information in the preliminary abnormality detection image, while reducing the impact of the bottleneck of the liquid bottle and the liquid level line of the liquid, thereby realizing the detection of weak foreign matter in the oral liquid. Specifically, the preliminary abnormality detection map S of the hyperspectral image sample of the liquid to be detected is converted into visual and spectral significance feature map S spectral Perform multiplicative transformation fusion to obtain the final foreign body detection image S corresponding to the hyperspectral image sample of the liquid to be detected:

[0133]

[0134] Where S is the final foreign body detection image corresponding to the hyperspectral image sample of the liquid to be detected, S visual is the preliminary abnormality detection map of the drug solution to be detected based on visual saliency, S spectral It is the spectral saliency feature map of the drug solution to be detected based on the dual spectral attention mechanism.

[0135] The above-mentioned hyperspectral liquid foreign body detection method that integrates vision and spectral saliency extracts three bands with the largest information entropy from the hyperspectral image samples of the liquid to be tested and converts them into pseudo-color images. This method not only retains the most important spectral features, but also enables the application of saliency detection algorithms to hyperspectral data. By fusing the color saliency features and multi-scale gradient features extracted from the pseudo-color images, it not only effectively extracts the color, contour, and texture information in the pseudo-color images, but also weakens the influence of noise. The total variation filter is used to filter the preliminary anomaly detection map, effectively reducing the background interference of the liquid bottle. The spectral angular distance and spectral Euclidean distance are calculated and fused for the hyperspectral image samples of the liquid to be tested, effectively extracting dual spectral saliency features. The spectral saliency feature map is multiplicatively transformed and fused with the preliminary anomaly detection map, further suppressing the background and noise information in the preliminary anomaly detection map. At the same time, the influence of the bottleneck of the liquid bottle and the liquid level line of the liquid are eliminated. The method can quickly and accurately detect weak foreign bodies in the liquid to be tested, realizing non-destructive testing and quality control of the liquid.

[0136] The above is a detailed introduction to the hyperspectral liquid foreign body detection method that integrates vision and spectral significance provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A hyperspectral method for detecting foreign matter in liquid medicine by integrating vision and spectral significance, characterized in that: The method comprises: S1, capturing a hyperspectral original image of a to-be-detected medicinal liquid, and constructing a hyperspectral image sample set of the to-be-detected medicinal liquid based on the hyperspectral original image; S2. Randomly select a hyperspectral image sample from the hyperspectral image sample set, select a preset number of bands from the hyperspectral image sample using a sorting band selection method based on information entropy, and convert the preset number of bands into a pseudo-color image corresponding to the hyperspectral image sample; S2, selecting a preset number of bands from the hyperspectral image sample using a sorting band selection method based on information entropy, and converting the preset number of bands into a pseudo-color image corresponding to the hyperspectral image sample, specifically includes: S21, calculating the probability density of each band in the hyperspectral image sample; S22, calculating the information entropy of the corresponding band in the hyperspectral image sample according to the probability density; S23, arranging the bands in the hyperspectral image sample in descending order according to the information entropy, selecting a preset number of bands as three primary color bands, and using the three primary color bands as the pseudo-color image corresponding to the hyperspectral image sample; S3, performing histogram contrast calculation and gradient feature extraction on the pseudo-color image, and performing feature fusion to obtain a visually significant feature map, and performing total variation filtering on the visually significant feature map to obtain a preliminary anomaly detection map; S3, performing histogram contrast calculation and gradient feature extraction on the pseudo-color image, and performing feature fusion to obtain a visually significant feature map, specifically includes: S31, converting the pseudo color image from RGB space to Lab space, and converting the RGB component of each pixel in the pseudo color image into L, a, b components accordingly; S32, performing color histogram statistics on the pseudo-color image, calculating the significant features of pixels of the same color, and obtaining color significant features of corresponding pixels; S33, calculating a horizontal gradient feature and a vertical gradient feature for each pixel in the pseudo-color image, and fusing the horizontal gradient feature and the vertical gradient feature to obtain a multi-scale gradient feature of the corresponding pixel; S34, fusing the color saliency feature with the multi-scale gradient feature to obtain a visually significant feature of the corresponding pixel point; S35, traversing all pixels in the pseudo-color image, and processing through steps S32 to S34 to obtain a visually significant feature map corresponding to the pseudo-color image; S4. Calculate the spectral angular distance and the spectral Euclidean distance based on the hyperspectral image samples and fuse them to obtain a spectral saliency feature map based on a dual spectral attention mechanism. Use the spectral saliency feature map to suppress background information of the preliminary anomaly detection map to obtain a foreign body detection map. S5. Traverse all hyperspectral image samples in the hyperspectral image sample set, and obtain a plurality of foreign matter detection images accordingly after processing in steps S2 to S4, thereby realizing foreign matter detection of the liquid medicine to be detected.

2. The hyperspectral liquid medicine foreign body detection method integrating vision and spectral significance as claimed in claim 1, characterized in that: In S1, a hyperspectral image sample set of the liquid to be detected is constructed based on the hyperspectral original image. The hyperspectral image sample set can be expressed as: ； Where, is the sample set of hyperspectral images of the liquid medicine to be detected, is the sample set of hyperspectral images of the liquid to be detected Middle Hyperspectral image samples of the liquid to be tested, i=1,2,…,m, m is the number of samples in the hyperspectral image sample set of the liquid to be tested, , is the number of pixels of the hyperspectral image of the liquid to be detected, is the number of bands of the hyperspectral image of the liquid to be detected, is the sample set of hyperspectral images of the liquid to be detected Middle The foreign body labels corresponding to the hyperspectral image samples of the liquid to be detected.

3. The hyperspectral liquid medicine foreign body detection method integrating vision and spectral significance as claimed in claim 2, characterized in that: In the step S22, the information entropy of the corresponding band in the hyperspectral image sample is calculated according to the probability density. The calculation formula of the information entropy is specifically as follows: ； in, ; Where, For the The information entropy of the band, For the The probability density of each band, w is the gray level of each band image pixel, is the grayscale color space, For the The grayscale histogram of the band, , is the number of bands of the hyperspectral image of the liquid to be detected, is the number of pixels in the hyperspectral image of the liquid medicine to be detected.

4. The hyperspectral liquid medicine foreign body detection method integrating vision and spectral significance as claimed in claim 3, characterized in that: The color saliency feature of the corresponding pixel point is obtained in S32, and the specific formula is: ； Where, Pixel The color saliency characteristics of Pixel The color value of , that is, the u-th color value in the pseudo-color image, is the vth color value in the pseudo-color image, is the total number of color values ​​in the pseudo-color image, , is the vth color value in the pseudo-color image Probability of occurrence.

5. The hyperspectral liquid medicine foreign body detection method integrating vision and spectral significance as claimed in claim 4, characterized in that: In S33, the multi-scale gradient features of the corresponding pixel points are obtained, and the specific formula is: ； in, ; ； Where, The pseudo-color image Rank Multi-scale gradient features of column pixels, The pseudo-color image Rank The horizontal gradient features of the column pixels, The pseudo-color image Rank Vertical gradient features of column pixels , Refers to the pixel value of the pixel at the xth row and y+1th column in the pseudo-color image. Refers to the pixel value of the pixel at the xth row and y-1th column in the pseudo-color image. Refers to the pseudo-color image Rank The pixel value of the column pixel, Refers to the pseudo-color image Rank The pixel value of the column pixel.

6. The hyperspectral liquid medicine foreign body detection method integrating vision and spectral significance as claimed in claim 5, characterized in that: In S34, the visually significant features of the corresponding pixel points are obtained, and the specific formula is: ； Where, The pseudo-color image Rank Column pixels are based on visual saliency features, Pixel The color saliency characteristics of The pseudo-color image Rank Multi-scale gradient features of column pixels.

7. The hyperspectral liquid medicine foreign body detection method integrating vision and spectral significance as claimed in claim 6, characterized in that: In S4, the spectral angular distance and the spectral Euclidean distance are calculated and fused according to the hyperspectral image samples. The calculation formula of the spectral angular distance is specifically: ， ； Where, is the spectral angle distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, is the first The spectral vector of pixels, is the first The spectral vector of pixels, is the number of pixels in the hyperspectral image of the liquid medicine to be detected.

8. The hyperspectral liquid medicine foreign body detection method integrating vision and spectral significance as claimed in claim 7, characterized in that: In S4, the spectral angular distance and the spectral Euclidean distance are respectively calculated based on the hyperspectral image samples and are fused. The calculation formula of the spectral Euclidean distance is specifically: ， ； Where, is the spectral Euclidean distance between the spectral vectors corresponding to the pixel points in the hyperspectral image sample of the liquid to be detected, is the first The spectral vector of pixels, is the first The spectral vector of pixels, is the number of pixels of the hyperspectral image of the liquid to be detected, is the number of bands of the hyperspectral image of the liquid to be detected, .

Images