Vision-based traditional Chinese medicinal material defect detection method

Abstract

The invention relates to the technical field of traditional Chinese medicinal material defect detection, in particular to a traditional Chinese medicinal material defect detection method based on vision, which comprises the following steps: regularly acquiring a time sequence image of a traditional Chinese medicinal material sample, acquiring a multi-view image, carrying out pixel alignment on the time sequence image, and carrying out structured organization on the multi-view image according to a shooting direction to associate camera parameters; and generating a registration time sequence image sequence and a multi-view image set. According to the method, the time sequence images of the traditional Chinese medicine samples are collected regularly, pixel alignment is carried out, image offset caused by environment illumination fluctuation and equipment jitter is eliminated, and time-space consistency of dynamic variable quantity calculation is ensured. Structured organization is carried out on multi-view-angle images according to shooting directions, camera parameters are associated, a geometric constraint relation between view angles is established, and the problem that three-dimensional reconstruction precision is insufficient due to view angle isolation in a traditional method is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of Chinese herbal medicine defect detection, and particularly to a vision-based Chinese herbal medicine defect detection method. Background Art

[0002] The vision-based Chinese herbal medicine defect detection method is used to automatically identify physical defects (such as moisture absorption deformation, oil seepage patches, and insect holes) and abnormal color regions on the surface of Chinese herbal medicines through computer vision technology, and to classify defect types and evaluate the severity.

[0003] In the prior art, the matching between viewpoints depends on manual selection of feature points, and the accuracy of the reconstructed three-dimensional model is affected by the operator's experience, making it difficult to accurately reflect the tiny depressions or protrusions on the surface. Feature extraction focuses on the color and texture dimensions, ignoring the quantitative analysis of geometric deformation parameters, and it is difficult to distinguish the differences between similar textures such as insect holes and natural wrinkles. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of the present invention is to solve the deficiencies in the prior art and propose a vision-based Chinese herbal medicine defect detection method.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions. The vision-based Chinese herbal medicine defect detection method includes the following steps: Regularly collect the sequential images of Chinese herbal medicine samples, and collect multi-viewpoint images. Align the pixels of the sequential images, and structurally organize and correlate the camera parameters for the multi-viewpoint images according to the shooting directions to generate a registered sequential image sequence and a multi-viewpoint image set; Based on the registered sequential image sequence, calculate the color value difference between corresponding pixel positions in adjacent frame images in the sequence to obtain a per-frame change dataset. Based on the per-frame change dataset, cumulatively calculate the total change value at each pixel position along the time dimension, and screen to obtain the dynamic change feature map of the Chinese herbal medicine surface; Based on the multi-viewpoint image set, obtain the sparse three-dimensional point cloud and the camera pose by matching the homologous feature points between different viewpoint images and combining the camera parameters. Based on the sparse three-dimensional point cloud and the camera pose, connect the adjacent points in the point cloud to construct triangular mesh patches and establish a three-dimensional model of the sample to be measured; Based on the dynamic change feature map of the Chinese herbal medicine surface and the three-dimensional model of the sample to be measured, analyze whether the pattern of the changing region in the dynamic change feature map of the Chinese herbal medicine surface conforms to the dynamic defect characteristics of moisture absorption or oil seepage to obtain a multi-dimensional defect index set. Based on the multi-dimensional defect index set, output whether the Chinese herbal medicine sample is qualified or the defect category, and establish a determination of the Chinese herbal medicine defect category.

[0006] Preferably, the steps for obtaining the registered sequential image sequence and the multi-viewpoint image set are: Timely collect the time-series images and multi-view images of Chinese herbal medicine samples of the same batch. Perform pixel alignment processing on the time-series images. By extracting the SIFT feature points of adjacent frame images and performing two-way matching, calculate the spatial Euclidean distance of the matching point pairs. Use the RANSAC algorithm to eliminate the outlier matching point pairs, solve the affine transformation matrix parameters and apply them to the entire image to generate a registered time-series image sequence; Based on the registered time-series image sequence and the multi-view images, structurally organize the multi-view images, analyze the focal length, principal point coordinates, and radial distortion coefficients in the camera parameter files of each image, establish the mapping relationship between the azimuth angle, pitch angle, and camera parameters, and generate a multi-view image set; Based on the registered time-series image sequence and the multi-view image set, verify the effectiveness of time-series alignment by calculating the SSIM structural similarity between adjacent frames in the time-series sequence, and generate a registered time-series image sequence and a multi-view image set.

[0007] Preferably, the steps for obtaining the frame-by-frame change amount data set are as follows: Based on the registered time-series image sequence, extract the RGB three-channel color values at each pixel position in adjacent frame images, establish the current frame color value matrix and the previous frame color value matrix, and generate a group of adjacent frame color value matrices; Based on the group of adjacent frame color value matrices, calculate the color value difference amount at the corresponding pixel positions; Based on the color value difference amount, traverse all pixel positions and frame sequence indexes, stack and arrange the color value difference amounts in the frame sequence, and generate a frame-by-frame change amount data set.

[0008] Preferably, the steps for obtaining the dynamic change feature map of the Chinese herbal medicine surface are as follows: Based on the frame-by-frame change amount data set, extract the color value difference amount sequence at each pixel position in all frames, establish a set of time-series change amounts at the pixel positions, and generate pixel-level time cumulative input data; Based on the pixel-level time cumulative input data, calculate the total change amount value at each pixel position; Based on the total change amount value, use the sliding window method to calculate the mean and standard deviation of the total change amount in the neighborhood around each pixel, set a dynamic threshold, and screen the pixel regions that meet the condition that the total change amount value is greater than the dynamic threshold to generate a dynamic change feature map of the Chinese herbal medicine surface.

[0009] Preferably, the steps for obtaining the sparse three-dimensional point cloud and camera pose are as follows: Based on the multi-view image set, SIFT feature points are extracted from every two view images and feature descriptors are generated. Bidirectional matching is performed by calculating the Hamming distance between the feature descriptors. The RANSAC algorithm is used to fit the fundamental matrix and remove the outlier matching point pairs with projection errors exceeding 0.5 pixels, generating a set of corresponding feature point matching pairs between the multi-view images; Based on the set of corresponding feature point matching pairs, the fundamental matrix constraint equation is constructed with the focal length, principal point coordinates, and radial distortion coefficient in the camera parameters. The camera rotation matrix and translation vector are solved by singular value decomposition of the essential matrix. At the same time, the linear triangulation method is used to calculate the initial three-dimensional space coordinates of the feature point pairs, generating the camera external parameter estimation results and the sparse three-dimensional point coordinate set; Based on the camera external parameter estimation results and the sparse three-dimensional point coordinate set, the rotation matrix, translation vector, and three-dimensional point coordinates are iteratively adjusted with the reprojection error as the loss function until the average reprojection error converges within 0.3 pixels. The abnormal three-dimensional points with residuals exceeding 1.0 pixel after optimization are removed, generating the sparse three-dimensional point cloud and the camera pose.

[0010] Preferably, the steps for obtaining the three-dimensional model of the sample to be measured are as follows: Based on the sparse three-dimensional point cloud and the camera pose, the optical center distances, the average three-dimensional point reprojection error, and the surface curvature change rate gradient are extracted between adjacent camera views, establishing an interpolation credibility evaluation parameter group; Based on the interpolation credibility evaluation parameter group, the dynamic connection distance threshold is calculated; Based on the dynamic connection distance threshold, the wavefront propagation algorithm is used to perform region growing on the point cloud, preferentially connecting adjacent point pairs with a distance less than the dynamic connection distance threshold and a curvature difference less than 0.1, generating a set of triangular mesh patches with adaptive density and establishing the three-dimensional model of the sample to be measured.

[0011] Preferably, the steps for obtaining the multi-dimensional defect index set are as follows: Based on the dynamic change feature map of the Chinese herbal medicine surface, the pixel point set of all change regions is extracted. The judgment conditions for the moisture absorption defect feature are set as: the color value rising rate ≥ 15% / second and the region area expansion rate ≥ 3% / second within 5 consecutive frames; the judgment conditions for the oiliness defect feature are set as: the texture direction consistency coefficient ≤ 0.3 and the region brightness mean descent gradient ≥ 20 lumens / second; Based on the three-dimensional model of the sample to be measured and the three-dimensional model of the same grade and specification level, rough alignment is performed with the bottom surface of the model as the reference plane. The model pose is iteratively adjusted until the average Euclidean distance between the top of the medicinal material and the root bifurcation point is less than 0.5 mm and the maximum single-point deviation does not exceed 1.2 mm, generating a three-dimensional model comparison group after spatial registration; Based on the three-dimensional model comparison group after spatial registration, the surface mesh is divided into detection units of 10 mm × 10 mm. The surface curvature difference, normal vector deflection angle, and volume expansion rate are synchronously calculated within each detection unit. The porosity change amount within the detection unit is superimposed on the moisture absorption defect area, and the surface gloss attenuation rate is superimposed on the oiling defect area to generate a multi-dimensional defect index set.

[0012] Preferably, the obtaining steps of the traditional Chinese medicine defect category determination are as follows: Based on the multi-dimensional defect index set, if the same detection unit simultaneously meets the conditions of moisture absorption defect candidate and oiling defect candidate, it is preferentially determined as a moisture absorption defect; if the defect candidate category distance between different detection units ≤ 5 mm, there is spatial adjacency, and they are merged into a composite defect area and upgraded to a serious defect category to generate a corrected defect category determination result set; Based on the corrected defect category determination result set, the area ratio and severity level of all defect areas are statistically analyzed. If the total area ratio of the moisture absorption defects ≥ 5% or there is a serious defect category area, the sample is determined to be unqualified; if the total defect area ratio < 5% and there is no serious defect category, a qualified label is output to generate the traditional Chinese medicine defect category determination.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The present invention captures the sequential images of traditional Chinese medicine samples at regular intervals and performs pixel alignment to eliminate image offsets caused by environmental light fluctuations and equipment jitters, ensuring the spatio-temporal consistency of dynamic change amount calculation. The multi-view images are structurally organized according to the shooting azimuth and associated with camera parameters to establish geometric constraint relationships between views, solving the problem of insufficient three-dimensional reconstruction accuracy caused by isolated views in traditional methods. Based on the registered sequential image sequence, the pixel color differences between adjacent frames are calculated, and the total change amount is accumulated along the time dimension to generate a surface dynamic change feature map, which can capture the moisture absorption and swelling gradual change process and the oiling area diffusion path that are difficult to identify by traditional static detection. Combining multi-view homologous feature point matching and camera parameter optimization, a sparse three-dimensional point cloud is constructed, and a surface geometric deformation model is established through neighboring point triangulation to enhance the quantitative characterization ability of the depth of wormhole holes and the concave and convex features of the oiling area. By fusing the dynamic change map with the spatial deformation parameters of the three-dimensional model, multi-dimensional indexes such as color change rate, volume expansion rate, and curvature difference are synchronously analyzed, and a composite determination mechanism based on logical rules and spatial distribution characteristics is established to improve the recognition accuracy of composite defects and reduce the missed detection risk caused by misjudgment of a single index. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the steps of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0016] See also Figure 1 The present invention provides a technical solution, a method for detecting defects in Chinese medicinal materials based on vision, comprising the following steps: Collect time-series images of Chinese medicinal materials samples at regular intervals, collect multi-view images, align the pixels of the time-series images, structurally organize and associate camera parameters of the multi-view images according to the shooting orientation, and generate a registered time-series image sequence and a multi-view image set; Based on the registration time-series image sequence, the color value difference between the corresponding pixel positions of adjacent frame images in the sequence is calculated to obtain a frame-by-frame variation data set. Based on the frame-by-frame variation data set, the total variation value of each pixel position is accumulated along the time dimension to screen out the dynamic variation feature map of the surface of Chinese medicinal materials; Based on a multi-view image set, by matching the feature points of the same name between images of different viewpoints and combining the camera parameters, a sparse 3D point cloud and camera posture are obtained. Based on the sparse 3D point cloud and camera posture, adjacent points in the point cloud are connected to construct a triangular mesh patch to establish a 3D model of the sample to be tested. Based on the dynamic change characteristic map of the Chinese medicinal materials surface and the three-dimensional model of the sample to be tested, it is analyzed whether the pattern of the change area in the dynamic change characteristic map of the Chinese medicinal materials surface conforms to the dynamic defect characteristics of moisture absorption or oil overflow, and a multi-dimensional defect indicator set is obtained. Based on the multi-dimensional defect indicator set, the qualification or defect category of the Chinese medicinal materials sample is output, and the defect category judgment of the Chinese medicinal materials is established.

[0017] The steps for obtaining the registration time-series image sequence and multi-view image set are: Time-series images and multi-view images of the same batch of Chinese medicinal materials samples are collected regularly, and pixel alignment is performed on the time-series images. The SIFT feature points of adjacent frame images are extracted and bidirectionally matched, and the spatial Euclidean distance of matching point pairs is calculated. The RANSAC algorithm is used to eliminate outlier matching point pairs, and the affine transformation matrix parameters are solved and applied to the entire image to generate a sequence of registered time-series images. Based on the registration of time-series image sequences and multi-view images, the multi-view images are structured and organized, the focal length, principal point coordinates and radial distortion coefficient in the camera parameter file of each image are parsed, the mapping relationship between azimuth angle, pitch angle and camera parameters is established, and a multi-view image set is generated; Based on the registered temporal image sequence and multi-view image set, the effectiveness of temporal alignment is verified by calculating the SSIM structural similarity of adjacent frames in the temporal sequence, and the registered temporal image sequence and multi-view image set are generated.

[0018] Specifically, after timing acquisition of the sequential images and multi-view images of the Chinese herbal medicine samples in the same batch, pixel alignment processing is performed on the obtained sequential images. In this process, each pair of adjacent image frames in the sequential image sequence is traversed first. For example, it is denoted as the frame and the frame. The key feature points and their corresponding 128-dimensional feature descriptors in the two frames of images are respectively extracted by using the Scale-Invariant Feature Transform (SIFT) algorithm. Then, bidirectional matching is performed on the feature descriptors extracted from the two frames of images. Specifically, for each feature point in the frame, the nearest neighbor and the second-nearest neighbor feature points are searched in the frame according to the Euclidean distance of the feature descriptors, and the ratio of the nearest neighbor distance to the second-nearest neighbor distance is calculated. If this ratio is less than the preset distance ratio threshold (for example, set to 0.75, this value is an empirical value widely used in the SIFT matching strategy and is applicable to distinguish good matches from ambiguous matches), it is considered a potential match. Conversely, for each feature point in the frame, the same search and ratio test are performed in the frame. Only when two feature points are the best matches for each other, they are regarded as a valid initial matching point pair. Subsequently, the spatial Euclidean distance of all these initial matching point pairs in their respective image coordinate systems is calculated. This distance is not used for screening, but as input information or an auxiliary judgment basis for subsequent algorithms. Immediately afterwards, the Random Sample Consensus (RANSAC) algorithm is used to eliminate the outlier matching point pairs caused by repeated textures or short-term occlusions. The execution process of the RANSAC algorithm is as follows: Randomly select 3 (the minimum number of point pairs required for affine transformation) initial matching point pairs, estimate an affine transformation matrix based on these 3 point pairs, and use this matrix to transform all the matching points in the frame to the coordinate system of the frame, calculate the pixel distance between the transformed points and the corresponding matching points in the frame, and regard the point pairs with a distance less than a specific pixel threshold as "inliers". This pixel threshold is set according to the image resolution and the expected maximum pixel drift amount. For example, for an image with a resolution of 1920x1080, the expected maximum misalignment between frames is 2 pixels, then the threshold can be set to 1.5 pixels. This value is obtained by testing the stability of the imaging system, observing the maximum drift range of feature points in the stable state and taking 75% of its upper limit. Repeat this process several times (for example, set the number of iterations to 1000 times, which is calculated according to the number of times required to ensure a high probability of selecting a sample set all of which are inliers, usually based on the estimated value of the inlier rate), and select the set of affine transformation matrices that produce the largest number of inliers as the optimal solution. Finally, apply the obtained optimal affine transformation matrix parameters (including 6 parameters of rotation, scaling, translation, and shear) to the For the entire set of pixel coordinates of a frame, the gray value or color value of a pixel at a non-integer coordinate after transformation is calculated by means of bilinear interpolation, generating a frame of image that is aligned with the pixels of the th frame. This alignment process is repeated for all adjacent frames in the entire time-series image sequence, ultimately generating a registered time-series image sequence.

[0019] Based on the registered time-series image sequence generated in the previous step and the multi-view images acquired, the unordered multi-view images need to be structurally organized. The specific operation is as follows: First, read the camera parameter file associated with each multi-view image. This file is usually in text format and records the internal parameters of the camera. By parsing the content of the file, key parameter values are extracted, including the focal lengths of the camera in the horizontal and vertical directions (denoted as and , usually in pixels), the coordinates of the principal point of the image (denoted as , representing the intersection point of the camera optical axis and the imaging plane, in pixels), and the coefficients describing the radial distortion of the lens (usually including at least and other low-order coefficients to correct the offset of pixel points towards the center or the edge due to the curvature of the lens). At the same time, the spatial orientation information of the camera when taking this multi-view image needs to be obtained, that is, the azimuth angle (the rotation angle around the vertical axis) and the elevation angle (the rotation angle around the horizontal axis). These angle information are usually directly recorded during acquisition by controlling the encoder of the camera pan-tilt head or robotic arm, or known according to the preset shooting point plan. For example, if the camera is installed on a rotating arm with a known radius and one image is taken every 10 degrees, then the azimuth angle is an integer multiple of 10 degrees, and the elevation angle is determined according to the shooting level. Then, a data structure is established, such as using a hash table or dictionary, associating each pair of obtained (azimuth angle, elevation angle) values as the key, and the set of ( ) parameters parsed from the corresponding camera parameter file as the value for associated storage. By traversing all multi-view images and their parameter files and orientation information, the construction of the mapping relationship is completed. In this way, the scattered multi-view images and their corresponding precise shooting parameters and spatial orientations are integrated to generate a structured set of multi-view images.

[0020] Based on the already obtained registered time-series image sequence and the structured set of multi-view images, in order to finally confirm the effectiveness of the alignment of the time-series images, a verification process needs to be performed. This process is achieved by calculating the structural similarity (SSIM) between each pair of adjacent frames in the registered time-series image sequence (for example, the th frame and the aligned th frame). The specific steps for calculating the SSIM value are as follows: The two images to be compared are divided into several overlapping or non-overlapping calculation windows (for example, using a size of A sliding window of pixels with a step size of 1 pixel. The selection of the window size is based on experience in balancing computational efficiency and local detail sensitivity. For each pair of windows and , calculate their pixel averages (representing brightness estimation ), pixel standard deviations (representing contrast estimation ), and covariances (representing structural similarity measure ), respectively. Then, use the following formula to calculate the SSIM value of the window: , where and are stability constants introduced to avoid the denominator approaching zero, represents the dynamic range of the image pixel values (e.g., for an 8-bit grayscale image, ), and are two small constants, implemented according to the SSIM standard, usually set to and . These values are empirical values selected based on the simulation of the perception characteristics of the human visual system. After calculating the SSIM values of all windows, take the average of all window SSIM values to obtain the mean SSIM (MSSIM) value between these two frames of images. Set an MSSIM threshold to judge whether the alignment is effective. The setting of this threshold is based on the evaluation of the actual alignment effect. For example, by observing the MSSIM value distribution of a large number of manually confirmed well-aligned image pairs, it is found that their values are generally higher than 0.95, while those of poorly aligned image pairs are significantly lower than this value. Considering the possible small real changes on the surface of the medicinal materials, a slightly lower threshold can be set, such as . If the MSSIM value of any adjacent frame pair calculated is greater than or equal to , it is considered that the alignment of the entire time series is effective. After passing this verification, formally generate and confirm the obtained registered time series image sequence and the multi-view image set for subsequent processing.

[0021] The steps for obtaining the per-frame change amount dataset are as follows: Based on the registered time series image sequence, extract the RGB three-channel color values at each pixel position in adjacent frame images, establish the current frame color value matrix and the previous frame color value matrix, and generate a group of adjacent frame color value matrices; Based on the group of adjacent frame color value matrices, calculate the color value difference amount at the corresponding pixel positions. The calculation formula is: ; where is the color value difference amount, is the current frame at the th row and the The RGB component values of column pixels, are the RGB component values at the corresponding positions in the previous frame; Based on the color value difference amount, traverse all pixel positions and frame sequence indices, and stack and arrange the color value difference amounts in the frame sequence to generate a per-frame change amount data set.

[0022] Specifically, based on the registration time sequence image sequence generated in the previous step, each pair of temporally adjacent image frames in the sequence needs to be processed. Specifically, set the currently processed frame as the frame (where starting from 2 until the total number of frames in the sequence ), its previous frame is the frame. Traverse each pixel position in the frame image. This position is uniquely determined by its row coordinate and column coordinate (where starting from 1 to the image height ), starting from 1 to the image width ). For each pixel position , read and record the intensity values of the red (Red), green (Green), and blue (Blue) color channels at this position in the frame image. Usually, these values are 8-bit unsigned integers, ranging from 0 to 255. Organize the RGB values of all pixels in the frame to form a data structure with a dimension of , which is called the current frame color value matrix and is denoted as . In the same way, read and record the RGB color values at the corresponding pixel position in the frame image, and organize these values into the previous frame color value matrix, denoted as . These two matrices and contain all pixel-level color information required for subsequent color difference calculations. Take these two matrices as a processing unit to generate an adjacent frame color value matrix group for calculating the color difference between frames.

[0023] Formula: , The advantage of the formula is that by calculating the Euclidean distance in the RGB color space, it can quantitatively and comprehensively reflect the overall change amplitude of the color of a single pixel point at two adjacent moments, not limited to the change in brightness. It is equally sensitive to the changes in the red, green, and blue channels, and is suitable for capturing the subtle or significant changes in the surface color of traditional Chinese medicine caused by moisture absorption (usually the color becomes darker) or oiling (usually the color becomes dull or shiny, resulting in reflection changes), providing a basic, pixel-by-pixel change metric for subsequent identification of these dynamic defects; Calculation process: The formula calculates the straight-line distance between two points in the three-dimensional RGB color space (representing the colors of the same pixel in the previous and next frames respectively), The calculation steps are as follows: 1. Calculate the square of the difference in the red channel: ; 2. Calculate the square of the difference in the green channel: ; 3. Calculate the square of the difference in the blue channel: ; 4. Add the three squared differences: ; 5. Take the square root of the added result: Substitute into an actual example: Calculate the color value difference between the 4th frame and the 5th frame at the pixel position (500, 800) : ; ; ; ; ; This result shows that at the pixel position (500, 800), a color change of about 7.348 occurred from the 4th frame to the 5th frame. This value itself has no absolute physical meaning, but it relatively represents the amplitude of the color change. The larger the value, the more significant the color change, and a value of 0 indicates that there is no color change at all. This calculated color value difference will be used as basic data to construct a per-frame change dataset required for subsequent analysis and is the key input information for identifying dynamic change regions.

[0024] Based on the color value difference representing the color change of each pixel between adjacent frames calculated in the previous step , it is necessary to integrate this change information scattered among different frame conversions. Specifically, first, for the entire registration time-series image sequence (a total of frame), calculate the color value difference of all pixel positions from the 1st frame to the 2nd frame, from the 2nd frame to the 3rd frame, ……, from the frame to the frame, which will generate a difference matrix with the same size as the image ( ), where the th matrix contains the color value differences of all pixels from the frame to the frame , then, to construct the required dataset, it is necessary to traverse all pixel positions in the image (where ranges from 1 to , ranges from 1 to ), for each fixed pixel position , extract its color value differences in all frame transitions, that is, obtain the sequence , this sequence records the frame-by-frame difference history of the color change of this pixel point over time. Then, organize the time series change amounts of all pixel positions together, and a three-dimensional data structure (such as a three-dimensional array) can be used, with dimensions (image height , image width , number of frame transitions ), where the stored value is the color value difference of the corresponding pixel at the corresponding frame transition moment, or use other equivalent data structures such as dictionaries or nested lists to collect these sets of color value differences arranged in frame order (or stacked in chronological order) to generate a structured dataset of frame-by-frame change amounts.

[0025] The steps to obtain the dynamic change characteristic map of the surface of traditional Chinese medicine materials are as follows: Based on the frame-by-frame change amount dataset, extract the color value difference sequence of each pixel position in all frames, establish a set of time series change amounts of pixel positions, and generate pixel-level time cumulative input data; Based on the pixel-level time cumulative input data, calculate the total change amount value of each pixel position, and the calculation formula is: ; Among them, is the total change amount value, is the color value difference of pixel in the th frame, is the variance of the color value differences of the entire image in the th frame, is the total number of frames; Based on the total change amount value, use the sliding window method to calculate around each pixel Mean of the total change within the neighborhood and the standard deviation , set a dynamic threshold , and filter the pixel regions where the total change value is greater than the dynamic threshold to generate a dynamic change feature map of the Chinese herbal medicine surface.

[0026] Specifically, based on the per-frame change amount dataset generated in the previous steps, which contains the color value difference amounts between adjacent frames for each pixel point, further processing is required to focus on the complete history of each pixel point's change over time. The operation process is as follows: traverse the dataset, and for each specific pixel position in the image (where is the row index is the column index), extract from the dataset all the color value difference amounts calculated at the frame transition moments corresponding to this pixel position (where represents the serial number of the frame transition, starting from the second transition, i.e., to the last transition , is the total number of frames), organize these extracted difference amounts in chronological order (i.e., according to the frame transition serial number ), for example, for pixel , its sequence may be , this sequence constitutes the set of time-series change amounts at this pixel position, which records the complete dynamic process of the color change at this point. Repeat this extraction and organization process for all pixel positions in the image , and gather together the sets of time-series change amounts of all pixels to form a structured data set. This set is the pixel-level time cumulative input data used for calculating the total change amount of each pixel in the next stage.

[0027] Formula: , the benefit of the formula is that it provides a method to measure the total cumulative color change amount of a pixel point over the entire observation time, and at the same time introduces a normalization factor based on the global change variance. By taking the absolute value of the color difference amount of each frame and accumulating it, the cumulativeness of the change is ensured, and the denominator term adjusts the contribution weight of the current frame difference amount according to the severity of the color change of the entire image at this frame transition moment (measured by the variance ). When the global change is severe (for example, due to sudden light changes resulting in being very large), the denominator increases, and the contribution of the single-point difference amount is suppressed. Conversely, when the global change is stable ( is small), the denominator approaches 1, and the contribution of the single-point difference amount is more significant. This design makes the calculated total change amount It can more prominently highlight the local pixel changes that are more abnormal compared to the global background changes during the same period, which helps to distinguish real local dynamics (such as defect development) from global interference; Explanation for obtaining formula parameters: The parameter represents the difference in color values of the pixels associated with the th frame. According to the context, this represents the difference in color values from the th frame to the th frame, that is, , where is obtained from the previously generated "per-frame change dataset". For , since there is no previous frame, it is defined as . For to to , directly read the corresponding pixel at the th frame conversion (i.e., ). For example, for the pixel , its change sequence from the 1st frame to the 10th frame ( to ) may be: [5.2, 3.1, 4.5, 6.0, 5.8, 4.9, 7.1, 6.5, 5.5], plus ; The parameter represents the variance of the difference in color values of the entire image associated with the th frame, that is, the variance of the difference in color values of all pixels from the th frame to the th frame conversion. For example, the calculated variance sequence of each frame conversion from the 1st frame to the 10th frame ( to ) is: [10.5, 8.2, 12.1, 15.0, 13.5, 11.8, 16.2, 14.8, 13.0], plus ; Calculation process: Calculate the total change value of the pixel , and use and the corresponding sequence and sequence to substitute into the calculation to obtain .

[0028] This result shows that the total color change of the pixel accumulated during the entire observation period (10 frames) and normalized by the global change variance is approximately 13.371.

[0029] Formula: , the advantage of the formula is that it defines a dynamic threshold based on local neighborhood statistical features, which is used to screen out significantly changed pixel regions from the total change value map, and calculates the local mean using a sliding window and the local standard deviation , making the threshold able to adapt to the baseline level and fluctuation amplitude of the background change amount in different regions of the image. Compared with the global threshold, it can better handle the situation of uneven background change, avoid missing detection in smooth change regions and false detection in violently changing regions, and is used to identify extreme values or outliers relative to the local environment.

[0030] Explanation of obtaining formula parameters: The parameter represents the total change value of all pixels within the neighborhood (window) centered on the pixel with a size of . It is necessary to extract the data of the corresponding window from the map containing the total change value of all pixels calculated in the previous step for calculation. The image boundary region is usually processed by padding zeros, mirroring, or repeating boundary pixels, etc., to ensure that the window is always in size. For example, if , then calculate the average value of the values of 5x5 = 25 pixels centered on ; The parameter represents the standard deviation of the total change values of all pixels within the same neighborhood. It measures the dispersion or volatility of the total change values within the neighborhood and also needs to extract data from the total change value map for calculation; The parameter is the side length (number of pixels) of the sliding window, which needs to be set in advance. Its selection affects the detection scale and the sensitivity to noise. A smaller is sensitive to small-scale changes but vulnerable to noise, while a larger has a good smoothing effect but may blur the boundaries of small features. Its value is usually set empirically according to the surface texture characteristics of the medicinal material and the expected defect size or optimized through experiments. For example, by testing different values (such as pixels) on sample images containing known-size defects (such as 5 mm in diameter), observing which size can best segment the defect region while suppressing noise, and determining a suitable value. In this example, is set; Substitute the parameters into the formula to calculate 17.3. This result shows that for the pixel And its neighborhood, whose dynamic threshold is calculated as 17.3. In the subsequent screening step, the total change value of the pixel itself (the neighborhood center value was 13.4 in the previous example) is compared with this threshold of 17.3. If , then the pixel is marked as part of the dynamically changing area, otherwise it is not marked. This calculation and comparison process is repeated for all pixels, and finally a binary image is obtained, where the pixels with a value of 1 constitute the identified dynamic change feature map of the surface of the traditional Chinese medicine material.

[0031] The steps for obtaining the sparse three-dimensional point cloud and camera pose are as follows: Based on the multi-view image set, SIFT feature points are extracted from every two view images and feature descriptors are generated. Bidirectional matching is performed by calculating the Hamming distance between the feature descriptors. The RANSAC algorithm is used to fit the fundamental matrix and remove the outlier matching point pairs with a projection error exceeding 0.5 pixels, generating a set of corresponding feature point matching pairs between multi-view images; Based on the set of corresponding feature point matching pairs and the focal length, principal point coordinates, and radial distortion coefficients in the camera parameters, a epipolar geometry constraint equation is constructed. The camera rotation matrix and translation vector are solved by singular value decomposition of the essential matrix. At the same time, the linear triangulation method is used to calculate the initial three-dimensional space coordinates of the feature point pairs, generating the estimation result of the camera external parameters and the set of sparse three-dimensional point coordinates; Based on the estimation result of the camera external parameters and the set of sparse three-dimensional point coordinates, the rotation matrix, translation vector, and three-dimensional point coordinates are iteratively adjusted with the reprojection error as the loss function until the average reprojection error converges within 0.3 pixels. The abnormal three-dimensional points with a residual error exceeding 1.0 pixel after optimization are removed, generating the sparse three-dimensional point cloud and camera pose.

[0032] Specifically, based on the multi-view image set generated in the previous step, it is first necessary to find the corresponding feature points between different-view images. The specific operation is as follows: Select any two different-view images from this set, apply the Scale-Invariant Feature Transform (SIFT) algorithm to each image, detect key points at different scales by constructing a Difference of Gaussian pyramid, and calculate a 128-dimensional feature descriptor for each key point. Next, match the sets of feature descriptors extracted from these two images. Measure the similarity of the descriptors by calculating the Hamming distance between the feature descriptors, and adopt a two-way matching strategy to filter reliable matches: For each feature point in the first image, find the matching point with the smallest Hamming distance in the second image, and vice versa. Only when two feature points are the best matches for each other are they regarded as a candidate matching pair. Subsequently, use the Random Sample Consensus (RANSAC) algorithm to estimate the fundamental matrix between these two images, and further eliminate the outlier matching point pairs that do not conform to the epipolar geometry constraint. The process of RANSAC is as follows: Randomly select 8 pairs (the minimum number of point pairs required to calculate the fundamental matrix) of candidate matching point pairs repeatedly, calculate a fundamental matrix model based on these 8 point pairs, and then use this model to verify all candidate matching point pairs. Calculate the distance of each point pair to its corresponding epipolar line (i.e., the projection error). If this distance is less than a preset pixel threshold, this point pair is regarded as an inlier. Here, the pixel threshold is set to 0.5 pixels. The basis for setting this value is that for a well-calibrated camera and accurate feature point extraction, the actual projection error of the matching point pairs is usually expected to be at the sub-pixel level. Setting 0.5 pixels is a relatively strict standard, aiming to filter out the large deviations caused by inaccurate feature localization or incorrect matching, and ensure that the matching point pairs used for subsequent calculations have high precision. This threshold is usually obtained by performing matching tests on image pairs with known geometric relationships, analyzing the distribution of inlier projection errors, and selecting a boundary value that can effectively distinguish inliers and outliers. The RANSAC algorithm will iterate (for example, set 1000 iterations or until the inlier set is stable) and select the fundamental matrix that can obtain the largest number of inliers as the optimal estimate. Finally, all the matching point pairs determined to be inliers by the optimal fundamental matrix model constitute the corresponding feature point matching pairs between these two view images. Repeat this process for all relevant image pairs in the multi-view image set to generate a set of corresponding feature point matching pairs between multi-view images.

[0033] Based on the set of corresponding feature point matching pairs between multi-view images obtained in the previous step, and combined with the camera intrinsics corresponding to each image that has been structured and stored in the multi-view image set previously (including the focal length , the principal point coordinates and the radial distortion coefficients ), start estimating the relative pose (rotation and translation) between cameras and the three-dimensional spatial positions of feature points. First, for any pair of perspective images with matching homologous feature points found (e.g., perspective 1 and perspective 2), using their camera intrinsics (denoted as ), and the fundamental matrix (denoted as ), through the relationship calculate the Essential Matrix, denoted as . Before applying this formula, it is necessary to undistort the two-dimensional image coordinates of the matching point pairs using the radial distortion coefficients in the camera parameters to obtain the corrected coordinates and . The Essential Matrix contains the rotation and translation information between the two camera coordinate systems and satisfies the epipolar constraint condition . Next, perform singular value decomposition on the Essential Matrix , that is , where is an orthogonal matrix, is a diagonal matrix. Theoretically, for the Essential Matrix, its singular value form should be . Four possible camera rotation matrices and translation vectors solutions can be deduced from the results of SVD (usually fixing the first camera pose as the identity matrix and the zero vector as the world coordinate system). To determine the unique correct solution, chirality checking is required, that is, using these four possible and camera intrinsics to construct projection matrices and perform three-dimensional reconstruction on at least one pair of homologous feature points, and check whether the reconstructed three-dimensional points are in front of both cameras (i.e., the depth value is positive). Only the set of that satisfies this condition is the physically possible solution. After selecting the correct , they constitute the external parameter estimation result of camera 2 relative to camera 1. At the same time, for all homologous feature point matching pairs that meet the epipolar constraint, using the linear triangulation method, and using the known projection matrices of the two cameras (constituted by the intrinsics and the solved extrinsics ) and the corresponding two-dimensional point coordinates , calculate the initial coordinates of each feature point pair in the three-dimensional space by solving an overdetermined linear equation system. Perform triangulation calculations on all matching point pairs, gather all the calculated and three-dimensional point coordinates to generate the initial camera external parameter estimation result and the sparse three-dimensional point coordinate set.

[0034] Based on the initial camera extrinsic parameter estimation results obtained in the previous step (including the rotation matrix and translation vector T) of each camera view relative to the reference view, as well as the corresponding sparse three-dimensional point coordinate set (including the initial three-dimensional spatial positions of each feature point ), joint optimization needs to be performed through bundle adjustment to improve the overall accuracy and consistency of the camera pose and three-dimensional point cloud. The core idea of bundle adjustment is to minimize the reprojection error, which is defined as follows: For a three-dimensional spatial point , using the intrinsic parameters and extrinsic parameters of a certain camera, project it back to the image plane of this camera to obtain the predicted two-dimensional projection coordinates . The pixel distance (usually the Euclidean distance) between this predicted coordinate and the two-dimensional feature point coordinate corresponding to the actually observed three-dimensional point is the reprojection error, denoted as . Bundle adjustment constructs a large-scale non-linear least squares problem, and its loss function is the sum of the squares of the reprojection errors of all observed feature points in all camera views: , where traverses all camera views, traverses all three-dimensional points, is the observed coordinate of the -th three-dimensional point in the -th camera, is a weight factor (usually 1, indicating whether the point is visible in the camera ). The optimization process simultaneously adjusts the rotation matrices , translation vectors of all cameras and the coordinates of all three-dimensional points through an iterative algorithm (such as the Levenberg-Marquardt algorithm) until the objective function converges. The convergence criterion is set as the average reprojection error of all valid observed points being less than 0.3 pixels. This 0.3-pixel threshold is set based on the experience of high-precision three-dimensional reconstruction tasks, aiming to achieve a high reconstruction accuracy. Usually, through the accuracy evaluation and comparison of the reconstruction results of multiple different scenes, a threshold that achieves a good balance between accuracy and computational cost is determined. When the average reprojection error satisfies this convergence condition, the optimization process stops. At this time, the optimized camera pose and three-dimensional point coordinates are obtained. Finally, an outlier removal is also required. Check the final reprojection error (also called the residual) of each three-dimensional point in all cameras that observe it. If a certain three-dimensional point If the optimized residual in at least one camera exceeds 1.0 pixel, the point is considered an abnormal 3D point (possibly caused by an initial matching error or a local minimum during the optimization process), and it is removed from the set of 3D points. This 1.0-pixel threshold is looser than the convergence threshold (0.3 pixel) and is used to remove points that still exhibit significant inconsistencies even after optimization. Its setting is also based on experience, aiming to remove obviously incorrect structural points while retaining most of the correctly reconstructed points. After optimization and removal, an accurate sparse 3D point cloud and camera poses are finally generated.

[0035] The steps for obtaining the 3D model of the sample to be measured are as follows: Based on the sparse 3D point cloud and camera poses, extract the optical center distance between adjacent camera views, the average reprojection error of 3D points, and the gradient of the surface curvature change rate, and establish an interpolation credibility evaluation parameter group. Based on the interpolation credibility evaluation parameter group, calculate the dynamic connection distance threshold, and the calculation formula is: ; Where, is the dynamic connection distance threshold, is the average distance between adjacent points in the sparse point cloud, is the optical center distance between adjacent cameras, is the average reprojection error of 3D points, is the gradient magnitude of the local curvature change rate of the point cloud; Based on the dynamic connection distance threshold, use the wavefront propagation algorithm to perform region growing on the point cloud, and preferentially connect adjacent point pairs with a distance less than the dynamic connection distance threshold and a curvature difference less than 0.1 to generate a set of triangular mesh patches with adaptive density and establish the 3D model of the sample to be measured.

[0036] Specifically, based on the sparse 3D point cloud optimized in the previous steps and the accurate poses of each camera (rotation matrix and translation vector ), in order to guide the subsequent process of generating a surface mesh from the sparse point cloud, it is necessary to first evaluate the interpolation credibility of different regions in the point cloud, which is achieved by calculating a set of parameters reflecting local data quality and geometric complexity. The specific operation is as follows: First, identify pairs of spatially adjacent camera views, which can be done by calculating the distances between all camera optical centers (given by the translation vector in the camera pose), and considering pairs with a distance less than a specific range (for example, less than 1.5 times the average camera spacing) as adjacent pairs. Then, for each pair of adjacent camera views, calculate the Euclidean distance between their optical centers, denoted as , this value reflects the length of the observed baseline. Then, all 3D points involved in the reconstruction of this pair of adjacent viewpoints are extracted, and the average reprojection error of these 3D points under these two specific viewpoints after being optimized by Bundle Adjustment is calculated, denoted as , this value reflects the consistency of the positioning accuracy of these points under these two viewpoints. A low error indicates high credibility. In addition, it is also necessary to evaluate the local geometric complexity of the point cloud. For each point in the sparse 3D point cloud, the local surface curvature at this point is estimated by analyzing the distribution of its k-nearest neighbors (for example, k = 10 or 15) (for example, calculating the principal curvature ), and further calculate how fast the curvature changes in space, that is, the gradient of the surface curvature change rate, and take its modulus length, denoted as , a high gradient modulus length means that the surface shape changes violently. Combining these three calculated parameters (the distance between the optical centers of adjacent cameras , the average reprojection error of 3D points , the gradient modulus length of the local curvature change rate of the point cloud ), organize them to form an interpolation credibility evaluation parameter group associated with different regions of the point cloud or adjacent viewpoint pairs.

[0037] Formula: , the benefit of the formula is that it dynamically calculates a connection distance threshold according to the quality and geometric characteristics of the local data, which is used to guide the subsequent surface mesh construction process, replacing the problem that using a global fixed threshold may lead to overly conservative connections in high-quality regions and overly aggressive connections in low-quality or complex regions. The numerator part is scaled based on the average point spacing of the point cloud , and the scaling factor considers the observation geometry (baseline ) and the reconstruction accuracy (reprojection error ). When the baseline length or error is small, a farther connection distance is allowed, indicating that the reconstruction in this region is relatively reliable; the denominator part then introduces a penalty for geometric complexity and observation conditions. When the surface curvature changes violently ( is large) and the observation baseline is long ( is large), this term increases, resulting in decreasing, so a more conservative connection strategy is adopted in regions with complex shapes or poor observation conditions to avoid crossing features or generating incorrect connections in sparse regions. The use of the function limits the growth amplitude of the denominator, making this penalty effect smoother, and overall realizes the adaptive adjustment of the connection distance; Instructions for obtaining formula parameters: The parameter represents the average spacing of neighboring points in the sparse point cloud, reflecting the overall or local density of the point cloud. Its calculation method is as follows: traverse each point in the point cloud and find its nearest neighbors (for example ), calculate the average distance from this point to these neighbors, and finally take the average of this average neighbor distance for all points to obtain the global . For example, when calculating a sparse point cloud containing 10,000 points, the average neighboring point spacing is obtained as millimeters; The parameter is the distance between the optical centers of adjacent camera views. This value is extracted from the "interpolation credibility evaluation parameter group" generated in the previous step and is related to the local area of the point cloud corresponding to the current calculation threshold. For example, the current area is mainly observed by a pair of cameras with an optical center distance of millimeters; The parameter is the average reprojection error (unit: pixel) of the three-dimensional points related to the above adjacent camera views. This value is also extracted from the "interpolation credibility evaluation parameter group" and reflects the accuracy of the positions of these three-dimensional points on the corresponding two-dimensional images. For example, the average reprojection error corresponding to this area is pixels; The parameter is the gradient magnitude of the surface curvature change rate of the point cloud in this local area. It quantifies how fast the surface curvature changes. This value is also extracted from the "interpolation credibility evaluation parameter group". First, the curvature of each point in the point cloud needs to be estimated (such as obtaining the normal vector and curvature through the principal component analysis of the neighborhood covariance matrix), and then the gradient magnitude of the curvature in space is calculated. For example, the of this area is calculated; Calculation process: Use the parameter values obtained above to calculate the dynamic connection distance threshold : millimeters, millimeters, pixels, mm ; Calculate the numerator: ; Calculate the terms in the denominator: ; Calculate the denominator: ; Calculate : ; The results show that in this specific point cloud region, based on the evaluation of point cloud density, observation geometry, reconstruction accuracy, and surface complexity, the calculated dynamic connection distance threshold is 25.29 mm. When constructing a triangular mesh using the wavefront propagation algorithm subsequently, only when the distance between two adjacent points is less than 25.29 mm and other conditions (such as curvature difference) are met will they be preferentially considered for connection into an edge.

[0038] Based on the dynamic connection distance thresholds calculated for different regions of the point cloud in the previous step , a surface reconstruction algorithm, such as the wavefront propagation algorithm, is used to generate triangular mesh patches from the sparse 3D point cloud. The algorithm process is roughly as follows: First, an initial seed edge or seed triangle needs to be selected. Usually, an initial edge can be formed by choosing any two points in the point cloud that are relatively close and have the same normal vector direction, or three neighboring points that form a well-shaped (non-degenerate) triangle can be found as the seed. Then, the algorithm maintains an active boundary edge list (i.e., the "wavefront"), which initially contains the seed edge (or the three edges of the seed triangle). In each iteration, an edge (for example, denoted as ) is selected from the active boundary edge list, and candidate points are searched in its neighborhood (for example, within a spherical region centered at the midpoint of the edge with a radius related to ). The candidate point needs to meet the following conditions to form a new triangle with the edge : First, the point cannot be a point already in the mesh, and the connection with cannot cause self-intersection of the mesh; Second, the distance between the point and the two vertices of the edge must be less than the dynamic connection distance threshold corresponding to this region ; Third, to ensure the smoothness of the surface or the reasonable transition of features, the normal vector or curvature of the point and the point (or the newly formed triangle and neighboring triangles) need to be similar. At the same time, one or more curvature values (such as mean curvature or Gaussian curvature) need to be estimated for each point in advance, and then the absolute value of the difference in curvature values between the potential connection point pair and is compared to see if it is less than 0.1. This 0.1 threshold is an empirical value obtained through experimental adjustment based on the expected smoothness of the surface of traditional Chinese medicine samples and the need to retain detailed features. For example, the effects of testing values such as 0.05, 0.1, 0.2, etc. on the reconstruction effect are tested, and 0.1 is selected as the best value to balance smoothness and details. If the best candidate point that meets all conditions is found (for example, the point that forms a triangle with the optimal angle), then the new triangle ​ Add to the grid, and for the two newly generated edges and (if they are not already on the active boundary), add them to the active boundary list, and remove the edge from the list. Repeat this process to continuously expand the grid boundary until the active boundary list is empty, that is, no more suitable points can be found to expand the grid. Since is dynamically calculated, this process can generate smaller triangular patches in areas with high point cloud density and good quality, and larger patches in sparse or complex areas, finally forming a set of triangular mesh patches with adaptive density. This set together constitutes the three-dimensional model of the sample to be measured.

[0039] The steps to obtain the multi-dimensional defect index set are as follows: Based on the dynamic change feature map of the surface of traditional Chinese medicine, extract the set of pixel points in all changed areas. Set the judgment conditions for the moisture absorption defect feature as: the rising rate of color value ≥ 15% / second and the regional area expansion rate ≥ 3% / second within 5 consecutive frames; set the judgment conditions for the oiliness defect feature as: the texture direction consistency coefficient ≤ 0.3 and the regional brightness mean decline gradient ≥ 20 lumens / second; Based on the three-dimensional model of the sample to be measured and the three-dimensional model of the same grade specification level, perform rough alignment with the bottom surface of the model as the reference plane, and iteratively adjust the model posture until the average Euclidean distance between the top of the medicinal material and the root bifurcation point is less than 0.5 mm and the maximum single-point deviation does not exceed 1.2 mm, generating a three-dimensional model comparison group after spatial registration; Based on the three-dimensional model comparison group after spatial registration, divide the surface grid into 10 mm × 10 mm detection units, and synchronously calculate the surface curvature difference degree, normal vector deflection angle, and volume expansion rate within each detection unit. Superimpose the porosity change amount within the detection unit on the moisture absorption defect area, and superimpose the surface glossiness attenuation rate on the oiliness defect area to generate a multi-dimensional defect index set.

[0040] Specifically, based on the dynamic change feature map of the surface of traditional Chinese medicinal materials obtained in the previous steps, pixel sets that are spatially interconnected are identified and extracted. Each set constitutes an independent change region. Then, for each identified change region, in combination with the original registration time-series image sequence and the per-frame change amount data set, its dynamic behavior characteristics are analyzed in the time dimension to determine whether it conforms to the pattern of specific defects. For the determination of moisture absorption defects, it is necessary to examine the color change and area change trends of this region within a continuous time window (for example, set to 5 consecutive frames of images, and the corresponding time length depends on the frame rate. If the frame rate is 2 frames per second, it is 2.5 seconds). The specific calculation method is as follows: Calculate the change rate of the average color value of the pixels within the region (which can be calculated in the RGB space or converted to a color space that is more in line with human perception, such as the Lab space) over time. For example, by calculating the linear growth slope of the average color value within this window or the average per-frame growth percentage. At the same time, calculate the change rate of the pixel area of the region over time, which can also be obtained through linear fitting or the average per-frame expansion percentage. When the color value increase rate (usually manifested as the color becoming darker) reaches or exceeds 15% per second, and the region area expansion rate reaches or exceeds 3% per second, this region is preliminarily determined to conform to the moisture absorption defect characteristics. These two thresholds (15% / second and 3% / second) are empirical values obtained based on the observation and data analysis of the moisture absorption processes of a large number of actual traditional Chinese medicinal materials, representing typical moisture absorption dynamic characteristics; for the determination of oil seepage defects, attention is paid to the texture change and brightness change of the region. Calculate the texture direction consistency coefficient within this region. For example, by calculating the direction histogram of the image gradient within the region, and statistically calculating the proportion of the main direction gradient energy in the total gradient energy, or using the gray-level co-occurrence matrix to calculate some texture features (such as contrast, correlation) and comprehensively evaluating its disorder degree to obtain a coefficient between 0 (completely disordered) and 1 (highly consistent). At the same time, calculate the gradient (i.e., the decrease rate) of the average brightness value of the pixels within the region over time (for example, the gray value or the brightness channel value). When the texture direction consistency coefficient is lower than or equal to 0.3, indicating that the original texture is significantly damaged or becomes messy, and the region average brightness decrease gradient reaches or exceeds 20 brightness units per second (the brightness unit corresponds to the gray level of an 8-bit image or the lumen equivalent unit), this region is preliminarily determined to conform to the oil seepage defect characteristics. Here, the 0.3 threshold and the 20 brightness units / second threshold are also set by quantitatively analyzing the texture and brightness changes of known oil seepage samples to set the boundary values that can effectively distinguish the oil seepage phenomenon from other changes, and complete the feature calculation and condition judgment for all change regions.

[0041] Based on the three-dimensional model of the sample to be tested generated in the previous step and the pre-prepared standard three-dimensional model representing the specification grade of this traditional Chinese medicine (this standard model is usually constructed by averaging or selecting typical representatives after high-precision three-dimensional scanning of multiple batches of qualified samples of the same grade), it is necessary to accurately align these two three-dimensional models in space for subsequent geometric comparison. The alignment process starts with a rough alignment: identify and select the planar regions on the two models that can represent their bottoms or placement benchmarks (which can be achieved through interactive selection or an automatic detection algorithm based on the larger planar features at the bottom of the model), calculate an initial transformation (mainly rotation and translation) to make the bottom surface of the three-dimensional model of the sample to be tested approximately parallel to the bottom surface of the standard three-dimensional model and close in height. After completing the rough alignment, perform iterative adjustment for fine alignment. This process depends on the key anatomical landmark points predefined or automatically detected on the two models, such as the top point of the medicinal material (the point farthest along the main axis) and the main root bifurcation points (the position points where the root system structure branches). Establish a one-to-one correspondence between these landmark points, and use a variant of the iterative closest point (ICP) algorithm or a landmark-based registration algorithm. In each iteration, calculate the rigid transformation (including 3 rotation parameters and 3 translation parameters) that can minimize the Euclidean distance in three-dimensional space between the corresponding landmark points, and apply this transformation to the three-dimensional model of the sample to be tested. The iteration continues until the preset convergence criterion is met. This criterion is set as: the average value of the Euclidean distances between all pairs of corresponding landmark points is less than 0.5 mm, and the Euclidean distance of any pair of landmark points does not exceed 1.2 mm. These two thresholds (the average distance of 0.5 mm and the maximum distance of 1.2 mm) are determined according to the allowable deviation range of morphological dimensions in the quality inspection standards of this grade of traditional Chinese medicine or the pharmacopoeia regulations, ensuring that the aligned models achieve sufficient matching accuracy in key morphological features. When the iteration meets the convergence criterion, stop the adjustment. At this time, the three-dimensional model of the sample to be tested and the standard three-dimensional model of the grade constitute a three-dimensional model comparison group after spatial registration.

[0042] Based on the three-dimensional model of the sample to be measured and the three-dimensional model of the standard grade (i.e., the three-dimensional model comparison group after spatial registration) that have been spatially registered and aligned, it is necessary to quantitatively evaluate the geometric deviation on the surface of the sample and the change in physical properties related to specific defects. First, divide the surface of the aligned three-dimensional model of the standard grade (or a reference surface covering it) into regular detection cells, for example, defined as a square grid area with a side length of 10 mm multiplied by 10 mm. This size of 10 mm × 10 mm is set according to the conventional spatial resolution requirements for traditional Chinese medicine defect detection and the convenience of subsequent statistical analysis, and can be adjusted according to the specific types and sizes of traditional Chinese medicine. Then, for each detection unit, synchronously calculate the geometric difference index between the surface area of the corresponding three-dimensional model of the sample to be measured and the surface area of the three-dimensional model of the standard grade in this unit. Specifically, it includes: calculating the difference degree of the average surface curvature (such as Gaussian curvature or mean curvature) of the two models within this unit; calculating the deflection angle between the average surface normal vectors of the two models within this unit; calculating the change in the local volume enclosed by the surface of the sample to be measured relative to the surface of the standard grade within this unit, and expressing it as a volume expansion rate (for example, the average height difference or the ratio of the local volume difference to the standard volume relative to the unit area). In addition to these general geometric difference indicators, it is also necessary to combine the information on the potential defect areas identified and mapped to the surface of the three-dimensional model for targeted index superposition calculation: if a certain detection unit overlaps with the determined moisture absorption defect area, then additionally calculate the change in porosity within this unit. This requires analyzing the local geometric features (such as the number, depth, and area distribution of small depressions) on the surface of the three-dimensional model of the sample to be measured within the unit to estimate its surface porosity, and comparing it with the porosity of the corresponding area of the standard grade model to obtain the change amount; if the detection unit overlaps with the determined oil seepage defect area, then additionally calculate the attenuation rate of the surface glossiness. This calculation requires combining color and texture information, or using a specific lighting model and material reflection properties (BRDF) to estimate the glossiness of the surface of the sample to be measured within this unit, and comparing it with the glossiness reference value of the standard grade model to quantify its attenuation degree. After calculating these indicators (surface curvature difference degree, normal vector deflection angle, volume expansion rate, and porosity change amount or surface glossiness attenuation rate according to the area type) for all detection units, summarize these quantified indicators to form a multi-dimensional defect index set covering the entire surface of the sample.

[0043] The acquisition steps for determining the defect categories of traditional Chinese medicine are as follows: Based on the multi-dimensional defect index set, if the same detection unit simultaneously meets the conditions of moisture absorption defect candidate and oil seepage defect candidate, it is preferentially determined as a moisture absorption defect; if the distance between the defect candidate categories of different detection units ≤ 5 mm, there is spatial adjacency, and they are merged into a composite defect area and upgraded to a serious defect category, generating a corrected set of defect category determination results; Based on the corrected defect category determination result set, calculate the area proportion and severity level of all defect areas. If the total area proportion of moisture absorption defects ≥ 5% or there are areas of severe defect categories, the sample is determined to be unqualified; if the total area proportion of defects < 5% and there are no severe defect categories, output a qualified label to generate the defect category determination of traditional Chinese medicine materials.

[0044] Specifically, based on the multi-dimensional defect index set obtained in the previous steps, which contains the geometric and physical property deviation information of each detection unit, it is necessary to finally confirm and correct the defect category of each unit. First, traverse each 10 mm × 10 mm detection unit, and comprehensively evaluate whether the unit simultaneously meets the previously set candidate conditions for moisture absorption defects and candidate conditions for oiling defects according to the various index values it contains (for example, the color rise rate, area expansion rate, and porosity change amount correspond to moisture absorption; the texture consistency coefficient, brightness decline gradient, and gloss attenuation rate correspond to oiling; and general indexes such as curvature difference degree, normal vector deflection angle, and volume expansion rate). If it is indeed found that the indexes of a unit strongly point to the possibility of both defects at the same time, then make a category ruling according to the preset rules: it is stipulated that such units should be preferentially determined to be moisture absorption defects. The basis for this preferential determination may be that based on expert experience, moisture absorption is considered to be a more common or more basic form of deterioration, or the manifestation of moisture absorption in visual or physical properties is more likely to cover up or confuse the oiling characteristics, thus guiding the determination to a more definite or more critical category. After the preliminary category determination of all units is completed, perform spatial correlation analysis and severity upgrade. Identify all detection units determined to have defects (whether moisture absorption or oiling), merge the units that are adjacent in space (sharing edges or vertices) and belong to the same defect category to form continuous defect areas. Then, calculate the minimum distance between different defect category areas. For example, calculate the shortest three-dimensional space distance between the boundary of a moisture absorption defect area and the boundary of the nearest oiling defect area. If this distance is less than or equal to 5 mm, it is considered that there is a significant spatial adjacency between these two different types of defects. This adjacency may mean the mutual influence of defects or represent a more complex deterioration process, so special treatment is required. This 5 mm threshold is based on the research on the deterioration mode of traditional Chinese medicine materials to determine a distance range sufficient to indicate that there may be an interaction or common cause between defects. For all defect areas of different categories (or the detection units that make them up) that meet this spatial adjacency condition, merge them into a larger "composite defect area", and upgrade the defect category of this area to be marked as "severe defect category". After the above conflict resolution, area merging, and severity upgrade processing, the corrected defect category determination result set is obtained.

[0045] Based on the corrected defect category determination result set generated in the previous step (which includes the final defect category and its severity level for each detection unit or merged region), it is necessary to make a final determination on the quality status of the entire Chinese herbal medicine sample. First, using this result set and the three-dimensional model data of the sample, calculate the total area of all regions determined to be defective on the sample surface, and specifically calculate the total area of all regions determined to be of the "moisture absorption defect" category. At the same time, obtain the total surface area of the sample (which can be directly calculated from the three-dimensional model). Then, calculate the percentage of the total moisture absorption defect area in the total surface area of the sample, and the formula is: (total moisture absorption defect area / total surface area of the sample) × 100%. Next, check whether there are any regions marked as "severe defect category" in the corrected defect category determination result set (i.e., regions upgraded by merging different types of spatially adjacent defects). Make a final judgment according to the preset quality determination criteria: Criterion 1, whether the calculated percentage of the total moisture absorption defect area is greater than or equal to 5%; Criterion 2, whether there is at least one severe defect category region. If the limit is exceeded, it is considered that the quality of the medicinal material is significantly affected. The existence of a severe defect category, regardless of its area, usually represents more complex or more severe deterioration and is therefore also regarded as a direct indicator of non-conformity. If at least one of the above two criteria is met (i.e., the percentage of the moisture absorption defect area ≥ 5% or there is a severe defect region), then finally determine that the Chinese herbal medicine sample is unqualified. If neither of the two criteria is met (i.e., the percentage of the moisture absorption defect area < 5% and there is no severe defect region), then determine that the sample is qualified and output a qualified label, and finally generate a defect category determination (qualified or unqualified) for the Chinese herbal medicine sample.

[0046] The above is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical solution content of the present invention, any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A vision-based defect detection method for traditional Chinese medicine materials, characterized in that The following steps are involved: Collect time-series images of Chinese medicinal materials samples at regular intervals, collect multi-view images, align the pixels of the time-series images, structurally organize and associate camera parameters of the multi-view images according to the shooting orientation, and generate a registered time-series image sequence and a multi-view image set; Based on the registered time-series image sequence, the color value difference between the corresponding pixel positions between adjacent frame images in the sequence is calculated to obtain a frame-by-frame variation data set. Based on the frame-by-frame variation data set, the total variation value of each pixel position is cumulatively calculated along the time dimension to screen and obtain a dynamic variation feature map of the surface of the Chinese medicinal materials; Based on the multi-view image set, a sparse three-dimensional point cloud and a camera posture are obtained by matching feature points of the same name between images of different view angles and combining camera parameters; based on the sparse three-dimensional point cloud and the camera posture, adjacent points in the point cloud are connected to construct a triangular mesh patch, and a three-dimensional model of the sample to be tested is established; Based on the dynamic change characteristic map of the Chinese medicinal material surface and the three-dimensional model of the sample to be tested, whether the pattern of the change area in the dynamic change characteristic map of the Chinese medicinal material surface conforms to the dynamic defect characteristics of moisture absorption or oil overflow is analyzed, and a multi-dimensional defect indicator set is obtained. Based on the multi-dimensional defect indicator set, the qualification or defect category of the Chinese medicinal material sample is output, and a Chinese medicinal material defect category determination is established.

2. The vision-based traditional Chinese medicine defect detection method according to claim 1, wherein The steps of obtaining the registration time-series image sequence and the multi-view image set are: Time-series images and multi-view images of the same batch of Chinese medicinal materials samples are collected regularly, and pixel alignment is performed on the time-series images. The SIFT feature points of adjacent frame images are extracted and bidirectionally matched, and the spatial Euclidean distance of matching point pairs is calculated. The RANSAC algorithm is used to eliminate outlier matching point pairs, and the affine transformation matrix parameters are solved and applied to the entire image to generate a sequence of registered time-series images. Based on the registered time-series image sequence and the multi-view images, the multi-view images are structured and organized, the focal length, principal point coordinates and radial distortion coefficient in the camera parameter file of each image are parsed, a mapping relationship between the azimuth angle, the pitch angle and the camera parameters is established, and a multi-view image set is generated; Based on the registered temporal image sequence and the multi-view image set, the validity of the temporal alignment is verified by calculating the SSIM structural similarity of adjacent frames in the temporal sequence, and the registered temporal image sequence and the multi-view image set are generated.

3. The vision-based Chinese medicinal material defect detection method according to claim 1, wherein The steps for acquiring the frame-by-frame variation data set are: Based on the registered time-series image sequence, extract the RGB three-channel color value of each pixel position in the adjacent frame image, establish the current frame color value matrix and the previous frame color value matrix, and generate an adjacent frame color value matrix group; Based on the adjacent frame color value matrix group, calculating the color value difference of the corresponding pixel position; Based on the color value difference, all pixel positions and frame sequence indexes are traversed, the color value difference is superimposed and arranged in frame sequence, and a frame-by-frame variation data set is generated.

4. The vision-based traditional Chinese medicine defect detection method according to claim 1, wherein The steps for obtaining the characteristic spectrum of dynamic changes on the surface of the Chinese medicinal material are as follows: Based on the frame-by-frame variation data set, extract the color value difference sequence of each pixel position in all frames, establish a time series variation set of the pixel position, and generate pixel-level time accumulation input data; Calculate the total change amount value at each pixel position based on the pixel-level time-cumulative input data; Based on the total change amount value, use the sliding window method to calculate the mean and standard deviation of the total change amount in the neighborhood around each pixel, set a dynamic threshold, screen the pixel regions where the total change amount value is greater than the dynamic threshold, and generate the dynamic change feature map of the Chinese herbal medicine surface.

5. The visual-based traditional Chinese medicine defect detection method according to claim 1, wherein The steps for obtaining the sparse three-dimensional point cloud and camera pose are as follows: Based on the multi-view image set, extract SIFT feature points from every two view images and generate feature descriptors, perform two-way matching by calculating the Hamming distance between the feature descriptors, use the RANSAC algorithm to fit the fundamental matrix and remove the outlier matching point pairs with a projection error exceeding 0.5 pixels, and generate a set of corresponding feature point matching pairs between multi-view images; Based on the set of corresponding feature point matching pairs and the focal length, principal point coordinates, and radial distortion coefficient in the camera parameters, construct the epipolar geometry constraint equation, solve the camera rotation matrix and translation vector by singular value decomposition of the essential matrix, and at the same time use the linear triangulation method to calculate the initial three-dimensional space coordinates of the feature point pairs, and generate the camera external parameter estimation result and the sparse three-dimensional point coordinate set; Based on the camera external parameter estimation result and the sparse three-dimensional point coordinate set, use the reprojection error as the loss function to iteratively adjust the rotation matrix, translation vector, and three-dimensional point coordinates until the average reprojection error converges within 0.3 pixels, and remove the abnormal three-dimensional points with a residual error exceeding 1.0 pixel after optimization, and generate the sparse three-dimensional point cloud and camera pose.

6. The vision-based traditional Chinese medicine defect detection method according to claim 1, wherein The steps for obtaining the three-dimensional model of the sample to be tested are as follows: Based on the sparse three-dimensional point cloud and camera pose, extract the optical center distance, the mean value of the three-dimensional point reprojection error, and the surface curvature change rate gradient between adjacent camera views, and establish an interpolation credibility evaluation parameter group; Calculate the dynamic connection distance threshold based on the interpolation credibility evaluation parameter group; Based on the dynamic connection distance threshold, use the wavefront propagation algorithm to perform region growing on the point cloud, preferentially connect the adjacent point pairs with a distance less than the dynamic connection distance threshold and a curvature difference less than 0.1, generate a set of triangular mesh patches with adaptive density, and establish the three-dimensional model of the sample to be tested.

7. The vision-based traditional Chinese medicine defect detection method according to claim 1, wherein The steps for obtaining the multi-dimensional defect index set are as follows: Based on the dynamic change feature map of the Chinese herbal medicine surface, extract the set of pixel points in all change regions, and set the judgment conditions for the moisture absorption defect feature as: the color value rising rate ≥ 15% / second and the regional area expansion rate ≥ 3% / second within 5 consecutive frames; set the judgment conditions for the oil exudation defect feature as: the texture direction consistency coefficient ≤ 0.3 and the regional brightness mean value decreasing gradient ≥ 20 lumens / second; Based on the three-dimensional model of the sample to be tested and the three-dimensional model of the same grade specification level, perform rough alignment with the bottom surface of the model as the reference plane, and iteratively adjust the model pose until the mean Euclidean distance between the top of the Chinese herbal medicine and the root bifurcation point is less than 0.5 mm and the maximum single-point deviation does not exceed 1.2 mm, and generate a three-dimensional model comparison group after spatial registration; Based on the three-dimensional model comparison group after spatial registration, the surface mesh is divided into detection units of 10 mm × 10 mm. The surface curvature difference, normal vector deflection angle, and volume expansion rate are synchronously calculated within each detection unit. The porosity change amount within the detection unit is superimposed on the moisture absorption defect area, and the surface gloss attenuation rate is superimposed on the oiling defect area to generate a multi-dimensional defect index set.

8. The vision-based traditional Chinese medicine defect detection method according to claim 1, wherein The acquisition steps for determining the defect categories of traditional Chinese medicinal materials are as follows: Based on the multi-dimensional defect index set, if a detection unit simultaneously meets the conditions of being a candidate for moisture absorption defect and a candidate for oiling defect, it is preferentially determined as a moisture absorption defect; if the distance between the defect candidate categories of different detection units ≤ 5 mm, there is spatial adjacency, and they are merged into a composite defect area and upgraded to a serious defect category, generating a corrected set of defect category determination results. Based on the corrected set of defect category determination results, the area ratio and severity level of all defect areas are statistically analyzed. If the total area ratio of moisture absorption defects ≥ 5% or there is a serious defect category area, the sample is determined to be unqualified. If the total area ratio of defects < 5% and there is no serious defect category, a qualified label is output to generate a determination of the defect categories of traditional Chinese medicinal materials.

Images