Chinese herbal medicine authentic identification method and system based on multi-modal large model, equipment and medium

Abstract

The invention provides a Chinese herbal medicine authentic identification method and system based on a multi-modal large model, equipment and a medium, and belongs to the technical field of data processing, and the method comprises the steps: responding to target multi-modal data as image data, carrying out the registration processing of the image data based on space matching, and obtaining the multi-class space registration image data; pixel normalization processing is carried out based on the multi-class space registration image data to obtain multi-class consistency image data; in response to the fact that the target multi-modal data is text data, performing standardization processing on the text data to obtain multi-class consistent text data; taking the multi-class consistency image data and the multi-class consistency text data as target consistency multi-modal data; and inputting the target consistency multi-modal data into the target multi-modal large model to obtain a quality identification result of the target Chinese herbal medicine. According to the invention, the identification capability under the condition of small samples can be improved, so that the accuracy of authentic identification is improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of data processing, and more specifically, relates to a method, system, device, and medium for authenticating Chinese herbal medicines based on a multimodal large model. Background Art

[0002] Currently, the methods for authenticating Chinese herbal medicines mainly rely on manual detection and traditional analysis means. The existing methods mainly include: empirical identification techniques, which achieve the purpose of identification by using the appearance traits (shape, color, smell, taste) of medicinal materials and using sensory methods such as seeing, touching, smelling, and tasting (when necessary, adding water or using fire testing methods); micro-trait identification, which is a method of observing the fine trait characteristics on the surface or cross-section of traditional Chinese medicines with the assistance of instruments such as stereomicroscopes, and then identifying the quality of traditional Chinese medicines; molecular identification techniques, which analyze the DNA or RNA information of Chinese herbal medicines and use molecular biology methods to identify the types, sources, quality, etc. of medicinal materials. Moreover, the problems of traditional Chinese herbal medicine authentication methods relying on a large amount of labeled data and insufficient discrimination ability in small sample cases affect the accuracy of authentication. Summary of the Invention

[0003] The purpose of the present disclosure is to provide a method, system, device, and medium for authenticating Chinese herbal medicines based on a multimodal large model, so as to improve the discrimination ability in small sample cases and thus improve the accuracy of authentication.

[0004] In the first aspect of the embodiments of the present disclosure, a method for authenticating Chinese herbal medicines based on a multimodal large model is provided, including: In response to the target multimodal data being image data, performing registration processing on the image data based on spatial matching to obtain multiple types of spatially registered image data; and performing pixel normalization processing on the multiple types of spatially registered image data to obtain multiple types of consistent image data; In response to the target multimodal data being text data, performing standardization processing on the text data to obtain multiple types of consistent text data; Taking the multiple types of consistent image data and the multiple types of consistent text data as target consistent multimodal data; the target multimodal data is obtained by collecting the target Chinese herbal medicine in different ways; Inputting the target consistent multimodal data into a target multimodal large model to obtain the quality discrimination result of the target Chinese herbal medicine.

[0005] In the second aspect of the embodiments of the present disclosure, a system for authenticating Chinese herbal medicines based on a multimodal large model is provided, including: An image preprocessing module, which is configured to, in response to the target multimodal data being image data, perform registration processing on the image data based on spatial matching to obtain multiple types of spatially registered image data; and perform pixel normalization processing based on the multiple types of spatially registered image data to obtain multiple types of consistent image data; A text preprocessing module, which is configured to, in response to the target multimodal data being text data, perform standardization processing on the text data to obtain multiple types of consistent text data; A consistent multimodal data determination module, which is configured to use the multiple types of consistent image data and the multiple types of consistent text data as target consistent multimodal data; the target multimodal data is obtained by collecting a target Chinese herbal medicine in different ways; An authenticity identification module, which is configured to input the target consistent multimodal data into a target multimodal large model to obtain a quality identification result of the target Chinese herbal medicine.

[0006] In a third aspect of the embodiments of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the above-mentioned Chinese herbal medicine authenticity identification method based on a multimodal large model are implemented.

[0007] In a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned Chinese herbal medicine authenticity identification method based on a multimodal large model are implemented.

[0008] The beneficial effects of the Chinese herbal medicine authenticity identification method, system, device, and medium based on a multimodal large model provided by the embodiments of the present disclosure are as follows: The present disclosure processes image data through spatial matching and pixel normalization, and processes text data through standardization, ensuring the consistency and comparability of the target multimodal data, and providing a reliable basis for subsequent quality identification. At the same time, the present disclosure can flexibly handle different forms of target multimodal data, whether it is image or text, and can effectively process and analyze them, improving the accuracy and comprehensiveness of authenticity identification. Subsequently, using the target multimodal large model for quality identification can make full use of the multi-dimensional features of Chinese herbal medicines to achieve more comprehensive and in-depth authenticity identification analysis. Compared with traditional single-modal authenticity identification methods, this method has higher authenticity identification accuracy and stronger generalization ability. In addition, the present disclosure also has high efficiency and convenience. Through automated data preprocessing and model identification, the efficiency of Chinese herbal medicine authenticity identification can be greatly improved, reducing labor costs and operation difficulties. Therefore, the present disclosure can improve the identification ability in the case of small samples, thereby improving the accuracy of authenticity identification. Description of the Drawings

[0009] To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0010] Figure 1 Schematic flowchart of a Chinese herbal medicine authentication method based on a multimodal large model provided by an embodiment of the present disclosure; Figure 2 Block diagram of the structure of a Chinese herbal medicine authentication system based on a multimodal large model provided by an embodiment of the present disclosure; Figure 3 Schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0011] In the following description, specific details such as specific system structures and technologies are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present disclosure. However, those skilled in the art should clearly understand that the present disclosure can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present disclosure.

[0012] To make the purpose, technical solutions, and advantages of the present disclosure clearer, the following will be described through specific embodiments in conjunction with the drawings.

[0013] Please refer to Figure 1 , Figure 1 Schematic flowchart of a Chinese herbal medicine authentication method based on a multimodal large model provided by an embodiment of the present disclosure. The method includes: S101: In response to the target multimodal data being image data, perform registration processing on the image data based on spatial matching to obtain multiple types of spatially registered image data; and perform pixel normalization processing on the multiple types of spatially registered image data to obtain multiple types of consistent image data.

[0014] In this embodiment, the image data may include RGB images, hyperspectral images, and microscope images. The image data can be collected by a high-resolution camera, a hyperspectral imager, or a microscope, and can reflect the overall, cross-sectional, and detailed texture features of the target Chinese herbal medicine from multiple dimensions.

[0015] Specifically, performing registration processing on the image data based on spatial matching to obtain multiple types of spatially registered image data includes: Select multiple pairs of corresponding feature points in the RGB image and the hyperspectral image; Calculate the homography affine transformation matrix for each pair of feature points based on the least squares method; Taking the RGB image as a reference, transform the hyperspectral image based on the homography affine transformation matrix, complete the registration process, and obtain multi-class spatially registered image data.

[0016] In image registration, it is the process of finding the same or similar feature points in different images, establishing corresponding relationships based on these points, and determining the spatial transformation parameters between the images. Based on the spatial matching results, geometric transformations are performed on different images to align the same objects or regions in the images in terms of spatial position. Multi-class spatially registered image data is a set of images that are registered, have different perspectives, different types (such as overall and local, macroscopic and microscopic), and are spatially aligned. The homography affine transformation matrix describes the spatial transformation relationship from the RGB image to the hyperspectral image.

[0017] Pixel normalization processing is an operation to uniformly transform the image pixel values in the multi-class spatially registered image data to a certain range (such as [0, 1] or [-1, 1]) to eliminate the problem of different pixel value ranges caused by differences in shooting equipment and environment for different images. Multi-class consistent image data is various types of image data that have reached consistency in terms of pixel value range, data format, etc. after pixel normalization processing.

[0018] S102: In response to the target multi-modal data being text data, perform standardization processing on the text data to obtain multi-class consistent text data.

[0019] In this embodiment, the text data may include three-dimensional topography data and pharmacopoeia text description data. The text data can obtain the three-dimensional topography data of the target Chinese herbal medicine through a structured light scanner, and can analyze the surface concave and convex features, which are used to effectively identify forgery means such as mold pressing.

[0020] Performing standardization processing on the text data is a process of cleaning, unifying the format, standardizing the vocabulary, etc. on the text to make the texts from different sources consistent.

[0021] Multi-class consistent text data is various types of text data that are unified in terms of format, semantic expression norms, etc. after standardization processing.

[0022] S103: Use the multi-class consistent image data and the multi-class consistent text data as the target consistent multi-modal data; the target multi-modal data is obtained by collecting the target Chinese herbal medicine in different ways.

[0023] In this embodiment, the target multi-modal data is a data set obtained for the target Chinese herbal medicine through a variety of different acquisition methods, which can include information in different dimensions. For example, the appearance image obtained by using a high-resolution camera shows the shape and color from a visual perspective; the hyperspectral image collected by means of a hyperspectral imaging unit can also collect the surface reflection spectrum of the medicinal material; the micro-character image collected by means of a microscopic imaging unit presents microscopic texture details; and there is also the medicinal property description text extracted from materials such as pharmacopoeias, providing literal information such as chemical components and efficacy.

[0024] The target-consistent multi-modal data is a multi-modal data set that integrates multiple types of consistent image data and multiple types of consistent text data, making them coordinated and consistent in terms of format, scale, semantics, etc., and can be used collaboratively for subsequent analysis.

[0025] In this embodiment, the image data and text data are first preprocessed separately, and then the two types of processed data are integrated. The principle is that the image data can be aligned in spatial position through spatial registration, facilitating comprehensive analysis; pixel normalization unifies the pixel value range, eliminating the numerical differences between images and facilitating subsequent model processing. The standardized processing of text data can standardize the expression and improve the text quality and consistency. Finally, the two types of consistent data are integrated to provide a coordinated and unified multi-modal input for the multi-modal large model, giving play to the complementary advantages of multi-modal data and improving the accuracy of the quality identification of Chinese herbal medicines.

[0026] The target-consistent multi-modal data is a data form that, after preprocessing, unifies multi-modal data from different sources and types in terms of format, scale, space, position, etc., and can be used collaboratively for subsequent processing.

[0027] S104: Input the target-consistent multi-modal data into the target multi-modal large model to obtain the quality identification result of the target Chinese herbal medicine.

[0028] In this embodiment, the target multi-modal large model is a trained multi-modal large model that can integrate data information of multiple different modalities. By learning the features and patterns in the data, it has the ability to analyze and judge the target Chinese herbal medicine, and is used to identify the quality of the target Chinese herbal medicine, that is, whether the target Chinese herbal medicine is genuine or fake, so as to realize the identification of counterfeit Chinese herbal medicines. The quality of the target Chinese herbal medicine is the authenticity of the target Chinese herbal medicine, that is, the evaluation of whether it is genuine. This embodiment can construct the target multi-modal large model based on the Transformer architecture.

[0029] Specifically, the steps of this embodiment can be as follows: First, input the target-consistent multi-modal data obtained through preprocessing into the target multi-modal large model. This step enables the target multi-modal large model to obtain the multi-modal information for analysis and prepares for subsequent feature extraction and quality identification.

[0030] Then, after the target multi-modal large model receives the input data, it will use its internal multi-layer neural network structure and various algorithms to extract features from data of different modalities respectively. In this embodiment, through a fusion mechanism (such as weighted fusion), the features of different modalities are fused to learn the internal connections and complementary information between multi-modal data, so as to form a comprehensive feature representation of the target Chinese herbal medicine.

[0031] Finally, based on the feature representation of the target Chinese herbal medicine learned, the target multi-modal large model compares it with the knowledge and patterns about the quality of Chinese herbal medicine learned in the training data in advance, and judges the quality of the target Chinese herbal medicine by calculating similarity, probability, etc., and outputs the identification result, such as judging whether it is a genuine product or a counterfeit product.

[0032] As can be seen from the above, the present disclosure ensures the consistency and comparability of the target multi-modal data by spatially matching and pixel-normalizing the image data, and normalizing the text data, providing a reliable basis for subsequent quality identification. At the same time, the present disclosure can flexibly handle different forms of target multi-modal data, whether it is an image or text, and can be effectively processed and analyzed, improving the accuracy and comprehensiveness of forgery detection. After that, using the target multi-modal large model for quality identification can make full use of the multi-dimensional features of Chinese herbal medicine to achieve a more comprehensive and in-depth forgery detection analysis. Compared with the traditional single-modal forgery detection method, this method has higher forgery detection accuracy and stronger generalization ability. In addition, the present disclosure also has high efficiency and convenience. Through automated data preprocessing and model identification, the efficiency of Chinese herbal medicine forgery detection can be greatly improved, reducing the labor cost and operation difficulty. Therefore, the present disclosure can improve the identification ability in the case of small samples, thereby improving the accuracy of forgery detection.

[0033] In an embodiment of the present disclosure, before pixel-normalizing the multi-class spatially registered image data, it further includes: Performing Fourier transform on each class of spatially registered image data to obtain a spectrogram; Calculating the total energy value of the high-frequency region of the spectrogram; the high-frequency region is the region where the frequency is greater than or equal to the frequency threshold; If the total energy value is greater than or equal to the first energy threshold, denoising processing is performed on the multi-class spatially registered image data.

[0034] In this embodiment, the Fourier transform is a mathematical transformation method that converts a time-domain or spatial-domain signal into a frequency-domain signal. In image processing, through two-dimensional Fourier transform, an image can be transformed from the spatial domain to the frequency domain to obtain a spectrogram, so as to analyze the frequency components of the image.

[0035] The high-frequency region is the region in the spectrogram where the frequency is greater than or equal to the set frequency threshold, and is related to the noise information in the image. The total energy value is the value obtained by calculating and summing the energy values corresponding to each frequency component in the high-frequency region of the spectrogram, which reflects the energy distribution in the high-frequency region. The first energy threshold is a preset reference value used to determine whether the energy in the high-frequency region of the spectrogram is too high. When the total energy value is greater than or equal to this threshold, it is considered that the energy in the high-frequency region is abnormal and there is more noise.

[0036] Noise reduction processing is a process of operating on an image to reduce or remove the noise in it and improve the image quality.

[0037] Specifically, noise reduction processing for multiple types of spatially registered images includes: In response to each type of spatially registered image being an RGB spatially registered image, perform noise reduction processing on the RGB spatially registered image based on the bilateral filtering algorithm; In response to each type of spatially registered image being a hyperspectral spatially registered image, perform noise reduction processing on the hyperspectral spatially registered image based on the spectral smoothing filtering algorithm; In response to each type of spatially registered image being a microscope spatially registered image, perform noise reduction processing on the microscope spatially registered image based on the Wiener filtering algorithm.

[0038] In this embodiment, performing noise reduction processing on the RGB spatially registered image based on the bilateral filtering algorithm includes: Determine the spatial domain standard deviation of the bilateral filtering algorithm; Based on the spatial domain standard deviation, determine the neighborhood range value of each pixel point in the RGB spatially registered image; Perform weighted average calculation on the pixel points within the neighborhood range value of each pixel point to obtain the filtered value of this pixel point; Map the filtered values of multiple pixel points to complete the noise reduction processing.

[0039] The spatial domain standard deviation of the bilateral filtering algorithm can be set according to experience.

[0040] Performing noise reduction processing on the hyperspectral spatially registered image based on the spectral smoothing filtering algorithm includes: Determine the filter window size; For each pixel in the hyperspectral spatially registered image, slide the window on the spectral curve according to the filter window size, calculate the average pixel value within the filter window, and obtain the smoothed spectral curve; Combine the spectral curves after smoothing for each pixel and perform reconstruction to obtain the noise-reduced hyperspectral spatially registered image.

[0041] In this embodiment, determining the filter window size includes: Determine the initial filter window size; In response to the total energy value being greater than or equal to the second energy threshold, controlling the initial filter window size to increase by the first step length; In response to the total energy value being less than the second energy threshold and greater than or equal to the first energy threshold, controlling the initial filtering window size to be reduced by a second step size; The second energy threshold is greater than the first energy threshold.

[0042] The second energy threshold, the first step length and the second step length are all set based on experience.

[0043] In this embodiment, the bilateral filtering algorithm is a nonlinear filtering method that combines the spatial proximity and pixel value similarity of the image. While smoothing the image, it can better preserve the edge information of the image and avoid the edge blurring problem caused by traditional filtering methods in the noise reduction process.

[0044] The spectral smoothing filtering algorithm is a filtering method that processes the spectral curve of each pixel of the hyperspectral image. It calculates the values ​​of adjacent bands on the spectral curve to smooth the spectral curve, reduce noise interference, and highlight the spectral characteristics.

[0045] The Wiener filtering algorithm is a linear filtering method based on statistical characteristics. It uses the minimum mean square error as the criterion and filters the noise-contaminated image according to the noise statistical characteristics of the image and the autocorrelation function of the image. It suppresses the noise while retaining the image details as much as possible.

[0046] From the above, it can be concluded that this embodiment obtains a spectrum diagram through Fourier transform, which can intuitively analyze the frequency characteristics of image data; secondly, the total energy of the high-frequency area is calculated and a threshold judgment is set to effectively identify and filter out image data with large noise interference; finally, pre-processing and denoising are performed on images that meet the conditions, which helps to improve the accuracy and stability of subsequent pixel normalization processing and reduce the negative impact of noise on image analysis results, thereby enhancing the overall quality of image data and the reliability of subsequent applications.

[0047] In one embodiment of the present disclosure, the training process of the target multimodal large model includes: Semantically match the historically consistent multimodal data with the pharmacopoeia knowledge base to obtain multiple data pairs; Perform feature extraction on multiple data pairs to obtain multiple types of modal features; Pre-training multiple types of modal features to obtain a multi-modal pre-training model; Based on the incremental parameter update strategy, the parameters of the multimodal pre-trained model are adjusted to obtain the target multimodal large model.

[0048] In this embodiment, a pharmacopoeia knowledge base is introduced, which can be used to train and optimize the model with the rich identification knowledge in the pharmacopoeia. Through the above method, the model can make judgments based on authoritative standards during the identification process, improving the accuracy and reliability of counterfeit identification.

[0049] Compare the semantic information contained in the historical multi-modal data with the drug knowledge in the pharmacopoeia knowledge base, and find data combinations with relevant semantics to form data pairs. For example, combine the image data describing the appearance of the drug with the text data of the appearance description of the drug in the pharmacopoeia.

[0050] For different modal data in each data pair, use corresponding feature extraction algorithms. For text data, methods such as word embedding can be used to extract features; for image data, a convolutional neural network can be used to extract features, thus obtaining feature representations of multiple modalities.

[0051] Input the extracted multi-modal features into the Transformer architecture and perform pre-training using a large amount of multi-modal data. By optimizing the parameters of the model, the model can learn the associations and feature representations between different modal data and possess preliminary multi-modal data processing capabilities.

[0052] Based on the pre-trained model, use new multi-modal data to fine-tune the model. According to the incremental parameter update strategy, calculate the update amount of the parameters based on the new data and gradually update the parameters of the model, so that the model can better adapt to specific tasks and data, and finally obtain the target multi-modal large model.

[0053] It can be concluded from the above that in this embodiment, by integrating historical consistent multi-modal data and the pharmacopoeia knowledge base, deep semantic matching of information is achieved, effectively improving the accuracy and reliability of the model. The feature extraction step ensures the comprehensive capture of multi-class modal features, providing a rich information basis for the model. The pre-training process further enhances the generalization ability of the model, enabling it to handle complex and changing task scenarios. And the incremental parameter update strategy ensures the continuous optimization of the model during continuous learning.

[0054] In an embodiment of the present disclosure, a multi-modal pre-trained model is obtained by performing pre-training processing on multi-class modal features, including: Divide the multi-class modal features into positive samples and negative samples; Perform global optimization on the positive samples and negative samples respectively based on the global contrast loss function; Perform local optimization on the positive samples and negative samples respectively based on the local contrast loss function; Determine the initial parameters of the multi-modal large model based on global optimization and local optimization; Determine the multi-modal pre-trained model based on the initial parameters of the multi-modal large model.

[0055] In this embodiment, positive samples are combinations of features with similar semantics or representing the same class of things in multi-modal features, that is, these samples are similar in semantics or category.

[0056] Negative samples are the opposite of positive samples. They are combinations of multi-modal features with different semantics or representing different classes of things, and are used to let the model learn to distinguish different things or semantics.

[0057] The global contrast loss function is used to measure the differences of multi-modal features in the overall space. By maximizing the similarity between positive samples and minimizing the similarity between negative samples, the model learns the global distribution law of features.

[0058] The calculation formula of the global contrast loss function is:

[0059]

[0060]

[0061] Among them, is the value of the global contrast loss function, is the feature vector of sample , and its positive sample feature vector is , and the negative sample feature vector is , , is a function for calculating the cosine similarity between two feature vectors, is the negative sample weight coefficient, ; and are both used to traverse the indexes of negative samples. is the number of negative samples. By traversing to find the maximum value of the similarity between negative samples and positive samples, and traversing is used to calculate the summation term in the denominator , so as to determine the weight coefficient of the negative sample ; is a dynamic adjustment parameter, which can be adjusted according to the training round . is the initial adjustment parameter, which is set according to experience. is a hyperparameter that controls the descent rate of the adjustment parameter. .

[0062] The local contrast loss function focuses on measuring the differences of multi-modal features in the local area, helps the model capture the local detailed information of features, and enables the model to better distinguish features that are locally similar but globally different.

[0063] The calculation formula of the local contrast loss function is as follows:

[0064] Among them, the feature vector is divided into local regions, which are respectively denoted as , , , , is a tuning parameter that can be set based on experience and is used to adjust the sensitivity of the model to local similarity differences.

[0065] Input the positive and negative samples into the model, and use the global contrast loss function to calculate the difference between the feature representation output by the model and the expected representation. Through the backpropagation algorithm, adjust the parameters of the model so that the features of the positive samples are closer in the feature space and the features of the negative samples are farther away, so that the model can learn the global distribution pattern of multimodal features.

[0066] On the basis of global optimization, input the positive and negative samples into the model again, and use the local contrast loss function to calculate the loss value. This function focuses on the local part of the features, and further adjusts the model parameters through backpropagation, enabling the model to capture the local similarities and differences of multimodal features and improving the model's ability to distinguish feature details.

[0067] In this embodiment, through multiple iterative optimizations of the global contrast loss function and the local contrast loss function, the parameters of the model gradually converge to a better state, and these optimized parameters are determined as the initial parameters of the multimodal large model. Load the obtained initial parameters into a pre-designed multimodal model architecture to construct a multimodal pre-trained model.

[0068] It can be concluded from the above that in this embodiment, classifying multiple types of modal features into positive and negative samples helps the model to more accurately understand the associations and differences between data. The use of the global contrast loss function enhances the model's ability to grasp the overall data distribution and achieves global optimization. At the same time, the introduction of the local contrast loss function further improves the model's sensitivity to detailed features and achieves local optimization. This embodiment ensures the accuracy and robustness of the initial parameters of the multimodal large model.

[0069] In an embodiment of the present disclosure, before adjusting the initial parameters of the multimodal pre-trained model based on the incremental parameter update strategy, it further includes: Calculate the distance between each type of modal feature and each type of genuine prototype cluster; Determine the genuine probability of each type of modal feature based on the distance; In response to the genuine probability being less than the first probability threshold, determine that this type of modal feature is a counterfeit; In response to the genuine probability being greater than or equal to the first probability threshold, determine that this type of modal feature is genuine.

[0070] In this embodiment, the incremental parameter update strategy does not update all parameters at once during model training. Instead, it gradually and incrementally updates the model parameters based on new data, enabling the model to continuously adapt to new data and continuously optimize its performance. Therefore, small-sample learning is introduced to divide the samples, that is, to determine the authenticity of each type of modal feature based on small-sample learning.

[0071] Small-sample learning is a learning method that enables the model to learn effective features and patterns in the case of a small number of samples, so as to accurately classify or predict new samples.

[0072] Modal features are information that can represent the essential characteristics of different modal data (such as text, images, audio, etc.) extracted from the modal data.

[0073] The genuine prototype cluster is a set of cluster centers representing the typical characteristics of genuine products obtained through clustering analysis of the modal features of known genuine products. The genuine probability represents the probability value of the likelihood that a certain type of modal feature belongs to a genuine product. The first probability threshold is a preset probability critical value used as a standard for judging whether a modal feature is genuine or a counterfeit.

[0074] Specifically, first, use the method of small-sample learning to measure the similarity between the modal feature and the typical features of genuine products by calculating the Euclidean distance between the modal feature and the genuine prototype cluster. Then, transform the distance into a genuine probability through a Gaussian function, so that this similarity can be intuitively reflected in the form of a probability. Finally, based on the preset first probability threshold, determine whether the modal feature is genuine or a counterfeit.

[0075] It can be concluded from the above that this embodiment improves the accuracy and robustness of model training by determining the authenticity of each type of modal feature. Subsequently, this embodiment combines the advantages of small-sample learning and can achieve efficient authenticity determination of modal features under limited data resources, providing a basis for the incremental parameter update of the subsequent model.

[0076] In an embodiment of the present disclosure, based on the incremental parameter update strategy, the initial parameters of the multi-modal pre-trained model are adjusted to obtain a target multi-modal large model, including: Based on gradient magnitude, identify target parameters in the initial parameters whose correlation with counterfeits / genuine products is greater than the first correlation threshold; Divide the target parameters into global parameters and local parameters; Fix the global parameters and fine-tune the local parameters until the error value of the target contrast loss function is less than the first error threshold to obtain the target local parameters; Determine the target multi-modal large model based on the global parameters and the target local parameters.

[0077] In this embodiment, the gradient magnitude is a value that measures the size of the gradient and reflects the potential for parameter update. The fake / genuine correlation is the degree of association between the model parameters and the features of fake or genuine items in the data. The higher the correlation, the greater the impact of the parameters on the features of fake or genuine items. The fake / genuine correlation can be calculated based on the cosine similarity.

[0078] The first correlation threshold is a pre-set critical correlation value used to determine whether the correlation between the parameters and the fake / genuine items is high enough. The target parameters are selected from the initial parameters and have a correlation with the fake / genuine items greater than the first correlation threshold.

[0079] Global parameters are crucial for the overall performance of the model and the macroscopic feature representation of multi-modal data. They are kept fixed during the fine-tuning process to avoid destroying the general features learned by the model. Local parameters are related to the learning and representation of local detailed features of the model and can be adjusted during the fine-tuning process to optimize the model's ability to handle specific local features.

[0080] The calculation formula of the target contrast loss function is:

[0081] where, is the value of the target contrast loss function, is the weight coefficient of the global contrast loss function, is the weight coefficient of the local contrast loss function, . When focusing on the global optimum, . When focusing on the local optimum, .

[0082] The first error threshold is a pre-set critical error value. When the error value of the target contrast loss function is less than this threshold, it is considered that the model training has reached the accuracy requirement.

[0083] The target multi-modal large model is a trained model obtained after adjusting and optimizing the parameters of the multi-modal pre-trained model. Specifically, the fixed global parameters and the target local parameters obtained through fine-tuning are integrated to update the parameters of the multi-modal pre-trained model, thereby obtaining the target multi-modal large model, enabling it to better complete tasks related to multi-modal data.

[0084] As can be seen from the above, this embodiment adopts an incremental parameter update strategy to adjust the initial parameters of the multi-modal pre-training model, which can accurately locate the target parameters highly relevant to the authenticity of the data, thereby effectively improving the model performance. In this embodiment, the target parameters are divided into two categories: global and local, and are processed separately, which not only retains the overall characteristics of the model, but also realizes the fine optimization of specific information. By fixing the global parameters and fine-tuning the local parameters, it ensures that the model makes adaptive adjustments to the characteristics of counterfeits / genuine products while maintaining its original advantages. This embodiment improves the recognition accuracy of the model.

[0085] Corresponding to the above-mentioned embodiment of the Chinese herbal medicine authentication method based on the multi-modal large model, Figure 2 This is a structural block diagram of a Chinese herbal medicine authentication system based on a multi-modal large model provided by an embodiment of the present disclosure. For the sake of convenience of description, only the parts related to the embodiments of the present disclosure are shown. Refer to Figure 2 The Chinese herbal medicine authentication system 20 based on the multi-modal large model includes: an image preprocessing module 21, a text preprocessing module 22, a consistency multi-modal data determination module 23, and an authenticity identification module 24.

[0086] Among them, the image preprocessing module 21 is used to, in response to the target multi-modal data being image data, perform registration processing on the image data based on spatial matching to obtain multiple types of spatially registered image data; and perform pixel normalization processing on the multiple types of spatially registered image data to obtain multiple types of consistent image data; The text preprocessing module 22 is used to, in response to the target multi-modal data being text data, perform standardization processing on the text data to obtain multiple types of consistent text data; The consistency multi-modal data determination module 23 is used to use the multiple types of consistent image data and the multiple types of consistent text data as the target consistent multi-modal data; the target multi-modal data is obtained by collecting the target Chinese herbal medicine in different ways; The authenticity identification module 24 is used to input the target consistent multi-modal data into the target multi-modal large model to obtain the quality identification result of the target Chinese herbal medicine.

[0087] In an embodiment of the present disclosure, the Chinese herbal medicine authentication system 20 based on the multi-modal large model further includes: a noise reduction processing module, which is used to perform Fourier transform on each type of spatially registered image data to obtain a spectrogram; Calculate the total energy value of the high-frequency region of the spectrogram; the high-frequency region is the region where the frequency is greater than or equal to the frequency threshold; If the total energy value is greater than or equal to the first energy threshold, perform noise reduction processing on the multiple types of spatially registered image data.

[0088] In one embodiment of the present disclosure, the noise reduction processing module is specifically configured to, in response to each type of spatially registered image being an RGB spatially registered image, perform noise reduction processing on the RGB spatially registered image based on the bilateral filtering algorithm; in response to each type of spatially registered image being a hyperspectral spatially registered image, perform noise reduction processing on the hyperspectral spatially registered image based on the spectral smoothing filtering algorithm; in response to each type of spatially registered image being a microscope spatially registered image, perform noise reduction processing on the microscope spatially registered image based on the Wiener filtering algorithm.

[0089] In one embodiment of the present disclosure, the Chinese herbal medicine authentication system 20 based on the multimodal large model further includes: a model training module, configured to perform semantic matching on the historical consistent multimodal data and the pharmacopoeia knowledge base to obtain multiple data pairs; extract features from the multiple data pairs to obtain multiple types of modal features; perform pre-training processing on the multiple types of modal features to obtain a multimodal pre-training model; adjust the parameters of the multimodal pre-training model based on the incremental parameter update strategy to obtain the target multimodal large model.

[0090] In one embodiment of the present disclosure, the model training module is specifically configured to divide the multiple types of modal features into positive samples and negative samples; perform global optimization on the positive samples and negative samples respectively based on the global contrast loss function; perform local optimization on the positive samples and negative samples respectively based on the local contrast loss function; determine the initial parameters of the multimodal large model based on the global optimization and local optimization; determine the multimodal pre-training model based on the initial parameters of the multimodal large model.

[0091] In one embodiment of the present disclosure, the Chinese herbal medicine authentication system 20 based on the multimodal large model further includes: a few-shot learning module, configured to calculate the distance between each type of modal feature and each type of genuine prototype cluster; determine the genuine probability of each type of modal feature based on the distance; in response to the genuine probability being less than the first probability threshold, determine that this type of modal feature is a counterfeit; in response to the genuine probability being greater than or equal to the first probability threshold, determine that this type of modal feature is genuine.

[0092] In one embodiment of the present disclosure, the model training module is specifically further configured to identify target parameters in the initial parameters whose correlation with counterfeits / genuines is greater than the first correlation threshold based on the gradient magnitude; divide the target parameters into global parameters and local parameters; Fix the global parameters and fine-tune the local parameters until the error value of the target contrast loss function is less than the first error threshold, obtaining the target local parameters; Determine the target multi-modal large model based on the global parameters and the target local parameters.

[0093] See Figure 3 , Figure 3 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. As Figure 3 shown, the electronic device 300 in this embodiment may include: one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The above-mentioned processors 301, input devices 302, output devices 303, and memories 304 communicate with each other through a communication bus 305. The memory 304 is used to store computer programs, and the computer programs include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. Among them, the processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned system embodiments, such as Figure 2 shown, the functions of the image preprocessing module 21, the text preprocessing module 22, the consistency multi-modal data determination module 23, and the authenticity identification module 24.

[0094] It should be understood that in the embodiments of the present disclosure, the so-called processor 301 may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0095] The input device 302 may include a touchpad, a fingerprint acquisition sensor (for acquiring the fingerprint information and the direction information of the fingerprint of the user), a microphone, etc., and the output device 303 may include a display (such as an LCD), a speaker, etc.

[0096] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A part of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0097] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure may execute the implementation manners described in the first and second embodiments of the Chinese herbal medicine authentication method based on a multimodal large model provided by the embodiments of the present disclosure, and may also execute the implementation manner of the electronic device described in the embodiments of the present disclosure, which will not be elaborated herein.

[0098] In another embodiment of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, all or part of the processes in the above-mentioned embodiment methods are implemented. It can also be completed by instructing relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0099] The computer-readable storage medium may be an internal storage unit of the electronic device in any of the foregoing embodiments, such as the hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device. Further, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store the data that has been output or will be output.

[0100] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this disclosure.

[0101] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described electronic devices and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0102] In several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection to each other can be an indirect coupling or communication connection through some interfaces or units, or can also be an electrical, mechanical, or other form of connection.

[0103] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of this disclosure.

[0104] In addition, the functional units in each embodiment of this disclosure can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0105] The above is only the specific implementation manner of this disclosure, but the protection scope of this disclosure is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed by this disclosure, and these modifications or substitutions should all be covered by the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be subject to the protection scope of the claims.

Claims

1. A method for identifying counterfeit Chinese herbal medicines based on a multimodal large model, characterized in that: include: In response to the target multimodal data being image data, performing registration processing on the image data based on spatial matching to obtain multiple types of spatially registered image data; and performing pixel normalization processing based on the multiple categories of spatially registered image data to obtain multiple categories of consistent image data; In response to the target multimodal data being text data, the text data is standardized to obtain multiple categories of consistent text data; Using the multiple categories of consistent image data and the multiple categories of consistent text data as target consistent multimodal data; The target multimodal data is obtained by collecting the target Chinese herbal medicine in different ways; The target consistency multimodal data is input into a target multimodal macro model to obtain a quality identification result of the target Chinese herbal medicine.

2. The method for identifying counterfeit Chinese herbal medicines based on a multimodal large model as claimed in claim 1, characterized in that: Before performing pixel normalization processing based on the multi-class spatial registration image data, the method further includes: Perform Fourier transform on each type of spatial registration image data to obtain a spectrum diagram; Calculate the total energy of the high-frequency region of the spectrum graph to obtain a total energy value; the high-frequency region is a region where the frequency is greater than or equal to the frequency threshold; If the total energy value is greater than or equal to the first energy threshold, noise reduction processing is performed on the multi-class spatial registration image data.

3. The method for identifying counterfeit Chinese herbal medicines based on a multimodal large model as claimed in claim 2, characterized in that: The performing noise reduction processing on the multiple types of spatially registered image data comprises: In response to each type of spatial registration image being an RGB spatial registration image, performing noise reduction processing on the RGB spatial registration image based on a bilateral filtering algorithm; In response to each type of spatial registration image being a hyperspectral spatial registration image, performing noise reduction processing on the hyperspectral spatial registration image based on a spectral smoothing filter algorithm; In response to each type of spatial registration image being a microscope spatial registration image, noise reduction processing is performed on the microscope spatial registration image based on a Wiener filtering algorithm.

4. The method for identifying counterfeit Chinese herbal medicines based on a multimodal large model as claimed in claim 1, characterized in that: The training process of the target multimodal large model includes: Semantically match the historically consistent multimodal data with the pharmacopoeia knowledge base to obtain multiple data pairs; Performing feature extraction on the multiple data pairs to obtain multiple types of modal features; Performing pre-training processing on the multiple types of modal features to obtain a multi-modal pre-training model; Based on an incremental parameter update strategy, the parameters of the multimodal pre-trained model are adjusted to obtain a target multimodal large model.

5. The method for identifying counterfeit Chinese herbal medicines based on a multimodal large model as claimed in claim 4, characterized in that: The pre-training of the multiple types of modal features to obtain a multi-modal pre-training model includes: Dividing the multi-class modal features into positive samples and negative samples; Performing global optimization on the positive sample and the negative sample respectively based on a global contrast loss function; Based on a local contrast loss function, locally optimize the positive sample and the negative sample respectively; Determining initial parameters of the multimodal large model based on the global optimization and the local optimization; A multimodal pre-trained model is determined based on the initial parameters of the multimodal large model.

6. The method for identifying counterfeit Chinese herbal medicines based on a multimodal large model as claimed in claim 4, characterized in that: Before adjusting the initial parameters of the multimodal pre-training model based on the incremental parameter update strategy, the method further includes: Calculate the distance between each type of modal feature and each type of authentic prototype cluster; Determining the authenticity probability of each type of modal feature based on the distance; In response to the authenticity probability being less than a first probability threshold, determining the type of modal features as counterfeit; In response to the authenticity probability being greater than or equal to a first probability threshold, the type of modal features is determined to be authentic.

7. The method for identifying counterfeit Chinese herbal medicines based on a multimodal large model as claimed in claim 6, characterized in that: The method of adjusting the initial parameters of the multimodal pre-trained model based on the incremental parameter update strategy to obtain a target multimodal large model includes: Identify, based on the gradient amplitude, a target parameter in the initial parameters whose correlation with the counterfeit / authentic product is greater than a first correlation threshold; Dividing the target parameters into global parameters and local parameters; Fixing the global parameters and fine-tuning the local parameters until the error value of the target contrast loss function is less than a first error threshold, thereby obtaining the target local parameters; The target multimodal large model is determined based on the global parameters and the target local parameters.

8. A Chinese herbal medicine authentication system based on a multimodal large model, characterized in that: include: An image preprocessing module, for performing registration processing on the image data based on spatial matching in response to the target multimodal data being image data, to obtain multiple types of spatially registered image data; and performing pixel normalization processing based on the multiple categories of spatially registered image data to obtain multiple categories of consistent image data; A text preprocessing module, for standardizing the text data to obtain multiple types of consistent text data in response to the target multimodal data being text data; A consistent multimodal data determination module, configured to use the multiple categories of consistent image data and the multiple categories of consistent text data as target consistent multimodal data; The target multimodal data is obtained by collecting the target Chinese herbal medicine in different ways; The authenticity identification module is used to input the target consistency multimodal data into the target multimodal macro model to obtain the quality identification result of the target Chinese herbal medicine.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images