A sample data expansion method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses a sample data expansion method, device, electronic device and storage medium, which relates to the field of digital imaging and deep learning technology, wherein the method comprises: obtaining digital microscopic imaging and annotation of a sample data set as a training set, designing an initial data expansion model, obtaining an output image according to the initial data expansion model, the initial data expansion model comprises a feature extraction model, a feature generation model and a feature discrimination model, using the feature discrimination model to compare the input image and the output image to obtain a loss value, optimizing the feature generation model according to the loss value, using the training set to iteratively optimize the initial data expansion model until the set conditions are met, obtaining a trained data expansion model, and using the data expansion model to expand the sample data set. The present invention solves the problem that the prior art can only perform model training on a small sample data set, is prone to overfitting, has poor generalization ability, and is difficult to achieve good results.

Description

Technical Field

[0001] The present invention relates to the field of digital imaging and deep learning technology, and in particular to a sample data expansion method, device, electronic device and storage medium. Background Art

[0002] Traditional identification of Chinese medicinal materials requires manual identification by experienced experts. This process not only requires a lot of manpower and material resources, but the accuracy of the identification results also depends largely on the experience and level of the identification personnel. It has problems such as strong subjectivity, low efficiency, and easy errors.

[0003] To address these issues, the application of cutting-edge technologies such as computer image recognition to intelligently identify traditional Chinese medicines has become a current research hotspot. By collecting digital images of traditional Chinese medicines and using artificial intelligence techniques such as deep learning for feature extraction and classification, rapid and accurate identification of traditional Chinese medicines can be achieved.

[0004] However, due to the wide variety and diverse forms of Chinese medicinal materials, and the lack of publicly available large-scale datasets, deep learning models for Chinese herbal medicine quality inspection tasks are often trained only on small sample datasets. This leads to overfitting and poor generalization, making it difficult to achieve good results in practical applications.

[0005] Therefore, there is an urgent need for a sample data expansion method that can improve the generalization ability and recognition accuracy of deep learning models in Chinese herbal medicine quality inspection tasks, reduce dependence on large amounts of labeled data, and improve the efficiency and accuracy of Chinese herbal medicine identification. Summary of the Invention

[0006] The embodiments of the present invention provide a sample data expansion method to address the problem in the prior art that deep learning models can only be performed on small sample data sets, are prone to overfitting, have poor generalization ability, and are difficult to achieve good results in practical applications. The technical solution is as follows:

[0007] According to one aspect of the present invention, a sample data expansion method comprises: obtaining digital microscopic imaging and annotation of a sample data set as a training set; the annotation comprises category labels, target location annotations and feature classification annotations; designing an initial data expansion model based on digital imaging and deep learning technology, taking the sample images in the training set as input images, and obtaining an output image according to the initial data expansion model; the initial data expansion model comprises a feature extraction model, a feature generation model and a feature discrimination model; using the feature discrimination model to compare the input image and the output image to obtain a loss value, and optimizing the feature generation model according to the loss value; using the training set to iteratively optimize the initial data expansion model until the set conditions are met, thereby obtaining a trained data expansion model, and using the data expansion model to expand the sample data set.

[0008] In one embodiment, obtaining digital microscopic imaging and annotation of a sample data set as a training set is achieved by the following steps: each sample in the sample data set of the classification task carries a category label, and each sample in the sample data set of the detection task carries a target positioning annotation and a feature classification annotation; the target positioning annotation is used to indicate the position coordinates of the sample image features; the feature classification annotation is used to indicate the image features of the sample image.

[0009] In one embodiment, obtaining an output image according to the initial data expansion model is achieved by the following steps: using the feature extraction model to extract features from the input image to obtain a feature map, and using the feature generation model to obtain an output image based on the feature map and noise information; the features include overall structural information and intra-class invariant information of the target in the image.

[0010] In one embodiment, the feature extraction model is used to extract features from the input image to obtain a feature map by the following steps: dividing the input image into image blocks and recording position codes, randomly masking some of the image blocks to obtain masked image blocks, and shuffling the order of the remaining image blocks to obtain an image block sequence; performing multi-layer feature extraction on the image block sequence using the feature extraction model to obtain a feature map; the feature extraction model includes multiple feature extraction layers connected by a residual structure; the feature extraction layer is formed by stacking a multi-head self-attention layer and a fully connected layer.

[0011] In one embodiment, the feature generation model is used to obtain an output image based on the feature map and noise information by the following steps: performing a multi-head self-attention operation on the feature map to obtain the mean and standard deviation of the Gaussian distribution, adding the noise vector to the mean of the Gaussian distribution and then multiplying it by the standard deviation of the Gaussian distribution to obtain a feature vector; adding the mask image block to the feature vector in the form of pure noise, and resetting each image block according to the position code; using a multi-layer feature mapping layer to obtain a generated image block sequence according to the feature vector, and combining the generated image block sequence according to the position code to obtain an output image; the feature mapping layer is composed of a stack of multi-head self-attention layers and fully connected layers connected by a residual structure.

[0012] In one embodiment, performing a multi-head self-attention operation on the feature map to obtain the mean and standard deviation of the Gaussian distribution is achieved by the following steps: using a query weight matrix, a key weight matrix and a value weight matrix to generate corresponding query vectors, key vectors and value vectors according to the feature map and dividing them into multiple parts; mapping the divided query vectors, key vectors and value vectors through a fully connected layer to obtain multiple vectors, and performing a fusion calculation on the multiple vectors to obtain multiple attention weights; splicing the multiple attention weights to obtain an overall attention weight, and mapping the overall attention weight through a fully connected layer to obtain the mean and standard deviation of the Gaussian distribution.

[0013] In one embodiment, the input image and the output image are compared using the feature discrimination model to obtain a loss value, and the feature generation model is optimized according to the loss value by the following steps: according to the degree of overfitting of the feature discrimination model, part of the output image is used as the input image, and the feature discrimination model is used to obtain a discrimination result based on the input image and the output image; the discrimination result includes a real image and a generated image; the feature discrimination model includes a multi-layer multi-head self-attention layer and a fully connected layer; the reconstruction loss is calculated by comparing the distance between each pixel point of the input image and the output image; the reconstruction loss is used to ensure that the overall layout of the output image is the same as that of the input image; the discrimination loss is calculated according to the discrimination result, the total loss is obtained according to the sum of the reconstruction loss and the discrimination loss, and the feature generation model is optimized according to the total loss.

[0014] According to one aspect of the present invention, a sample data expansion device includes: a training set acquisition module for acquiring digital microscopic imaging and annotation of a sample data set as a training set; a model construction module for designing an initial data expansion model based on digital imaging and deep learning technology, taking the sample images in the training set as input images, and obtaining an output image according to the initial data expansion model; the initial data expansion model includes a feature extraction model, a feature generation model and a feature discrimination model; a model optimization module for using the feature discrimination model to compare the input image and the output image to obtain a loss value, and optimizing the feature generation model according to the loss value; a sample data expansion module for iteratively optimizing the initial data expansion model using the training set until the set conditions are met, thereby obtaining a trained data expansion model, and using the data expansion model to expand the sample data set.

[0015] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements a sample data expansion method as described above.

[0016] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement a sample data expansion method as described above.

[0017] The beneficial effects brought about by the technical solution provided by the present invention are:

[0018] In the above technical solution, the present invention first obtains digital microscopic imaging and annotation of a sample data set as a training set, designs an initial data expansion model based on digital imaging and deep learning technology, takes the sample image in the training set as the input image, obtains the output image according to the initial data expansion model, and the initial data expansion model includes a feature extraction model, a feature generation model and a feature discrimination model. The feature discrimination model is used to compare the input image and the output image to obtain a loss value, and the feature generation model is optimized according to the loss value. The training set is used to iteratively optimize the initial data expansion model until the set conditions are met to obtain a trained data expansion model, and the data expansion model is used to optimize the sample data. The data set is expanded, and the collaborative work of feature extraction, generation and discrimination models is used to achieve efficient and accurate expansion of the sample data set. The feature discrimination model's precise comparison of input and output images effectively guides the optimization of the feature generation model, thereby improving the quality of the expanded data. After iterative training, the resulting data expansion model can generate highly realistic images, significantly enhancing the diversity and richness of the original data set, providing strong support for the training of deep learning models, thereby effectively solving the problem that deep learning models in existing technologies can only be performed on small sample data sets, are prone to overfitting, have poor generalization ability, and are difficult to achieve good results in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0020] Figure 1 is a flow chart showing a method for expanding sample data according to an exemplary embodiment;

[0021] Figure 2 is a flow chart of a sample data expansion method shown in an exemplary embodiment;

[0022] Figure 3 yes Figure 2 A schematic diagram of the structure of the feature extraction model in the corresponding embodiment;

[0023] Figure 4 yes Figure 2 A schematic diagram of the structure of the feature generation model in the corresponding embodiment;

[0024] Figure 5 yes Figure 2 A schematic structural diagram of a feature discrimination model in the corresponding embodiment;

[0025] Figure 6This is an example of digital imaging of a dodder sample and digital microscopic imaging of palisade cells of southern dodder in an application scenario;

[0026] Figure 7 yes Figure 6 Example diagram of labeled palisade cells of Cuscuta australis in the corresponding application scenario;

[0027] Figure 8 yes Figure 6 Comparison of images generated by the present invention and prior art in corresponding application scenarios;

[0028] Figure 9 is a block diagram of a sample data expansion device according to an exemplary embodiment;

[0029] Figure 10 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0030] Figure 11 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0032] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0033] To this end, the present invention provides a sample data expansion method that achieves efficient and accurate expansion of sample data sets through the collaborative work of feature extraction, generation, and discrimination models. This method can effectively solve the problem that deep learning models in the prior art can only be performed on small sample data sets, are prone to overfitting, have poor generalization capabilities, and are difficult to achieve good results in practical applications. The sample data expansion method is suitable for use in a sample data expansion device, which can be an electronic device. A sample data expansion method in an embodiment of the present invention can be applied to a variety of scenarios, such as expanding small sample data of Chinese herbal medicines.

[0034] See also Figure 1 , an embodiment of the present invention provides a sample data expansion method, which is applicable to electronic devices.

[0035] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0036] like Figure 1 As shown, the method may include the following steps:

[0037] Step 110 : Obtain digital microscopic imaging and annotation of a sample data set as a training set.

[0038] Among them, annotations include category labels, target location annotations and feature classification annotations.

[0039] In one possible implementation, each sample in the sample data set of the classification task carries a category label, and each sample in the sample data set of the detection task carries a target location annotation and a feature classification annotation.

[0040] Among them, the target positioning annotation is used to indicate the position coordinates of the sample image features, and the feature classification annotation is used to indicate the image features of the sample image.

[0041] Step 130: Design an initial data expansion model based on digital imaging and deep learning technology, use the sample images in the training set as input images, and obtain an output image according to the initial data expansion model.

[0042] Among them, the initial data expansion model includes a feature extraction model, a feature generation model and a feature discrimination model.

[0043] In one possible implementation, a feature extraction model is used to extract features from an input image to obtain a feature map, and a feature generation model is used to obtain an output image based on the feature map and noise information.

[0044] In one possible implementation, the input image is divided into image blocks and the position codes are recorded. Some image blocks in the image blocks are randomly masked to obtain masked image blocks. The remaining image blocks are shuffled to obtain an image block sequence. The image block sequence is subjected to multi-layer feature extraction by a feature extraction model to obtain a feature map.

[0045] Among them, the feature extraction model includes multiple feature extraction layers connected by a residual structure. The feature extraction layer is composed of a stack of multi-head self-attention layers and fully connected layers. The features include the overall structural information of the target in the image and the invariant information within the class.

[0046] In one possible implementation, a multi-head self-attention operation is performed on the feature map to obtain the mean and standard deviation of the Gaussian distribution, the noise vector is added to the mean of the Gaussian distribution and then multiplied by the standard deviation of the Gaussian distribution to obtain a feature vector, the mask image block is added to the feature vector in the form of pure noise, and each image block is reset according to the position code, a multi-layer feature mapping layer is used to obtain a generated image block sequence according to the feature vector, and the generated image block sequence is combined according to the position code to obtain the output image.

[0047] Among them, the feature mapping layer is composed of a stack of multi-head self-attention layers and fully connected layers connected by a residual structure.

[0048] In one possible implementation, the query weight matrix, key weight matrix, and value weight matrix are used to generate corresponding query vectors, key vectors, and value vectors according to the feature map and split into multiple parts. The split query vectors, key vectors, and value vectors are mapped through a fully connected layer to obtain multiple vectors. The multiple vectors are fused and calculated to obtain multiple attention weights. The multiple attention weights are spliced ​​to obtain the overall attention weight. The overall attention weight is mapped through a fully connected layer to obtain the mean and standard deviation of the Gaussian distribution.

[0049] Step 150: Use the feature discrimination model to compare the input image and the output image to obtain a loss value, and optimize the feature generation model based on the loss value.

[0050] In one possible implementation, part of the output image is used as the input image according to the degree of overfitting of the feature discriminant model. The feature discriminant model is used to obtain a discrimination result based on the input image and the output image. The reconstruction loss is calculated by comparing the distance between each pixel point of the input image and the output image. The discrimination loss is calculated based on the discrimination result. The total loss is obtained by summing the reconstruction loss and the discrimination loss. The feature generation model is optimized based on the total loss.

[0051] Among them, the discrimination results include real images and generated images, the feature discrimination model includes multi-layer multi-head self-attention layers and fully connected layers, and the reconstruction loss is used to ensure that the overall layout of the output image is the same as that of the input image.

[0052] Step 170: Use the training set to iteratively optimize the initial data expansion model until the set conditions are met, obtain a trained data expansion model, and use the data expansion model to expand the sample data set.

[0053] Through the above process, the embodiment of the present invention first obtains digital microscopic imaging and annotation of the sample data set as a training set, designs an initial data expansion model based on digital imaging and deep learning technology, uses the sample image in the training set as the input image, and obtains the output image according to the initial data expansion model. The initial data expansion model includes a feature extraction model, a feature generation model and a feature discrimination model. The feature discrimination model is used to compare the input image and the output image to obtain a loss value, and the feature generation model is optimized according to the loss value. The training set is used to iteratively optimize the initial data expansion model until the set conditions are met to obtain a trained data expansion model, and the data expansion model is used to optimize the sample data. The data set is expanded, and the collaborative work of feature extraction, generation and discrimination models is used to achieve efficient and accurate expansion of the sample data set. The feature discrimination model accurately compares the input and output images, effectively guiding the optimization of the feature generation model, thereby improving the quality of the expanded data. After iterative training, the obtained data expansion model can generate highly realistic images, significantly enhancing the diversity and richness of the original data set, and providing strong support for the training of deep learning models, thereby effectively solving the problem that deep learning models in existing technologies can only be performed on small sample data sets, are prone to overfitting, have poor generalization ability, and are difficult to achieve good results in practical applications.

[0054] In an exemplary embodiment, a training method for expanding small sample data of Chinese herbal medicine based on image generation is provided, which is applied to data expansion of small sample data sets in the field of Chinese herbal medicine quality inspection to expand the data volume and thereby improve the intelligent recognition capabilities of downstream classification and detection models.

[0055] like Figure 2 As shown, the following steps may be specifically included:

[0056] S100, obtain digital imaging and design the initial data expansion model for the small sample dataset of Chinese herbal medicine.

[0057] Among them, the data expansion model of the initial Chinese herbal medicine small sample data set includes feature extraction model, feature generation model and feature discrimination model, and digital imaging has feature annotation.

[0058] Specifically, for samples of classification tasks, each imaging sample has a category label, and for samples of detection tasks, each imaging sample has a target positioning label and a feature classification label.

[0059] Among them, the category label is to classify the digital imaging according to the sample targets therein, and the digital images of each category of samples are classified according to their category label classification. The feature classification annotation includes marking the image features of various types of Chinese herbal medicine samples on the digital imaging, and the target positioning annotation includes the target positioning annotation of the image features on the digital imaging.

[0060] S200: Extract image features of the sample image using a feature extraction model.

[0061] In one possible implementation, digital imaging is masked to improve the global feature extraction capability of the model. At the same time, noise variables are introduced into image generation to enhance the diversity of image generation. The feature extraction model performs multi-head self-attention operations on the image block sequence to obtain a feature map.

[0062] Specifically, the input digital image is cut into a series of 16*16 image blocks and the position code of each block is recorded. Then, a part of the image blocks are randomly masked and the remaining image blocks are shuffled to form an image block sequence and sent to the feature extraction model. Multi-layer feature extraction is performed on the image block sequence through a deep neural network. Each layer of feature extraction is composed of a multi-head self-attention layer and a fully connected layer stacked together. Different feature extraction layers are connected with a residual structure to obtain feature maps corresponding to feature vectors of various dimensions.

[0063] Furthermore, the feature map is mapped to transform the feature vector into a Gaussian distribution, which describes the characteristics of the digital imaging. A multi-head self-attention operation is performed on the feature map using two modules consisting of a multi-head self-attention layer and a fully connected layer. These two modules respectively map the feature vector into the mean and standard deviation of the Gaussian distribution.

[0064] Specifically, the query weight matrix, key weight matrix and value weight matrix are used to generate the query vector, key vector and value vector corresponding to the data, and the query vector, key vector and value vector are respectively divided into h parts, where h is the number of heads of multi-head self-attention. Each query vector, key vector and value vector after division is mapped through a fully connected layer, and each mapped query vector, key vector and value vector are fused respectively. The query vector, key vector and value vector of the i-th part are fused to obtain the attention weight of the i-th region, and h attention weights are obtained. The h attention weights are spliced ​​to obtain the overall attention weight corresponding to the overall data, and the overall attention weight is mapped by a fully connected layer to obtain the output result of this step.

[0065] like Figure 3As shown, the structure of the feature extraction model is demonstrated. The digital imaging of the small sample data set of Chinese herbal medicine is cut and randomly masked, and then 8 layers of feature extraction are performed. Each layer of feature extraction is composed of a stack of multi-head self-attention layers and fully connected layers. Different feature extraction layers are connected with a residual structure to obtain 8 different dimensions of feature maps corresponding to feature vectors, so as to obtain features from local to overall levels. The features of multiple levels are mapped into a Gaussian distribution and output to the feature generation model together. It can provide the network with comprehensive information about the input image, effectively ensure the association between the generated image and the input image, improve the quality of the generated image, and achieve the purpose of data expansion.

[0066] S300, using a feature generation model to generate an image using image features and noise, the generated image has a data expansion effect on the original image.

[0067] In one possible implementation, a noise vector is generated, added to the mean of the Gaussian distribution output by the feature extraction model, and then multiplied by the standard deviation of the Gaussian distribution output by the feature extraction model to obtain a feature vector for image generation. The image blocks masked by the feature extraction module are added to the feature vector in the form of pure noise image blocks, and each image block is reset to the position order after the image slicing operation according to the position coding. Through multi-head self-attention operation, a generated image block sequence is obtained.

[0068] Specifically, a deep neural network is used to perform multi-layer feature mapping on the supplemented and reset feature vectors. Each layer of feature mapping is composed of a stack of multi-head self-attention layers and fully connected layers. Different feature mapping layers are connected with a residual structure to obtain feature maps corresponding to feature vectors of various dimensions. The generated image block sequence is combined into a generated image according to the image encoding corresponding to each image block.

[0069] like Figure 4 The figure shows the structure of the feature generation model. It uses a randomly generated noise vector as inspiration. After being fused with the output of the feature extraction model, the masked image blocks are complemented with noise image blocks and reset. Then, they are gradually decoded into a sequence of image blocks for the generated image in eight feature mapping layers. Each mapping layer is composed of a stack of multi-head self-attention layers and fully connected layers. Finally, the image blocks are spliced ​​together according to position to form the generated image. By combining the noise vector with the Gaussian distribution representing the input image provided by the feature extraction model and the noise image blocks, the generated image can maintain similar features to the input image while maintaining a certain degree of variation, ensuring that the generated image can serve as data augmentation.

[0070] S400, using the feature discrimination model to compare the image generated by the feature generation model with the real image, and discriminate between the two, giving feedback to the feature generation model to improve the generation capability of the feature generation model and the quality of the generated image.

[0071] like Figure 5 As shown in the figure, the structure of the feature discrimination model is shown. The feature discrimination model is a deep neural network composed of multiple layers of multi-head self-attention layers and fully connected layers. The input image is extracted through multiple layers of multi-head self-attention layers and fully connected layers, and discrimination is performed based on the output of the last fully connected layer. The discrimination result is a real image or a generated image.

[0072] Specifically, feature extraction is performed on the input image, and discrimination is performed based on the feature map obtained by feature extraction. The discrimination result is a real image or a generated image. Then, the loss is calculated based on the discrimination result and the real result to provide training guidance for itself and the feature extraction model and feature generation model.

[0073] S500: training a feature extraction model, a feature generation model, and a feature discrimination model to obtain a data expansion model for a small sample dataset of target Chinese herbal medicines.

[0074] Specifically, the real image input to the feature extraction model and the generated image output by the feature recognition model are input into the feature discrimination model. To prevent overfitting problems in small sample scenarios, some generated images will be sent to the discrimination model as real images based on the current overfitting degree of the feature discrimination model to alleviate the overfitting problem of the feature discrimination model. The feature discrimination model discriminates the image and obtains the discrimination results for the real image and the generated image. The discrimination loss of the feature discrimination model is calculated based on the discrimination results to supervise the entire learning process.

[0075] Furthermore, the reconstruction loss is calculated by comparing the L1 distance between each pixel of the real image input to the feature extraction model and the generated image output by the feature generation model. This reconstruction loss can ensure that the generated image has the same overall layout as the input real image to achieve its data expansion goal. At the same time, the generated image is sent to the feature discrimination model for discrimination. The discrimination result will be used to calculate the discrimination loss. The total loss is obtained according to the sum of the reconstruction loss and the discrimination loss of the feature generation model, which is used to supervise the entire learning process.

[0076] In the above process, the embodiment of the present invention uses artificial intelligence technologies such as deep learning to expand the data of small sample data sets of Chinese herbal medicines, expand the data volume of small sample data sets in the field of Chinese herbal medicine quality inspection, so as to improve the generalization ability of deep learning classification and detection models in Chinese herbal medicine quality inspection tasks, thereby improving the recognition accuracy of the model in related work, reducing the work pressure of Chinese herbal medicine quality inspectors and the workload of data collection work required for training deep neural networks, and has significant application prospects. By combining deep learning technology and Chinese herbal medicine identification and detection knowledge, a technology for expanding small sample data of Chinese herbal medicines based on image generation is realized, further improving the accuracy and efficiency of Chinese herbal medicine identification and detection, effectively solving the problem of data scarcity that is prevalent in Chinese herbal medicine intelligent identification tasks, and promoting the application and development of a new generation of artificial intelligence technology in the intelligent detection and identification of Chinese herbal medicines.

[0077] In one application scenario, a neural network model was trained using digital images of thousands of high-quality, high-resolution Chinese herbal medicine samples from multiple datasets acquired through collaboration with a testing institute. The annotation work was completed by several professional Chinese herbal medicine inspectors, and the dataset includes a classification dataset of Cuscuta australis traits and a dataset of palisade cell microstructures of Cuscuta australis.

[0078] like Figure 6 As shown in the figure, examples of digital imaging of Cuscuta species and digital microscopic imaging of palisade cells of Cuscuta australis are shown. Figure 6 (a) is an example of a large sample of dodder. Figure 6 (b) is an example of a salt dodder sample. Figure 6 (c) is an example of a southern dodder sample. Figure 6 (d) in the figure is an example of a green spike amaranth sample. Figure 6 (e) in the figure is an example of a dodder sample; Figure 6 (f) is an example of a complete side view of the palisade cells of Cuscuta australis. Figure 6 (g) is an example of an incomplete side view of the palisade cells of Cuscuta australis. Figure 6 (h) is an example of the surface view of the palisade cells of Cuscuta australis.

[0079] The Cuscuta trait classification dataset contains 500 annotated images of Cuscuta seed traits, and the Southern Cuscuta palisade cell microstructure dataset contains 1,239 digital microscopic images of southern Cuscuta palisade cells. Table 1 shows the number of annotated samples for each type of Cuscuta seed, and Table 2 shows the number of annotated samples for each type of southern Cuscuta palisade cell microstructure.

[0080] like Figure 7 The figure shows an example of annotation of palisade cells of Cuscuta australis, which is the mapping result of the annotation information on the digital microscopic imaging.

[0081] Table 1: Five types of samples and their labeled quantities of Cuscuta seeds

[0082]

[0083] Table 2: Three types of observed morphologies and labeled numbers of palisade cells in Cuscuta australis

[0084]

[0085] As shown in Table 3 below, in an embodiment of the present invention, 250 of the 500 digital images of dodder samples were selected (the remaining samples were used for testing), and the data augmentation model obtained through training achieved an FID score of 42.22 when generating images for this dataset; by selecting 300 of the 1239 digital images of dodder samples (the remaining samples were used for testing), the data augmentation model obtained through training achieved an FID score of 45.78 when generating images for this dataset; Table 3 below shows the detailed test results of this example.

[0086] Table 3: Results of image generation on various datasets by the embodiments of the present invention

[0087]

[0088] As shown in Table 4 below, in an embodiment of the present invention, 250 of the 500 digital images of dodder samples were selected (the remaining samples were used for testing). After the data set was expanded using the data augmentation model proposed in the present invention (doubling the data volume, i.e., expanding to 500 training images), the accuracy of various classification models trained was improved compared to when the data augmentation model proposed in the present invention was not applied.

[0089] As shown in Table 5 below, by selecting 300 of 1,239 digital images of dodder samples (the remaining samples were used for testing), applying the proposed data augmentation model to expand the dataset (doubling the data volume to 600 training images) also improved the accuracy of the detection model trained without the proposed data augmentation model. These results demonstrate that the proposed data augmentation model can effectively expand the dataset to improve the training of downstream classification or detection models.

[0090] Table 4: Test results of intelligent classification of Cuscuta seeds

[0091]

[0092] Table 5: Test results of the intelligent detection model for palisade cells of Cuscuta australis

[0093]

[0094] like Figure 8 As shown, the comparison between the generated image of the embodiment of the present invention and the existing excellent image generation model is shown. Figure 8 (a) is a real sample of Cuscuta australis. Figure 8 (b) is the data expansion model proposed by this invention. Figure 8 (a) is the input image and the generated dodder image is Figure 8 (c) in the figure is a dodder image generated by the StyleGAN2 image generation model after sufficient training on dodder data.

[0095] Among them, since other image generation models have no constraints on the input image, their generated images may introduce features that do not exist in this type of sample. At the same time, changes in the overall layout of the image will also bring additional workload for sample annotation of the detection task. The model proposed in this invention can generate images for the detection task and directly apply the sample annotation of the input image to the generated image.

[0096] Through the above process, the embodiment of the present invention realizes the efficient and accurate expansion of the sample data set through the collaborative work of feature extraction, generation and discrimination models. The feature discrimination model accurately compares the input and output images, effectively guiding the optimization of the feature generation model, thereby improving the quality of the expanded data, and the data expansion effect of the generated image is more obvious. Compared with other image generation models, the image generated by the present invention is based on a noise vector, a random mask and an input image, and the reconstruction loss constraint model is used during the training process so that the generated image has the same overall layout as the input image. Data expansion through image generation improves the training accuracy of downstream tasks on the data set. It requires that the generated images have higher quality and cannot introduce features that do not belong to each type of sample. The present invention has verified through experiments on multiple data sets that the data expansion model proposed by the present invention can bring accuracy improvement to downstream models in classification and detection tasks.

[0097] The following is an embodiment of the device of the present invention, which can be used to perform a sample data expansion method involved in the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the method embodiment of a sample data expansion method involved in the present invention.

[0098] See also Figure 9 , an embodiment of the present invention provides a sample data expansion device 800.

[0099] The apparatus 800 includes but is not limited to: a training set acquisition module 810 , a model construction module 830 , a model optimization module 850 and a sample data expansion module 870 .

[0100] The training set acquisition module 810 is used to acquire digital microscopic imaging and annotation of a sample data set as a training set.

[0101] The model building module 830 is used to design an initial data expansion model based on digital imaging and deep learning technology, take the sample images in the training set as input images, and obtain the output image according to the initial data expansion model. The initial data expansion model includes a feature extraction model, a feature generation model and a feature discrimination model.

[0102] The model optimization module 850 is used to use the feature discrimination model to compare the input image and the output image to obtain a loss value, and optimize the feature generation model according to the loss value.

[0103] The sample data expansion module 870 is used to iteratively optimize the initial data expansion model using the training set until the set conditions are met, obtain a trained data expansion model, and use the data expansion model to expand the sample data set.

[0104] It should be noted that, when expanding the sample data provided in the above embodiment, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the sample data expansion device will be divided into different functional modules to complete all or part of the functions described above.

[0105] In addition, the sample data expansion device provided in the above embodiment and an embodiment of a sample data expansion method belong to the same concept, wherein the specific manner in which each module performs operations has been described in detail in the method embodiment and will not be repeated here.

[0106] Figure 10 The following is a schematic diagram of the structure of an electronic device according to an exemplary embodiment.

[0107] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as needing to rely on or must have Figure 10 One or more components of exemplary electronic device 2000 are shown.

[0108] The hardware structure of the electronic device 2000 may vary greatly due to different configurations or performances, such as Figure 10 As shown, the electronic device 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0109] Specifically, the power supply 210 is used to provide operating voltage for various hardware devices on the electronic device 2000 .

[0110] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 10 As shown, this does not constitute a specific limitation.

[0111] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.

[0112] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0113] The application 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 10 (not shown), each module may respectively include computer-readable instructions for the electronic device 2000. For example, the sample data expansion device may be considered as an application 253 deployed on the electronic device 2000.

[0114] The data 255 may be signal information, etc., and is stored in the memory 250 .

[0115] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on the massive data 255 in the memory 250. For example, a sample data expansion method can be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0116] In addition, the present invention can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or combination thereof.

[0117] See also Figure 11 In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 400 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capabilities.

[0118] exist Figure 11 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0119] The data exchange between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0120] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0121] Processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 4001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0122] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to these.

[0123] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0124] The computer-readable instructions are executed by one or more processors 4001 to implement a sample data expansion method in the above embodiments.

[0125] In addition, an embodiment of the present invention provides a storage medium having computer-readable instructions stored thereon. The computer-readable instructions are executed by one or more processors to implement the sample data expansion method described above.

[0126] In an embodiment of the present invention, a computer program product is provided. The computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements a sample data expansion method as described above.

[0127] Compared with the related art, the present invention has the following beneficial effects:

[0128] 1. The present invention first obtains digital microscopic imaging and annotation of a sample data set as a training set, designs an initial data expansion model based on digital imaging and deep learning technology, takes the sample image in the training set as the input image, obtains the output image according to the initial data expansion model, and the initial data expansion model includes a feature extraction model, a feature generation model and a feature discrimination model. The feature discrimination model is used to compare the input image and the output image to obtain a loss value, and the feature generation model is optimized according to the loss value. The initial data expansion model is iteratively optimized using the training set until the set conditions are met to obtain a trained data expansion model, and the data expansion model is used to perform Through the collaborative work of feature extraction, generation and discrimination models, efficient and accurate expansion of sample data sets is achieved. The feature discrimination model's precise comparison of input and output images effectively guides the optimization of the feature generation model, thereby improving the quality of the expanded data. After iterative training, the resulting data expansion model can generate highly realistic images, significantly enhancing the diversity and richness of the original data set, providing strong support for the training of deep learning models, thereby effectively solving the problem that deep learning models in existing technologies can only be performed on small sample data sets, are prone to overfitting, have poor generalization ability, and are difficult to achieve good results in practical applications.

[0129] 2. The present invention can enhance image diversity: through image generation, it can fuse the image features of multiple images to generate diverse images, thereby enhancing the richness of the data set; it can also improve data expansion effects: the generated images are based on noise vectors, random masks and input images, and the reconstruction loss constraint model is used during the training process, so that the generated images have the same overall layout as the input images, and the data expansion effect is more obvious.

[0130] 3. This invention improves the accuracy of downstream tasks: Data augmentation through image generation can improve the training accuracy of downstream tasks (such as classification and detection) on datasets. Experiments have demonstrated that this method effectively improves model accuracy across multiple datasets. This invention also reduces the annotation workload: For samples in detection tasks, since the generated image has the same overall layout as the input image, the sample annotations from the input image can be directly applied to the generated image, thus reducing the annotation workload.

[0131] 4. The present invention has strong adaptability: it is not only applicable to the problem of small sample data sets in the field of quality inspection of Chinese herbal medicines, but can also be extended to the problem of small sample data sets in other fields, and has strong adaptability.

[0132] 5. This invention can promote the intelligent quality inspection of Chinese herbal medicines: it can realize the intelligent quality inspection of Chinese herbal medicines, improve the efficiency and accuracy of quality inspection, provide strong support for the traditional Chinese medicine industry, combine digital imaging and deep learning technology, and promote technological progress and innovative development in the field of Chinese herbal medicine quality inspection.

[0133] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0134] The above descriptions are only partial embodiments of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A sample data expansion method, characterized in that: The method comprises: Obtaining digital microscopic imaging and annotation of a sample data set as a training set; the annotations include category labels, target location annotations, and feature classification annotations; An initial data expansion model is designed based on digital imaging and deep learning technology, and the sample images in the training set are used as input images; the initial data expansion model includes a feature extraction model, a feature generation model and a feature discrimination model; The input image is divided into image blocks and position codes are recorded, some image blocks in the image blocks are randomly masked to obtain masked image blocks, and the remaining image blocks are shuffled to obtain an image block sequence; The feature extraction model is used to perform multi-layer feature extraction on the image block sequence to obtain a feature map; the feature extraction model includes multiple feature extraction layers, the feature extraction layers are stacked by multi-head self-attention layers and fully connected layers, and each feature extraction layer is connected by a residual structure; the feature map includes the overall structural information of the target in the image and the invariant information within the class; Generate a corresponding query vector, a key vector and a value vector according to the feature graph using a query weight matrix, a key weight matrix and a value weight matrix and divide them into multiple parts; The split query vector, key vector and value vector are mapped through a fully connected layer to obtain multiple vectors, and the multiple vectors are fused and calculated to obtain multiple attention weights; The multiple attention weights are concatenated to obtain an overall attention weight, the overall attention weight is mapped through a fully connected layer to obtain a mean and a standard deviation of a Gaussian distribution, the noise vector is added to the mean of the Gaussian distribution and then multiplied by the standard deviation of the Gaussian distribution to obtain a feature vector; Supplementing the mask image block into the feature vector in the form of pure noise, and resetting each of the image blocks according to the position code; Using a multi-layer feature mapping layer to obtain a generated image block sequence according to the feature vector containing the pure noise, and combining the generated image block sequence according to the position encoding to obtain an output image; the feature mapping layer is formed by stacking a multi-head self-attention layer and a fully connected layer connected by a residual structure; Using the feature discrimination model to compare the input image and the output image to obtain a loss value, and optimizing the feature generation model according to the loss value; The initial data expansion model is iteratively optimized using the training set until a set condition is met to obtain a trained data expansion model, and the sample data set is expanded using the data expansion model.

2. A sample data expansion method as claimed in claim 1, characterized in that: The step of obtaining digital microscopic imaging and annotation of a sample data set as a training set includes: Each sample in the sample data set of the classification task carries a category label, and each sample in the sample data set of the detection task carries a target location annotation and a feature classification annotation; the target location annotation is used to indicate the position coordinates of the sample image feature; the feature classification annotation is used to indicate the image feature of the sample image.

3. A sample data expansion method as claimed in claim 1, characterized in that: The step of using the feature discrimination model to compare the input image and the output image to obtain a loss value, and optimizing the feature generation model according to the loss value, comprises: According to the degree of overfitting of the feature discrimination model, part of the output image is used as the input image, and the feature discrimination model is used to obtain a discrimination result according to the input image and the output image; the discrimination result includes a real image and a generated image; the feature discrimination model includes a multi-layer multi-head self-attention layer and a fully connected layer; The reconstruction loss is calculated by comparing the distance between each pixel point of the input image and the output image; the reconstruction loss is used to ensure that the overall layout of the output image is the same as that of the input image; The discrimination loss is calculated according to the discrimination result, the total loss is obtained according to the sum of the reconstruction loss and the discrimination loss, and the feature generation model is optimized according to the total loss.

4. A sample data expansion device, characterized in that: The device is applied to the sample data expansion method according to any one of claims 1 to 3, and the device comprises: A training set acquisition module, used to acquire digital microscopic imaging and annotation of a sample data set as a training set; A model building module, used to design an initial data expansion model based on digital imaging and deep learning technology, taking the sample image in the training set as an input image, and obtaining an output image according to the initial data expansion model; the initial data expansion model includes a feature extraction model, a feature generation model and a feature discrimination model; A model optimization module, used to compare the input image and the output image using the feature discrimination model to obtain a loss value, and optimize the feature generation model according to the loss value; The sample data expansion module is used to iteratively optimize the initial data expansion model using the training set until the set conditions are met, obtain a trained data expansion model, and use the data expansion model to expand the sample data set.

5. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements a sample data expansion method as described in any one of claims 1 to 3.

6. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement a sample data expansion method as described in any one of claims 1 to 3.

Images