Drug image recognition method, electronic device, and storage medium
By training a location detection network and data augmentation processing, and combining convolutional neural networks and recurrent neural networks to extract drug image features, the problem of poor accuracy in drug image identification was solved, and efficient drug image identification was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FULIAN PRESION ELECTRONICS (TIANJIN) CO LTD
- Filing Date
- 2021-12-16
- Publication Date
- 2026-06-09
Smart Images
Figure CN116434256B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing, and more particularly to a method for drug image recognition, an electronic device, and a storage medium. Background Technology
[0002] Current methods for drug image recognition require a significant amount of manpower to label drug data. Insufficient training data can also lead to poor recognition accuracy of the training model. Therefore, improving the accuracy of drug image recognition has become an urgent technical problem to be solved. Summary of the Invention
[0003] In view of the above, it is necessary to provide a drug image recognition method, electronic device and storage medium that can solve the technical problem of difficulty in accurately and efficiently recognizing drug images.
[0004] This application provides a drug image recognition method, the drug image recognition method comprising:
[0005] Acquire multiple images of the drug and the drug to be tested;
[0006] Obtain a pre-trained drug detection model, which includes a location detection network, a text recognition network, and a category recognition network;
[0007] The multiple drug images are input into the location detection network to obtain multiple target images, each containing an image of a single drug.
[0008] Based on the multiple target images and the text recognition network, multiple text feature matrices are generated;
[0009] The multiple target images are input into the category recognition network to obtain multiple image feature matrices;
[0010] A reference matrix is generated based on each image feature matrix and its corresponding text feature matrix;
[0011] The drug detection model is used to process the drug image to obtain the drug matrix.
[0012] The identification result of the drug image to be tested is generated based on the similarity between the test matrix and each reference matrix.
[0013] According to a preferred embodiment of this application, before inputting the plurality of drug images into the location detection network to obtain a plurality of target images, the drug image recognition method further includes:
[0014] A location detection learner and a location image are acquired, wherein the location image includes a first image and a second image, and the second image includes multiple labeled images and multiple unlabeled images;
[0015] The location detection learner is trained using the first image to obtain a first pre-trained network;
[0016] The first pre-trained network is adjusted based on the multiple labeled images to obtain the first labeled network;
[0017] The multiple unlabeled images are input into the first labeling network to obtain the output images and the predicted probability value of the drug contained in each output image;
[0018] The output image corresponding to the predicted probability value that is greater than a preset threshold is used to adjust the first labeling network to obtain the second labeling network;
[0019] Calculate the first loss value of the second labeling network, and adjust the second labeling network multiple times based on the first loss value until the first loss value drops to the minimum, and then stop adjusting to obtain the location detection network.
[0020] According to a preferred embodiment of this application, before generating multiple text feature matrices based on the multiple target images and the text recognition network, the drug image recognition method further includes:
[0021] Acquire text images and a text recognition learner, wherein the text images include a third image and a fourth image;
[0022] The text recognition learner is trained using the third image to obtain a second pre-trained network, wherein the second pre-trained network includes a convolutional neural network model and a recurrent neural network model.
[0023] Calculate the second loss value of the second pre-trained network, and backpropagate using the second loss value to adjust the parameters of the second pre-trained network multiple times until the second pre-trained model reaches convergence and the adjustment stops, thus obtaining the text recognition network.
[0024] According to a preferred embodiment of this application, generating multiple text feature matrices based on the multiple target images and the text recognition network includes:
[0025] Each target image is color-converted to obtain multiple grayscale text images;
[0026] Each grayscale image of the text is binarized to obtain multiple binarized images;
[0027] Each binarized image is filtered to obtain multiple filtered images;
[0028] Locate the position of the drug text in each filtered image to obtain the text position;
[0029] The text image is selected from each target image based on the text position;
[0030] Each text image is input into the convolutional neural network model for feature extraction to obtain a feature sequence;
[0031] The feature sequence is input into the recurrent neural network model to obtain the multiple text feature matrices.
[0032] According to a preferred embodiment of this application, before inputting the multiple target images into the category identification network to obtain multiple image feature matrices, the drug image recognition method further includes:
[0033] Obtain a category recognition learner, wherein the category recognition learner uses the soft-max function as the activation function;
[0034] Calculate the third loss value of the category recognition learner, and adjust the category recognition learner based on the third loss value until the third loss value drops to the minimum, then stop adjusting and remove the activation function from the adjusted category recognition learner to obtain the category recognition network.
[0035] According to a preferred embodiment of this application, calculating the third loss value of the category recognition learner includes:
[0036] Acquire multiple category images, wherein the multiple category images include multiple categories;
[0037] The multiple category images are subjected to data augmentation processing to obtain multiple augmented images;
[0038] The formula for determining the third loss value is as follows:
[0039]
[0040] in, 2 refers to the third loss value, i refers to the multiple augmented images, yi refers to the category of the i-th augmented image, j refers to the j-th augmented image among the multiple augmented images, yj refers to the category of the j-th augmented image, and N refers to the category of the j-th augmented image. yi This refers to the number of all augmented images of the same category as i, || i≠j The first indicator function takes the value zero if and only if i = j, and takes the value 1 if i ≠ j. yi＝yj The second indicator function takes the value zero if and only if yi = yj, and takes the value 1 if yi ≠ yj. i≠k The third indicator function, z, takes zero if and only if i = k, and takes 1 if i ≠ k. iThis refers to the unit vector z obtained by inputting i into the category recognition network. j This refers to the unit vector obtained by inputting j into the category recognition network, k refers to any augmented image other than i, and z k It refers to the unit vector obtained by inputting k into the category recognition network, and τ is a preset scalar adjustment parameter.
[0041] According to a preferred embodiment of this application, generating a reference matrix based on each image feature matrix and the corresponding text feature matrix includes:
[0042] The reference matrix is obtained by adding each image feature matrix and its corresponding text feature matrix together, wherein each image feature matrix and its corresponding text feature matrix have the same number of rows and columns.
[0043] According to a preferred embodiment of this application, the step of generating the identification result of the drug image based on the similarity between the test matrix and each reference matrix includes:
[0044] Calculate the similarity between the matrix to be tested and each reference matrix;
[0045] The reference matrix corresponding to the highest similarity is determined as the target matrix;
[0046] The target matrix is mapped based on a preset label mapping table to obtain the identification result.
[0047] This application provides an electronic device, the electronic device comprising:
[0048] Memory, storing at least one instruction; and
[0049] The processor executes the at least one instruction to implement the drug image recognition method.
[0050] This application provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the drug image recognition method.
[0051] As can be seen from the above technical solution, the location detection network is obtained by repeatedly adjusting the second pre-trained network using the multiple labeled images. Since the location detection network learns the features of the multiple labeled images, it can mark the location of the image to be tested. When training the category recognition network, data augmentation processing is performed on the multiple category images to avoid the problem of insufficient training data. Multiple reference matrices are generated based on each image feature matrix and the corresponding text feature matrix. The image to be tested is used to generate the test matrix in the same way. The similarity between the test matrix and each reference matrix is calculated, and the label information of the reference matrix corresponding to the largest similarity is selected as the recognition result. Since the test matrix contains both the text features and image features of the image to be tested, it can comprehensively reflect the features of the image to be tested, thereby making the recognition result more accurate. Attached Figure Description
[0052] Figure 1 This is an application environment diagram of a preferred embodiment of the drug image recognition method of this application.
[0053] Figure 2 This is a flowchart of a preferred embodiment of the drug image recognition method of this application.
[0054] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the drug image recognition method of this application. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0056] like Figure 1 The diagram illustrates the application environment of a preferred embodiment of the drug image recognition method of this application. The drug image recognition method can be applied to one or more electronic devices 1, which communicate with a camera device 2. The camera device 2 can be a webcam or other device for capturing images, such as a drug to be tested, which can be photographed to obtain an image of the drug. The drug to be tested can be a capsule or tablet, such as amoxicillin capsules or clarithromycin dispersible tablets.
[0057] The electronic device 1 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0058] The electronic device 1 can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.
[0059] The electronic device 1 may further include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.
[0060] The network where the electronic device 1 is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).
[0061] like Figure 2 The diagram shown is a flowchart of a preferred embodiment of a drug image recognition method according to this application. Depending on different needs, the order of the steps in this flowchart can be adjusted according to actual detection requirements, and some steps can be omitted. The method is executed by an electronic device, such as... Figure 1 Electronic device 1 shown.
[0062] S10: Acquire multiple drug images and images of the drug to be tested.
[0063] In at least one embodiment of this application, the plurality of drug images refers to drug images carrying label information, and the plurality of drug images can be used to generate a reference matrix.
[0064] The label information may include, but is not limited to: the name of the drug, the category of the drug, and the usage of the drug.
[0065] In at least one embodiment of this application, the drug image to be tested refers to a drug image that does not carry the label information, and the surface of the drug to be tested in the drug image to be tested has drug text, which can be letters or numbers.
[0066] In at least one embodiment of this application, the electronic device obtains the plurality of drug images and the tag information corresponding to each drug image from a pre-set target database.
[0067] In at least one embodiment of this application, the electronic device controls the camera device 2 to capture images of the drug to be tested, thereby obtaining an image of the drug to be tested.
[0068] The camera device 2 can be a camera.
[0069] S11, Obtain a pre-trained drug detection model, which includes a location detection network, a text recognition network, and a category recognition network.
[0070] In at least one embodiment of this application, the drug detection model refers to a network model that detects the position of drugs in the image to be tested and in each drug image.
[0071] In at least one embodiment of this application, the location detection network is used to select an image of a single drug from each drug image.
[0072] In at least one embodiment of this application, the text recognition network can be used to acquire text information in each drug image.
[0073] In at least one embodiment of this application, the category recognition network can be used to identify the type of drug in each drug image.
[0074] S12, input the multiple drug images into the location detection network to obtain multiple target images, each target image containing an image of a single drug.
[0075] In at least one embodiment of this application, the target image refers to an image containing a single drug selected from each drug image.
[0076] In at least one embodiment of this application, before inputting the plurality of drug images into the location detection network to obtain a plurality of target images, the drug image recognition method further includes:
[0077] The electronic device acquires a location detection learner and a location image. The location image includes a first image and a second image. The second image includes multiple labeled images and multiple unlabeled images. The electronic device uses the first image to train the location detection learner to obtain a first pre-trained network. Based on the multiple labeled images, the first pre-trained network is adjusted to obtain a first labeled network. The electronic device inputs the multiple unlabeled images into the first labeled network to obtain an output image and a predicted probability value of the drug contained in each output image. The first labeled network is adjusted based on the output images corresponding to predicted probability values greater than a preset threshold to obtain a second labeled network. Further, the electronic device calculates a first loss value for the second labeled network and adjusts the second labeled network multiple times based on the first loss value until the first loss value drops to its minimum, and then stops adjusting to obtain the location detection network.
[0078] Specifically, the electronic device adjusts the second labeling network multiple times based on the first loss value until the first loss value drops to its minimum, at which point the adjustment stops, resulting in the location detection network comprising:
[0079] The electronic device inputs the output image corresponding to the predicted probability value that is greater than a preset threshold into the first labeling network for training, and iteratively updates the weights of the first labeling network until the first labeling network converges, thereby obtaining the location detection network.
[0080] The location detection learner can be an efficient target detector (efficientDet), which can be used to accurately locate the position of a single drug in each drug image. The first image refers to an image obtained from a pre-set first database, which can contain any object. The first database can be a database such as COCO, ImageNet, or CPTN. The first image includes images of various types of objects, such as animals (e.g., puppies, kittens) and plants (e.g., flowers, trees). The second image refers to an image containing a drug, which can be obtained from a pre-set second database. The multiple labeled images refer to images where the positions of the drugs in the images have been labeled, and the multiple unlabeled images refer to images where the positions of the drugs in the images have not been labeled.
[0081] The output image refers to an image that may include a single drug, and the output image can be used to make multiple adjustments to the first labeling network.
[0082] The predicted probability value refers to the probability that the output image contains a single drug.
[0083] The preset threshold can be set by the user, and this application does not impose any restrictions on it.
[0084] The first pre-trained network refers to the network obtained after pre-training using the first image.
[0085] The first labeling network refers to the network obtained by retraining the first pre-trained network based on the multiple labeled images. The first labeling network can be used to label the multiple unlabeled images.
[0086] The formula for calculating the first loss value is:
[0087] FL(p t )=-α t (1- t ) γ log(p t );
[0088] Among them, FL(p t ) represents the first loss value, p t Let α be the predicted probability value. t ∈[0,1], γ≥0.
[0089] Through the above implementation method, the second labeling network with the smallest first loss value can be selected as the location detection network. Since the second labeling network has learned the features of the multiple labeled images, the location detection network can accurately label each drug image.
[0090] S13, Generate multiple text feature matrices based on the multiple target images and the text recognition network.
[0091] In at least one embodiment of this application, the plurality of text feature matrices refer to matrices containing text features from the plurality of target images, and each text feature matrix can be used to generate the reference matrix.
[0092] In at least one embodiment of this application, before generating multiple text feature matrices based on the multiple target images and the text recognition network, the drug image recognition method further includes:
[0093] The electronic device acquires text images and a text recognition learner. The text images include a third image and a fourth image. The text recognition learner is trained using the third image to obtain a second pre-trained network. The second pre-trained network includes a convolutional neural network model and a recurrent neural network model. The electronic device calculates a second loss value for the second pre-trained network and performs backpropagation using the second loss value to adjust the parameters of the second pre-trained network multiple times until the second pre-trained model converges and the adjustment stops, thus obtaining the text recognition network.
[0094] The text recognition learner refers to a learner that recognizes the text in each drug image, and the text image refers to an image used to train the text recognition learner, which contains text on any drug.
[0095] The convolutional neural network model can be a VGG16 network, which can be used to extract text features from the fourth image. The recurrent neural network model can be a Long Short-Term Memory (LSTM) network, which can be used to extract temporal information of the text features.
[0096] The third image can be used to train the weights of the text recognition learner. The fourth image refers to an image containing any drug-related text, which may include, but is not limited to, letters and numbers. The third image can be obtained from the first database, and the fourth image can be obtained from a pre-set third database, which stores multiple images of drug-related text.
[0097] The loss function used by the second pre-trained network can be the Connectionist Temporal Classification (CTC) loss function.
[0098] By using the above implementation method, selecting the second pre-trained network corresponding to the lowest second loss value as the text recognition network can improve the reliability and accuracy of the text recognition network, enabling the text recognition network to accurately extract the text features of the target image.
[0099] In at least one embodiment of this application, the electronic device generates multiple text feature matrices based on the multiple target images and the text recognition network, including:
[0100] The electronic device performs color conversion on each target image to obtain multiple grayscale images of text, and binarizes each grayscale image of text to obtain multiple binarized images. Each binarized image is then filtered to obtain multiple filtered images. Further, the electronic device locates the position of the drug text in each filtered image to obtain the text position, and selects the text image from each target image based on the text position. Further still, the electronic device inputs each text image into the convolutional neural network model for feature extraction to obtain a feature sequence. The electronic device then inputs the feature sequence into the recurrent neural network model to obtain the multiple text feature matrices.
[0101] The feature sequence refers to the features extracted by the convolutional neural network model for each filtered image, and the multiple text feature matrices refer to the features extracted by the recurrent neural network model from the feature sequence.
[0102] By performing color conversion, binarization, filtering, and other processing on each target image through the above implementation method, a clearer filtered image can be obtained. Based on the multiple filtered images, text features can be accurately obtained, which is beneficial for generating the text feature matrix.
[0103] S14, input the multiple target images into the category recognition network to obtain multiple image feature matrices.
[0104] In at least one embodiment of this application, the plurality of image feature matrices refers to matrices containing image features from the plurality of target images.
[0105] In at least one embodiment of this application, before inputting the plurality of target images into the category recognition network to obtain a plurality of image feature matrices, the drug image recognition method further includes:
[0106] The electronic device acquires a category recognition learner, in which a soft-max function is used as the activation function. Further, the electronic device calculates a third loss value of the category recognition learner and adjusts the category recognition learner based on the third loss value until the third loss value drops to the minimum, at which point the adjustment stops, and the activation function is removed from the adjusted category recognition learner to obtain the category recognition network.
[0107] Specifically, the electronic device acquires a category recognition learner, and the category recognition learner uses a soft-max function as the activation function, including:
[0108] The electronic device constructs the category recognition learner based on the ResNet50 network and uses soft-max as the activation function.
[0109] In at least one embodiment of this application, the electronic device calculates the third loss value of the category recognition learner by:
[0110] The electronic device acquires multiple category images, which include multiple categories. The multiple category images are subjected to data augmentation processing to obtain multiple augmented images. The multiple augmented images exist in pairs, and each pair of augmented images includes a first augmented image and a second augmented image. The first augmented image and the second augmented image originate from the same category image.
[0111] The multiple category images refer to the output images corresponding to the predicted probability values that are greater than a preset threshold.
[0112] The data augmentation process refers to the process of rotating and cropping each category of image to obtain the final image.
[0113] The aforementioned categories include, but are not limited to, antibiotics, vitamins, etc.
[0114] The formula for determining the third loss value is as follows:
[0115]
[0116] in, 2 refers to the third loss value, i refers to the multiple augmented images, yi refers to the category of the i-th augmented image, j refers to the j-th augmented image among the multiple augmented images, yj refers to the category of the j-th augmented image, and N refers to the category of the j-th augmented image. yi This refers to the number of all augmented images of the same category as i, || i≠j The first indicator function takes the value zero if and only if i = j, and takes the value 1 if i ≠ j. yi＝yj The second indicator function takes the value zero if and only if yi = yj, and takes the value 1 if yi ≠ yj. i≠k The third indicator function, z, takes zero if and only if i = k, and takes 1 if i ≠ k. i This refers to the unit vector z obtained by inputting i into the category recognition network. j This refers to the unit vector obtained by inputting j into the category recognition network, k refers to any augmented image other than i, and z k It refers to the unit vector obtained by inputting k into the category recognition network, and τ is a preset scalar adjustment parameter.
[0117] By performing data augmentation processing on the multiple category images as described above, the training data is expanded. The category recognition network is trained using more training data, which can improve the recognition accuracy of the category recognition network.
[0118] S15, Generate a reference matrix based on each image feature matrix and its corresponding text feature matrix.
[0119] In at least one embodiment of this application, the reference matrix refers to a matrix that includes image features and text features of the plurality of target images, and the reference matrix can be used to indicate the label information of the drug image to be tested.
[0120] In at least one embodiment of this application, the electronic device generates a reference matrix based on each image feature matrix and the corresponding text feature matrix, including:
[0121] The electronic device adds each image feature matrix and its corresponding text feature matrix together to obtain the reference matrix, wherein each image feature matrix and its corresponding text feature matrix have the same number of rows and columns.
[0122] In at least one embodiment of this application, the reference matrix can be generated in other ways, such as: the electronic device multiplies each image feature matrix and its corresponding text feature matrix to obtain the reference matrix, or the electronic device subtracts each image feature matrix and its corresponding text feature matrix to obtain the reference matrix.
[0123] Through the above implementation method, it is possible to extract the image features and text features of the drug in each drug image and generate a reference matrix that simultaneously possesses image features and text features.
[0124] S16, Process the image of the drug to be tested based on the drug detection model to obtain the matrix to be tested.
[0125] In at least one embodiment of this application, the matrix to be tested refers to a matrix containing image features and text features of the image to be tested.
[0126] Since the process of generating the matrix to be tested is the same as that of generating the reference matrix, it will not be described in detail here.
[0127] Through the above implementation method, the image to be tested is processed using the method of processing multiple drug images to obtain the matrix to be tested, so that the matrix to be tested and each reference matrix have the same number of rows and columns, which makes it easier to calculate the similarity between the matrix to be tested and each reference matrix.
[0128] S17, generate the identification result of the drug image to be tested based on the similarity between the matrix to be tested and each reference matrix.
[0129] In at least one embodiment of this application, the identification result refers to the label information corresponding to the drug to be tested.
[0130] In at least one embodiment of this application, the electronic device generates the identification result of the drug image to be tested based on the similarity between the test matrix and each reference matrix, including:
[0131] The electronic device calculates the similarity between the matrix to be tested and each reference matrix, determines the reference matrix corresponding to the highest similarity as the target matrix, and performs mapping processing on the target matrix based on a preset label mapping table to obtain the identification result.
[0132] The preset label mapping table refers to a mapping table between each reference matrix and the corresponding label information, wherein each reference matrix in the preset label mapping table corresponds one-to-one with the label information.
[0133] The similarity may include, but is not limited to, cosine similarity and Euclidean distance.
[0134] The formula for calculating the cosine similarity is:
[0135]
[0136] cosine refers to the cosine similarity, n refers to all elements in the matrix to be tested and any reference matrix, i refers to the i-th element in the matrix to be tested and any reference matrix, and A i It refers to the i-th element in the matrix to be tested, B i It refers to the i-th element in any reference matrix.
[0137] Specifically, the electronic device maps the target matrix based on a preset tag mapping table to obtain the identification result, including:
[0138] The electronic device determines the label information corresponding to the drug to be tested in the preset label mapping table according to the target matrix, and uses the corresponding label information as the identification result.
[0139] Through the above implementation method, the reference matrix corresponding to the highest similarity is selected as the target matrix, so that the drug image corresponding to the target matrix is more similar to the drug image to be tested. The label information of the corresponding drug image is used as the label information of the drug to be tested. According to the one-to-one correspondence between the target matrix and the label information, the label information of the drug to be tested can be found quickly, thereby improving the identification efficiency of drug images.
[0140] As can be seen from the above technical solution, the location detection network is obtained by repeatedly adjusting the second pre-trained network using the multiple labeled images. Since the location detection network learns the features of the multiple labeled images, it can mark the location of the image to be tested. When training the category recognition network, data augmentation processing is performed on the multiple category images to avoid the problem of insufficient training data. Multiple reference matrices are generated based on each image feature matrix and the corresponding text feature matrix. The image to be tested is used to generate the test matrix in the same way. The similarity between the test matrix and each reference matrix is calculated, and the label information of the reference matrix corresponding to the largest similarity is selected as the recognition result. Since the test matrix contains both the text features and image features of the image to be tested, it can comprehensively reflect the features of the image to be tested, thereby making the recognition result more accurate.
[0141] like Figure 3 The diagram shown is a schematic diagram of the structure of an electronic device that implements the drug image recognition method of this application.
[0142] In one embodiment of this application, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and a computer program, such as a drug image recognition program, stored in the memory 12 and executable on the processor 13.
[0143] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. It may include more or fewer components than shown in the diagram, or combine certain components, or different components. For example, electronic device 1 may also include input / output devices, network access devices, buses, etc.
[0144] The processor 13 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 13 is the computing core and control center of the electronic device 1, connecting various parts of the electronic device 1 through various interfaces and lines, and acquiring the operating system and installed applications and program code of the electronic device 1. For example, the processor 13 can acquire the image of the drug to be tested captured by the camera device 2 through an interface.
[0145] The processor 13 acquires the operating system and various installed applications of the electronic device 1. The processor 13 acquires these applications to implement the steps in the various drug image recognition method embodiments described above, for example... Figure 2 The steps are shown.
[0146] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and retrieved by the processor 13 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the process of retrieving the computer program from the electronic device 1.
[0147] The memory 12 can be used to store the computer programs and / or modules. The processor 13 implements various functions of the electronic device 1 by running or retrieving the computer programs and / or modules stored in the memory 12, and by calling the data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 12 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.
[0148] The memory 12 can be the external memory and / or internal memory of the electronic device 1. Furthermore, the memory 12 can be a physical memory, such as a memory module, a TF card (Trans-flash Card), etc.
[0149] If the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is acquired by a processor, it can implement the steps of the various method embodiments described above.
[0150] The computer program includes computer program code, which may be in the form of source code, object code, accessible file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, and read-only memory (ROM).
[0151] Combination Figure 2The memory 12 in the electronic device 1 stores multiple instructions to implement a drug image recognition method. The processor 13 can acquire the multiple instructions to: acquire multiple drug images and a drug image to be tested; acquire a pre-trained drug detection model, which includes a location detection network, a text recognition network, and a category recognition network; input the multiple drug images into the location detection network to obtain multiple target images, each containing an image of a single drug; generate multiple text feature matrices based on the multiple target images and the text recognition network; input the multiple target images into the category recognition network to obtain multiple image feature matrices; generate a reference matrix based on each image feature matrix and its corresponding text feature matrix; process the drug image to be tested based on the drug detection model to obtain a test matrix; and generate the recognition result of the drug image to be tested based on the similarity between the test matrix and each reference matrix.
[0152] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.
[0153] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0154] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0155] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0156] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0157] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this application may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.
[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.
Claims
1. A drug image recognition method, applied to electronic devices, characterized in that, The drug image recognition method includes: Acquire multiple images of the drug and the drug to be tested; A pre-trained drug detection model is obtained, which includes a location detection network, a text recognition network, and a category recognition network; the text recognition network includes a convolutional neural network model and a recurrent neural network model. The multiple drug images are input into the location detection network to obtain multiple target images, each containing an image of a single drug. Based on the multiple target images and the text recognition network, a text feature matrix corresponding to each target image is generated, including: locating the position of the drug text in the filtered image corresponding to each target image to obtain the text position; selecting the text image from each target image based on the text position; inputting each text image into the convolutional neural network model for feature extraction to obtain a feature sequence; and inputting the feature sequence into the recurrent neural network model to obtain the text feature matrix corresponding to each target image. The multiple target images are input into the category recognition network to obtain the image feature matrix corresponding to each target image; A reference matrix is generated based on each image feature matrix and its corresponding text feature matrix; The drug detection model is used to process the drug image to obtain the drug matrix. The identification result of the drug image to be tested is generated based on the similarity between the test matrix and each reference matrix.
2. The drug image recognition method as described in claim 1, characterized in that, Before inputting the multiple drug images into the location detection network to obtain multiple target images, the drug image recognition method further includes: A location detection learner and a location image are acquired, wherein the location image includes a first image and a second image, and the second image includes multiple labeled images and multiple unlabeled images; The location detection learner is trained using the first image to obtain a first pre-trained network; The first pre-trained network is adjusted based on the multiple labeled images to obtain the first labeled network; The multiple unlabeled images are input into the first labeling network to obtain the output images and the predicted probability value of the drug contained in each output image; The output image corresponding to the predicted probability value that is greater than a preset threshold is used to adjust the first labeling network to obtain the second labeling network; Calculate the first loss value of the second labeling network, and adjust the second labeling network multiple times based on the first loss value until the first loss value drops to the minimum, and then stop adjusting to obtain the location detection network.
3. The drug image recognition method as described in claim 1, characterized in that, Before generating the text feature matrix corresponding to each target image based on the multiple target images and the text recognition network, the drug image recognition method further includes: Acquire text images and a text recognition learner, wherein the text images include a third image and a fourth image; The text recognition learner is trained using the third image to obtain a second pre-trained network, wherein the second pre-trained network includes the convolutional neural network model and the recurrent neural network model; Calculate the second loss value of the second pre-trained network, and perform backpropagation using the second loss value to adjust the parameters of the second pre-trained network multiple times until the second pre-trained network reaches convergence and the adjustment stops, thus obtaining the text recognition network.
4. The drug image recognition method as described in claim 1, characterized in that, The method further includes: Each target image is color-converted to obtain multiple grayscale text images; Each grayscale image of the text is binarized to obtain multiple binarized images; Each binarized image is filtered to obtain the filtered image corresponding to each target image.
5. The drug image recognition method as described in claim 1, characterized in that, Before inputting the multiple target images into the category recognition network to obtain the image feature matrix corresponding to each target image, the drug image recognition method further includes: Obtain a category recognition learner, wherein the category recognition learner uses the soft-max function as the activation function; Calculate the third loss value of the category recognition learner, and adjust the category recognition learner based on the third loss value until the third loss value drops to the minimum, then stop adjusting and remove the activation function from the adjusted category recognition learner to obtain the category recognition network.
6. The drug image recognition method as described in claim 5, characterized in that, The calculation of the third loss value of the category recognition learner includes: Acquire multiple category images, wherein the multiple category images include multiple categories; The multiple category images are subjected to data augmentation processing to obtain multiple augmented images; The formula for determining the third loss value is as follows: in, This refers to the third loss value. This refers to the multiple augmented images. It refers to the first of the multiple augmented images. Zhang Zengguang's image, This refers to the first The categories of Zhang Zengguang's images It refers to and The first augmented image of the same category Zhang Zengguang's image, This refers to the first The categories of Zhang Zengguang's images It refers to and The number of all augmented images of the same category, It is the first indicator function if and only if When, take zero. Take 1 at time. It is the second indicator function if and only if When, take zero. Take 1 at time. It is a third indicator function if and only if When, take zero. Take 1 at time. This refers to The unit vector obtained by inputting into the category recognition network. This refers to The unit vector obtained by inputting into the category recognition network. It means except Any augmented image other than [the one mentioned above] This refers to The unit vector obtained by inputting into the category recognition network. These are preset scalar adjustment parameters.
7. The drug image recognition method as described in claim 1, characterized in that, The step of generating a reference matrix based on each image feature matrix and its corresponding text feature matrix includes: The reference matrix is obtained by adding each image feature matrix and its corresponding text feature matrix together, wherein each image feature matrix and its corresponding text feature matrix have the same number of rows and columns.
8. The drug image recognition method as described in claim 1, characterized in that, The step of generating the identification result of the drug image based on the similarity between the target matrix and each reference matrix includes: Calculate the similarity between the matrix to be tested and each reference matrix; The reference matrix corresponding to the highest similarity is determined as the target matrix; The target matrix is mapped based on a preset label mapping table to obtain the identification result.
9. An electronic device, characterized in that, The electronic device includes: Memory, storing at least one instruction; and The processor retrieves instructions stored in the memory to implement the drug image recognition method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in an electronic device to implement the drug image recognition method as described in any one of claims 1 to 8.