Drug quantity conveying supervision method based on image recognition

Abstract

The invention discloses a medicine quantity conveying supervision method based on image recognition, and relates to the technical field of image data processing and machine vision. Comprising the steps that medicine boxes are conveyed to a detection platform in order, and surface information of the medicine boxes is collected through an adjustable camera and a code scanner; performing multi-dimensional analysis and semantic understanding on the acquired information by applying a deep learning algorithm; a database is connected in real time, medicine box information is subjected to multiple verification, and abnormal conditions are marked and classified; and according to a classification result, accurately distributing the medicine boxes to corresponding outlets through a self-adaptive mechanism. According to the invention, the efficiency and accuracy of medicine supervision are obviously improved, and the risk of human errors is greatly reduced.

Description

Technical Field

[0001] The invention relates to the technical field of image data processing and machine vision, and in particular to a method for supervising the delivery of drug quantities based on image recognition. Background Art

[0002] In the modern pharmaceutical industry, accurate drug delivery and supervision are crucial to ensuring drug quality and patient safety. Traditional drug delivery and testing methods mostly rely on manual operation or simple mechanical sorting, which is prone to problems such as drug quantity counting errors and misreading of drug box labels.

[0003] In addition, with the diversification of drug types and packaging forms, existing systems are unable to efficiently cope with these challenges. Therefore, there is an urgent need for an automated drug delivery and supervision method based on image recognition technology to improve the accuracy and efficiency of drug delivery. Summary of the invention

[0004] The present invention is proposed in view of the existing problems in accurate delivery and supervision of existing drugs.

[0005] Therefore, the problem to be solved by the present invention is how to improve the efficiency and accuracy of drug quantity delivery supervision, reduce manual intervention, and improve the intelligent level of drug management.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, the present invention provides a method for monitoring the quantity of drug delivery based on image recognition, which includes transporting medicine boxes to a detection platform in an orderly manner, and using an adjustable camera and a scanner to collect surface information of the medicine boxes; using a deep learning algorithm to perform multi-dimensional analysis and semantic understanding of the collected information; connecting to a database in real time, multi-verifying the medicine box information, marking and classifying abnormal situations; and according to the classification results, accurately diverting the medicine boxes to the corresponding exits through an adaptive mechanism.

[0008] As a preferred solution of the drug quantity delivery supervision method based on image recognition described in the present invention, the use of a deep learning algorithm to perform multi-dimensional analysis and semantic understanding of the collected information includes the following steps: transmitting the collected images and scanned data to a central control center in real time, starting an image recognition algorithm based on deep learning, and performing a multi-dimensional analysis of the barcode and drug supervision code of the medicine box; performing a medicine box integrity test to identify existing damage or printing defects; performing a semantic analysis of the text information on the medicine box through natural language processing technology; performing multi-modal information fusion, and calculating a quality score.

[0009] As a preferred solution of the drug quantity delivery supervision method based on image recognition of the present invention, wherein: the multi-dimensional analysis of the barcode and drug supervision code of the medicine box includes: starting the pre-trained convolutional neural network model for image recognition, locating and segmenting the barcode and drug supervision code areas, recording the model output and confidence of each recognition step, and generating a deep learning recognition behavior feature set DLF={DL1,DL2,...,DL k}, where k is the total number of recognition steps; for each recognition step i, define the feature Among them, O i is the model output, C i is the confidence score:

[0010] C i =σ(z i )

[0011] σ(z)=1 / (1+e ( -z) )

[0012] Among them, σ is the Sigmoid function and z is the original output score of the model.

[0013] As a preferred solution of the drug quantity delivery supervision method based on image recognition of the present invention, the drug box integrity detection and identification of existing damage or printing defects include: dividing the drug box image into multiple small areas, performing anomaly detection on each area, calculating the anomaly score of each area, and marking the area where the anomaly score exceeds the threshold, wherein the anomaly score calculation formula is:

[0014] AS(x)=‖xD(E(x))‖2

[0015] Where x is the input image area, E is the encoder function, D is the decoder function, and ||〃||2 represents the L2 norm.

[0016] As a preferred solution of the drug quantity delivery supervision method based on image recognition of the present invention, wherein: the semantic analysis of the text information on the medicine box by natural language processing technology includes: using OCR technology to extract all the text information on the medicine box, recording the position and extraction confidence of each text area, and generating a text extraction behavior feature set Where q is the total number of text regions extracted; the extracted text is segmented, POS tagged and named entity recognized, and the pre-trained BERT model is used for semantic understanding. The output and confidence of each NLP task are recorded to generate the natural language processing behavior feature set NLPF = {NLP1, NLP2, ..., NLP r}, where r is the total number of NLP tasks; for the extracted text and the standard text, calculate the cosine similarity:

[0017] cos sim (w t , w b ) = (w t × w b ) / (‖w t ‖ × ‖w b ‖)

[0018] where w t and w b are the vector representations of the extracted text and the standard text respectively.

[0019] As a preferred solution of the drug quantity transportation supervision method based on image recognition according to the present invention, wherein: performing multimodal information fusion and calculating the quality score includes: performing multimodal information fusion on the results of image recognition, integrity detection and semantic analysis, using a multimodal Transformer model with an attention mechanism to integrate information from different sources to generate a comprehensive feature vector F; based on the fused feature vector F, designing a multi-layer perceptron network MLP for the final quality score, and outputting the comprehensive quality score Q of the medicine box:

[0020] Q = σ(ω2 × RELU(ω1 × F + b1) + b2)

[0021] where ω1, ω2, b1, b2 are the weight and bias parameters of the MLP respectively, RELU is the activation function, and σ is the Sigmoid function; setting multiple thresholds T1 < T2 < T3, and classifying according to the quality score Q: Q < T1: unqualified, requiring manual review; T1 ≤ Q < T2: suspicious, requiring additional inspection; T2 ≤ Q < T3: qualified, but detailed information needs to be recorded; Q ≥ T3: excellent; storing the analysis results, scores and classification results in the database.

[0022] As a preferred solution of the drug quantity transportation supervision method based on image recognition according to the present invention, wherein: marking and classifying abnormal situations includes: further refining the abnormal classification based on the quality score: defining the abnormal type set A = {A1, A2,..., An}, where n is the number of abnormal types; assigning a weight βi to each abnormal type Ai to reflect the severity, and calculating the comprehensive abnormal score:

[0023] AS = ∑(βi × Ii) / ∑βi

[0024] Among them, Ii is the indicator function of the abnormal type Ai; according to the comprehensive anomaly score AS and the quality score Q, the decision tree algorithm is used to classify the anomalies again: Severe anomaly: AS > 0.7 or Q < T1; Moderate anomaly: 0.3 < AS ≤ 0.7 and T1 ≤ Q < T2; Mild anomaly: 0 < AS ≤ 0.3 and T2 ≤ Q < T3; Normal: AS = 0 and Q ≥ T3; For severe anomalies, immediately trigger a red alarm and import the medicine box into a dedicated buffer; for moderate anomalies, trigger a yellow alarm and import the medicine box into a dedicated buffer; for mild anomalies, trigger a green alarm, increase the proportion of additional manual spot checks, and update the anomaly handling status to the central monitoring system in real time.

[0025] In a second aspect, the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and: when the computer program instructions are executed by the processor, the steps of the method for supervising the transportation of the quantity of medicines based on image recognition as described in the first aspect of the present invention are implemented.

[0026] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and: when the computer program instructions are executed by the processor, the steps of the method for supervising the transportation of the quantity of medicines based on image recognition as described in the first aspect of the present invention are implemented.

[0027] The beneficial effects of the present invention are as follows: The present invention realizes the full-process intelligent management from medicine box transportation, information collection, data analysis to final diversion, not only significantly improves the efficiency and accuracy of medicine supervision, but also greatly reduces the risk of human errors, and makes an important contribution to ensuring medicine safety and improving the public health level. Description of the Drawings

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1 It is the flow of the method for supervising the transportation of the quantity of medicines based on image recognition.

[0030] Figure 2 It is the structural schematic diagram of the computer device in the first embodiment.

[0031] Figure 3 It is the structural schematic diagram of the terminal in the first embodiment. Detailed Embodiments

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0034] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0035] Example 1

[0036] Reference Figure 1 to Figure 3 , which is the first embodiment of the present invention, provides a method for supervising the delivery of drug quantity based on image recognition, comprising:

[0037] S1: The medicine boxes are transported to the inspection platform in an orderly manner through lifting, pushing and precise positioning, and the adjustable camera and scanner are used to collect all-round information on the surface of the medicine boxes.

[0038] Preferably, the medicine box is lifted to a fixed outlet by a hoist, and the medicine box is pushed into the entrance channel of the detection platform by a push rod mechanism. Rotatable levers are provided on both sides of the entrance channel. When the sensor detects the entry of the medicine box, the lever swings downward and pushes the medicine box forward. After the medicine box enters the detection platform, the inner eight-shaped limit device is activated to accurately position the medicine box in the center of the channel.

[0039] Furthermore, a linear motor module with a vertical movement function is installed above the detection platform, and a high-resolution camera group and a multi-band barcode scanner are fixed on the linear motor module; the linear motor module drives the camera group and the barcode scanner to descend to a preset height, performs an all-round scan on the upper part of the medicine box, and collects and identifies the image of the top of the medicine box; at the same time, the high-definition camera fixed under the detection platform, i.e., the lower camera, is started, wherein the lower camera is fixed and does not move, and images of the bottom of the medicine box are collected; all image acquisition devices are equipped with an intelligent light source system, which can automatically adjust the light intensity and angle according to the ambient light and the material of the medicine box to ensure image quality.

[0040] Specifically, automatically adjusting the light intensity and angle according to the ambient light and the material of the medicine box includes the following steps:

[0041] Four photometric sensors (such as TSL2561) are installed around the detection platform. The sensor sampling frequency is set to 1Hz, which is sufficient to capture general ambient light changes. The average reading of the four sensors is calculated as the ambient light intensity value, and a simple threshold method is used to divide the ambient light intensity into three levels: low, medium, and high.

[0042] An existing high-definition camera is used to capture images of the medicine box surface, and the image resolution is set to 1920x1080 pixels.

[0043] Use pre-trained lightweight convolutional neural networks (such as MobileNetV2) for material classification, and divide common medicine box materials into three categories: smooth reflective, matte, and translucent. The model outputs the material category and confidence score, and queries the preset light intensity table based on the ambient light level and material type.

[0044] A 5x5 matrix of 25 adjustable LEDs is installed, each LED can be independently controlled in brightness from 0-100%, and the LED matrix is ​​mounted on a tiltable bracket that can be adjusted from 0° to 45° using a stepper motor.

[0045] Set an optimal illumination angle for each material type, for example: glossy reflective: 15°, matte: 30°, translucent: 45°, select the corresponding angle based on the material classification results.

[0046] When a medicine box enters the detection area, the ambient light sensing system first evaluates the current lighting conditions. The high-definition camera quickly scans the surface of the medicine box, compares the data with the material database, identifies the material of the medicine box, and adjusts the illumination angle according to the optimal angle of the corresponding material. The high-resolution camera group and the barcode scanner begin image acquisition. At the same time, the system continuously monitors the image quality. If poor image quality is detected, the lighting parameters are fine-tuned in real time to ensure the best imaging effect.

[0047] S2: Use deep learning algorithms to conduct multi-dimensional analysis and semantic understanding of the collected information.

[0048] S2.1: The image and scanned data collected in step S1 are transmitted to the central control center in real time, and the image recognition algorithm based on deep learning is started to perform multi-dimensional analysis on the barcode and drug regulatory code of the medicine box, as follows:

[0049] Start the pre-trained convolutional neural network model for image recognition, locate and segment the barcode and drug regulatory code areas, record the model output and confidence of each recognition step, and generate a deep learning recognition behavior feature set DLF = {DL1, DL2, ..., DL k}, where k is the total number of identification steps;

[0050] For each recognition step i, define the features Among them, O i is the model output (such as bounding box coordinates, category labels, etc.), C i is the confidence score:

[0051] C i =σ(z i )

[0052] σ(z)=1 / (1+e ( -z) )

[0053] Among them, σ is the Sigmoid function and z is the original output score of the model.

[0054] S2.2: Perform box integrity inspection to identify possible damage or printing defects.

[0055] Use anomaly detection algorithms (such as autoencoders or One-Class SVM) to identify damage or printing defects on the surface of the medicine box, record the type, location and severity of the detected defects, and generate a surface defect detection behavior feature set SDF = {SD1, SD2, ..., SD j}, where j is the total number of defects detected. The specific operations are as follows:

[0056] The medicine box image is divided into multiple small areas, and the anomaly detection model is applied to each area. The anomaly score of each area is calculated, and the areas where the anomaly score exceeds the threshold are marked. The anomaly score calculation formula (taking the autoencoder as an example) is:

[0057] AS(x)=‖xD(E(x))‖2

[0058] Where x is the input image area, E is the encoder function, D is the decoder function, and ||〃||2 represents the L2 norm.

[0059] S2.3: Use natural language processing technology to perform semantic analysis on the text information on the medicine box to ensure the accuracy and consistency of the information.

[0060] Use OCR technology to extract all text information on the medicine box, record the location and extraction confidence of each text area, and generate a text extraction behavior feature set Among them, q is the total number of text areas extracted.

[0061] Perform word segmentation, part-of-speech tagging and named entity recognition on the extracted text, use the pre-trained BERT model for semantic understanding, record the output and confidence of each NLP task, and generate a natural language processing behavior feature set NLPF = {NLP1, NLP2, ..., NLP r, where r is the total number of NLP tasks.

[0062] For the extracted text and the standard text, calculate their cosine similarity:

[0063] cos sim (w t , w b ) = (w t × w b ) / (‖w t ‖ × ‖w b ‖)

[0064] where w t and w b are the vector representations of the text (such as the [CLS] vector output by BERT), respectively.

[0065] S2.4: Perform multimodal information fusion and calculate the quality score.

[0066] Fuse the results of image recognition, integrity detection, and semantic analysis for multimodal information, and use a multimodal Transformer model with an attention mechanism to integrate information from different sources to generate a comprehensive feature vector F.

[0067] Based on the fused feature vector F, design a multi-layer perceptron network MLP for the final quality scoring, and output the comprehensive quality score Q of the medicine box:

[0068] Q = σ(ω2 × RELU(ω1 × F + b1) + b2)

[0069] where ω1, ω2, b1, and b2 are the weight and bias parameters of the MLP, RELU is the activation function, and σ is the Sigmoid function used to compress the output to the range of 0 - 1.

[0070] Set multiple thresholds T1 < T2 < T3, and classify according to the quality score Q:

[0071] Q < T1: Unqualified, requiring manual review;

[0072] T1 ≤ Q < T2: Suspicious, requiring additional inspection;

[0073] T2 ≤ Q < T3: Qualified, but detailed information needs to be recorded;

[0074] Q ≥ T3: High quality, can be used as a standard sample;

[0075] Store the analysis results, scores, and classification results in the database for subsequent model updates and quality tracking.

[0076] S3: Connect to the database in real time, verify the medicine box information multiple times, and mark and classify abnormal situations.

[0077] Connect to the drug database in real time for dynamic comparison of medicine box information, construct a query index using the comprehensive feature vector F generated in S2 to improve the database retrieval efficiency; set up a multiple verification mechanism, including but not limited to drug types, quantities, production dates, batch numbers, and supervision codes, etc. For example: use an encrypted hash function to verify the authenticity of the supervision code, check whether the date is within the valid production date range, use cosine similarity to compare the extracted drug names with database records to determine whether the types match, etc.

[0078] Preferably, based on the quality score in S2.4, further refine the anomaly classification:

[0079] Define the set of anomaly types A = {A1, A2,..., An}, including type mismatch, quantity anomaly, invalid date, suspicious batch number, invalid supervision code, etc., where n is the number of anomaly types.

[0080] Assign a weight βi to each anomaly type Ai to reflect its severity, and calculate the comprehensive anomaly score:

[0081] AS = ∑(βi × Ii) / ∑βi

[0082] Among them, Ii is the indicator function (0 or 1) of the anomaly type Ai. According to AS and the quality score Q, use the decision tree algorithm to classify the anomalies again:

[0083] Severe anomaly: AS > 0.7 or Q < T1;

[0084] Moderate anomaly: 0.3 < AS ≤ 0.7 and T1 ≤ Q < T2;

[0085] Mild anomaly: 0 < AS ≤ 0.3 and T2 ≤ Q < T3;

[0086] Normal: AS = 0 and Q ≥ T3.

[0087] For severe anomalies, immediately trigger a red alarm and import the medicine box into a dedicated buffer; for moderate anomalies, trigger a yellow alarm and import the medicine box into a dedicated buffer; for mild anomalies, trigger a green alarm, increase the additional manual sampling ratio, and update the anomaly handling status to the central monitoring system in real time.

[0088] S4: According to the classification results, precisely divert the medicine boxes to the corresponding exits through an adaptive mechanism.

[0089] Preferably, based on the classification results, the control system drives the lever through a precision stepper motor to perform a "rattle"-like rotational motion, and adopts an adaptive Y-shaped slide design, which can automatically adjust the diversion angle and speed according to the size and weight of the medicine box; abnormal and normal medicine boxes are accurately diverted to different outlets, and special types of medicine boxes or those that require further inspection are imported into a special buffer zone to wait for manual processing.

[0090] This embodiment also provides a computer device, which is suitable for the case of a drug quantity delivery supervision method based on image recognition, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the drug quantity delivery supervision method based on image recognition as proposed in the above embodiment.

[0091] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0092] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for monitoring the quantity delivery of drugs based on image recognition as proposed in the above embodiment is implemented.

[0093] In summary, the present invention realizes the intelligent management of the whole process from medicine box transportation, information collection, data analysis to final diversion, which not only significantly improves the efficiency and accuracy of drug supervision, but also greatly reduces the risk of human error, and makes important contributions to ensuring drug safety and improving public health.

[0094] Example 2

[0095] Referring to Table 1, which is the second embodiment of the present invention, this embodiment provides a method for monitoring the quantity of drug delivery based on image recognition. In order to verify the beneficial effects of the present invention, a scientific demonstration is carried out through simulation experiments.

[0096] First, a complete drug delivery and inspection system was built, including a lift, push rod mechanism, entry channel, inner eight-shaped limit device, inspection platform, linear motor module, high-resolution camera group, multi-band barcode scanner, lower camera, intelligent light source system, etc. The system is also equipped with a central control center for running deep learning algorithms and data processing.

[0097] The experiment selected 6 different types of medicine boxes as test objects, with 100 samples of each medicine box, for a total of 600 samples. These medicine boxes are representative in terms of material, size, surface information, etc., including different materials such as smooth reflective, matte and translucent.

[0098] The experimental process is as follows:

[0099] The medicine boxes are lifted one by one to the fixed exit by the hoist, and then pushed into the entrance channel of the testing platform by the push rod mechanism. The rotatable levers on both sides of the entrance channel swing downward when the sensor detects the entry of the medicine box, pushing the medicine box forward. When the medicine box enters the testing platform, the inner eight-shaped limit device is activated to accurately position the medicine box in the center of the channel.

[0100] The linear motor module above the detection platform drives the high-resolution camera group and multi-band scanner to descend to the preset height to perform a full-scale scan of the upper part of the medicine box. At the same time, the high-definition camera fixed below the detection platform is started to capture images of the bottom of the medicine box.

[0101] The system first uses the surrounding photometric sensors (TSL2561) to collect ambient light data with a sampling frequency of 1Hz. The calculated average reading is used to determine the ambient light intensity level (low, medium, high). Subsequently, a 1920x1080 resolution image of the surface of the medicine box is captured using a high-definition camera, and the material classification is performed using a pre-trained MobileNetV2 network. Based on the ambient light level and material type, the system queries a preset light intensity table to control the brightness and angle of a 5x5 matrix of 25 adjustable LEDs. For example, for smooth reflective materials, the illumination angle is set to 15°; for matte materials to 30°; and for translucent materials to 45°. The system continuously monitors image quality throughout the image acquisition process and fine-tunes lighting parameters in real time when necessary.

[0102] The collected images and scanned data are transmitted to the central control center in real time, and the deep learning-based image recognition algorithm is started. First, the barcode and drug regulatory code areas are located and segmented using a pre-trained convolutional neural network model, and the model output and confidence of each recognition step are recorded to generate a deep learning recognition behavior feature set.

[0103] An anomaly detection algorithm (autoencoder) is used to identify damage or printing defects on the surface of the medicine box. The medicine box image is segmented into multiple small areas, and the anomaly detection model is applied to each area. The anomaly score is calculated and the areas that exceed the threshold are marked.

[0104] Use OCR technology to extract all text information on the medicine box, record the location of each text area and the extraction confidence. Then, use the BERT model for semantic understanding, perform word segmentation, part-of-speech tagging and named entity recognition on the extracted text. Calculate the cosine similarity between the extracted text and the standard text to evaluate the accuracy and consistency of the information.

[0105] A multimodal Transformer model using an attention mechanism integrates the results of image recognition, completeness detection, and semantic analysis to generate a comprehensive feature vector. Based on this vector, a multi-layer perceptron network is used to calculate the comprehensive quality score of the medicine box.

[0106] Connect to the drug database in real time, use comprehensive feature vectors to build query indexes, perform dynamic comparison of medicine box information, and set up multiple verification mechanisms, including drug type, quantity, production date, batch number, and regulatory code.

[0107] Based on the quality score and anomaly score, the decision tree algorithm is used to classify the anomalies into four categories: severe anomaly, moderate anomaly, mild anomaly, and normal.

[0108] According to the classification results, the control system drives the lever to perform a "rattle"-like rotational motion through a precision stepper motor, and adopts an adaptive Y-shaped slide design to automatically adjust the diversion angle and speed to accurately divert the medicine box to the corresponding outlet or buffer zone.

[0109] Some experimental data are as follows:

[0110] Table 1

[0111]

[0112] It can be seen that the method of the present invention shows extremely high recognition accuracy on different types of medicine boxes. Except for the D-type special material medicine box, the correct recognition rate of other types is above 98%, among which the recognition rate of the B-type matte medicine box is the highest, reaching 99.8%. This shows that the method has strong adaptability and stability when processing medicine boxes with different materials and surface characteristics; and the average processing time is between 1.0 and 1.5 seconds, showing that the method has high efficiency, especially for the F-type small-size medicine box, the average processing time is only 1.0 second, which reflects the flexibility of the system in processing medicine boxes of different sizes. Compared with traditional manual inspection methods, this automated system can greatly improve the efficiency of drug supervision.

[0113] In addition, the false alarm rate is controlled between 1.0% and 2.5%, and the missed alarm rate is controlled between 0.2% and 1.5%. Although the false alarm rate of type D special material medicine boxes is slightly higher (2.5%), the system as a whole shows a good balance, which can effectively identify anomalies without excessively interfering with the circulation of normal medicine boxes.

[0114] It can be seen that the present invention exhibits excellent advantages.

[0115] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for monitoring the quantity of drug delivery based on image recognition, characterized in that: including Orderly convey the medicine boxes to the detection platform, and use adjustable cameras and scanners to collect the surface information of the medicine boxes; Apply deep learning algorithms to perform multi-dimensional analysis and semantic understanding on the collected information; Connect to the database in real time, verify the medicine box information multiple times, mark and classify abnormal situations; According to the classification results, accurately divert the medicine boxes to the corresponding exits through an adaptive mechanism.

2. The method for monitoring the quantity of medicines delivered based on image recognition according to claim 1, characterized in that: The step of applying deep learning algorithms to perform multi-dimensional analysis and semantic understanding on the collected information includes the following steps: Transmit the collected images and scan data to the central control center in real time, start the image recognition algorithm based on deep learning, and perform multi-dimensional analysis on the barcodes and drug supervision codes of the medicine boxes; Conduct integrity detection of the medicine boxes to identify existing damages or printing defects; Through natural language processing technology, perform semantic analysis on the text information on the medicine boxes; Conduct multi-modal information fusion and calculate the quality score.

3. The method for monitoring the quantity of medicines delivered based on image recognition according to claim 2, characterized in that: The multi-dimensional analysis of the barcodes and drug supervision codes of the medicine boxes includes: Start the pre-trained convolutional neural network model for image recognition, locate and segment the barcode and drug regulatory code areas, record the model output and confidence of each recognition step, and generate a deep learning recognition behavior feature set DLF = {DL1, DL2, ..., DL k }, where k is the total number of identification steps; For each recognition step i, define the features Among them, O i is the model output, C i is the confidence score: C i =σ(z i ) σ(z)=1 / (1+e ( -z) ) where σ is the Sigmoid function and z is the original output score of the model.

4. The method for monitoring the quantity of medicines delivered based on image recognition according to claim 3, characterized in that: The step of conducting integrity detection of the medicine boxes to identify existing damages or printing defects includes: Segment the medicine box image into multiple small regions, perform anomaly detection on each region, calculate the anomaly score of each region, and mark the regions where the anomaly score exceeds the threshold. The anomaly score calculation formula is: AS(x) = ‖x - D(E(x))‖2 where x is the input image region, E is the encoder function, D is the decoder function, and ||〃||2 represents the L2 norm.

5. The method for monitoring the quantity of medicines delivered based on image recognition according to claim 4, characterized in that: The step of performing semantic analysis on the text information on the medicine boxes through natural language processing technology includes: Use OCR technology to extract all text information on the medicine box, record the location and extraction confidence of each text area, and generate a text extraction behavior feature set Among them, q is the total number of text regions extracted; Perform word segmentation, part-of-speech tagging and named entity recognition on the extracted text, use the pre-trained BERT model for semantic understanding, record the output and confidence of each NLP task, and generate a natural language processing behavior feature set NLPF = {NLP1, NLP2, ..., NLP r }, where r is the total number of NLP tasks; Calculate the cosine similarity for the extracted text and the standard text; cos sim (w t ,w b )=(w t ×w b ) / (‖w t ‖×‖w b ‖) Among them, w t and w b are the vector representations of the extracted text and the standard text respectively.

6. The method for monitoring the quantity of medicines delivered based on image recognition according to claim 5, characterized in that: The step of conducting multi-modal information fusion and calculating the quality score includes: Perform multi-modal information fusion on the results of image recognition, integrity detection, and semantic analysis, use the multi-modal Transformer model with attention mechanism to integrate information from different sources, and generate the comprehensive feature vector F; Based on the fused feature vector F, design a multi-layer perceptron network MLP for the final quality scoring, and output the comprehensive quality score Q of the medicine box: Q = σ(ω2 × RELU(ω1 × F + b1) + b2) where ω1, ω2, b1, b2 are the weight and bias parameters of the MLP respectively, ReLU is the activation function, and σ is the Sigmoid function; Set multiple thresholds T1 < T2 < T3, and classify according to the quality score Q: Q < T1: Unqualified, requiring manual review; T1 ≤ Q < T2: Suspicious, requiring additional inspection; T2 ≤ Q < T3: Qualified, but detailed information needs to be recorded; Q ≥ T3: High quality; Store the analysis results, scores, and classification results in the database.

7. The method for monitoring the quantity of medicines delivered based on image recognition according to claim 6, characterized in that: The step of marking and classifying abnormal situations includes: Based on the quality score, further refine the abnormal classification: Define the abnormal type set A = {A1, A2,..., An}, where n is the number of abnormal types; Assign a weight βi to each abnormal type Ai to reflect the severity, and calculate the comprehensive abnormal score: AS = ∑(βi × Ii) / ∑βi where, Ii is the indication function of the abnormal type Ai; Based on the comprehensive anomaly score AS and the quality score Q, the decision tree algorithm is used to classify the anomalies again: Severe anomaly: AS > 0.7 or Q < T1; Moderate anomaly: 0.3 < AS ≤ 0.7 and T1 ≤ Q < T2; Mild anomaly: 0 < AS ≤ 0.3 and T2 ≤ Q < T3; Normal: AS = 0 and Q ≥ T3; For severe anomalies, immediately trigger a red alarm and import the medicine box into a dedicated buffer; for moderate anomalies, trigger a yellow alarm and import the medicine box into a dedicated buffer; for mild anomalies, trigger a green alarm, increase the proportion of additional manual spot checks, and update the anomaly handling status to the central monitoring system in real time.

Images