Intelligent medication consultation method and device based on large model

Abstract

The invention discloses an intelligent medication consultation method and device based on a large model. The method comprises the following steps: acquiring user input information; and under the condition that the user input information is a text, performing feature extraction on the text by adopting a pre-training language model BGE to obtain a first feature representation, performing block retrieval on the electronic specification of the medicine based on the first feature representation, and performing context integration on a block retrieval result to obtain a second feature representation of the electronic specification of the medicine. Generating medication consultation information for responding to the information input by the user; under the condition that the user input information is a picture, medicine box detection is conducted on the picture, a target medicine box is recognized, cross-modal feature extraction is conducted on the basis of the target medicine box, second feature representation is obtained, cross-modal retrieval is conducted on the basis of the second feature representation, and medicine use consultation information used for responding to the user input information is generated. The technical problems of low safety, accuracy and convenience of medication management in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the medical field, and in particular, to an intelligent medication consultation method and device based on a large model. Background Art

[0002] With the continuous intensification of the global population aging trend, the demand for medications for chronic and common diseases is increasing, and the needs of the elderly population in drug management and medication guidance are becoming increasingly prominent. However, the drug packages currently on the market generally have problems such as cumbersome information and small fonts, which significantly contradicts the physiological characteristics of the elderly with reduced visual ability and declining cognitive function. When faced with a wide variety of drugs, the elderly often have difficulty accurately identifying drug information, are prone to confusion and memory deviation, and may further lead to safety hazards such as missed doses, wrong doses, or overdose, which not only affects the treatment effect but also poses potential risks to the physical health of elderly patients. In addition, most of the existing drug management solutions lack an aging-friendly design for the elderly population and cannot meet their special usage needs, which further exacerbates the dilemma of medication management for the elderly.

[0003] The current drug identification and consultation systems on the market mainly have the following technical limitations: 1) The information acquisition method is single. Existing systems mostly rely on single-modal information processing, such as pure text input or barcode scanning for drug identification. This single-modal information acquisition method is not only cumbersome to operate but also easily affected by environmental factors (such as light, device performance, etc.), reducing the practicality and reliability of the system. 2) The interaction design is not user-friendly enough. Existing systems generally lack an aging-friendly design for the elderly population, such as functions like font enlargement, voice interaction, and interface simplification. This design flaw severely restricts the usage experience and acceptance of intelligent drug management tools by elderly users. 3) The degree of intelligence is insufficient: Traditional systems cannot achieve intelligent matching of multi-modal information and in-depth semantic understanding, and are often limited to simple keyword matching or fixed template responses, making it difficult to meet the diverse and personalized medication consultation needs of users. 4) The degree of technology integration is not high: Although significant progress has been made in multi-modal large models in the fields of image recognition and natural language processing, there has not been a dedicated application that deeply integrates them with drug identification and medication consultation, especially in the design of aging-friendly systems, which is still blank.

[0004] No effective solution has been proposed for the above problems. Summary of the Invention

[0005] Embodiments of the present invention provide an intelligent medication consultation method and device based on a large model to at least solve the technical problems of low safety, accuracy, and convenience in medication management in the prior art.

[0006] According to one aspect of the embodiments of the present invention, there is provided an intelligent medication consultation method based on a large model, including: obtaining user input information; when the user input information is text, using the pre-trained language model BGE to extract features from the text to obtain a first feature representation, performing chunk retrieval on the electronic drug instructions based on the first feature representation, and integrating the context of the chunk retrieval results to generate medication consultation information for responding to the user input information; when the user input information is a picture, performing medicine box detection on the picture, identifying a target medicine box, performing cross-modal feature extraction based on the target medicine box to obtain a second feature representation, and performing cross-modal retrieval based on the second feature representation to generate medication consultation information for responding to the user input information.

[0007] According to another aspect of the embodiments of the present invention, there is also provided an acquisition module configured to obtain user input information; a text processing module configured to, when the user input information is text, extract features from the text to obtain a first feature representation, perform chunk retrieval on the electronic instructions based on the first feature representation, and integrate the context of the chunk retrieval results to generate medication consultation information for responding to the user input information; a picture processing module configured to, when the user input information is a picture, perform medicine box detection on the picture, identify a target medicine box, perform cross-modal feature extraction based on the target medicine box to obtain a second feature representation, and perform cross-modal retrieval based on the second feature representation to generate medication consultation information for responding to the user input information.

[0008] In the embodiments of the present invention, user input information is obtained; when the user input information is text, the pre-trained language model BGE is used to extract features from the text to obtain a first feature representation, chunk retrieval is performed on the electronic drug instructions based on the first feature representation, and the context of the chunk retrieval results is integrated to generate medication consultation information for responding to the user input information; when the user input information is a picture, medicine box detection is performed on the picture, a target medicine box is identified, cross-modal feature extraction is performed based on the target medicine box to obtain a second feature representation, and cross-modal retrieval is performed based on the second feature representation to generate medication consultation information for responding to the user input information. This application solves the technical problems of low security, accuracy, and convenience in medication management in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and the illustrative embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0010] Figure 1 It is a flowchart of an intelligent medication consultation method based on a large model according to an embodiment of the present invention;

[0011] Figure 2 It is a flowchart of a method for using AI to identify a medicine box and perform correction according to an embodiment of the present invention;

[0012] Figure 3 It is a flowchart of a cross-modal retrieval and matching method for medicine information based on AI according to an embodiment of the present invention;

[0013] Figure 4 It is a flowchart of an aging-friendly method for intelligent medicine box recognition and medication consultation of a multi-modal large model according to an embodiment of the present invention;

[0014] Figure 5 It is a process diagram of the training of a medicine box detection model and a multi-modal medicine matching model according to an embodiment of the present invention;

[0015] Figure 6 It is an interface diagram of medication consultation and an electronic medicine instruction according to an embodiment of the present invention;

[0016] Figure 7 It is an interface diagram of multi-modal retrieval of a medicine vector library according to an embodiment of the present invention;

[0017] Figure 8 It is a structural diagram of a ChineseClip model according to an embodiment of the present invention;

[0018] Figure 9 It is a schematic diagram of a corrected picture according to an embodiment of the present invention;

[0019] Figure 10 It is a flowchart of user interaction according to an embodiment of the present invention;

[0020] Figure 11 It is an interface diagram of medicine box recognition and aging-friendly adaptation according to an embodiment of the present invention;

[0021] Figure 12 It is an interface diagram of an electronic medicine instruction and intelligent medication consultation according to an embodiment of the present invention;

[0022] Figure 13 It is an interface diagram of medication reminder and expiration reminder setting according to an embodiment of the present invention;

[0023] Figure 14 It shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. Detailed implementation manners

[0024] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to an embodiment of the present invention, a method embodiment of an intelligent medication consultation method based on a large model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described here can be executed in a different order from here.

[0027] Figure 1 is a flowchart of an intelligent medication consultation method based on a large model according to an embodiment of the present invention, as Figure 1 shown, the method includes the following steps:

[0028] Step S102, obtain user input information.

[0029] Step S104, in the case where the user input information is text, use the pre-trained language model BGE to extract features from the text to obtain a first feature representation, perform block retrieval on the electronic drug instructions based on the first feature representation, and integrate the context of the block retrieval results to generate medication consultation information for responding to the user input information.

[0030] In the case where the text is intended to be recognized as a question, QA feature extraction is performed on the text to obtain the first feature representation. Based on the first feature representation, a block retrieval is performed on the electronic drug instruction manual, and the retrieval results are contextually integrated to generate drug use consultation information for responding to the user input information. In this embodiment, the first feature representation of the question text is obtained through QA feature extraction, and a block retrieval is performed in the electronic drug instruction manual based on this feature, effectively improving the pertinence and accuracy of the retrieval. At the same time, the retrieval results are optimized through context integration, making the generated drug use consultation information more complete and coherent, and avoiding understanding deviations caused by information fragmentation.

[0031] In the case where the text is intended to be recognized as non-question, cross-modal feature extraction is performed on the text to obtain the first feature representation. Based on the first feature representation, a cross-modal retrieval is performed to match the associated medicine box image and text description from the drug graphic database to obtain the cross-modal retrieval results. Based on the cross-modal retrieval results, a block retrieval is performed on the electronic drug instruction manual, and the retrieval results are contextually integrated to generate drug use consultation information for responding to the user input information. For example, based on the first feature representation, the cosine similarity algorithm is used to retrieve relevant content segments in the instruction manual vector library generated based on the electronic drug instruction manual, and through semantic similarity ranking, the most relevant content paragraphs are selected from the relevant content segments; the most relevant content paragraphs are integrated as context information, and based on the context information, a large language model is used to generate drug use consultation information for responding to the user input information. This embodiment combines cross-modal feature extraction and retrieval technologies. When the user inputs non-question type text, it can intelligently match the medicine box image and text description, and further combine the block retrieval of the electronic drug instruction manual to ensure the comprehensiveness and accuracy of the information. Through the cosine similarity algorithm and semantic similarity ranking, the screening of relevant content segments is optimized to improve the precision of the retrieval results.

[0032] In some embodiments, the instruction manual vector library is obtained as follows: all relevant fields of the electronic drug instruction manual are concatenated into a complete document in logical order, and the concatenated document is subjected to standardized preprocessing; for the preprocessed document, it is divided into blocks with a preset number of characters as the basic unit, and an overlapping area with a preset overlap degree is set between adjacent blocks, where the preset number of characters is 1024 characters, and the preset overlap degree is 10% of the number of characters in adjacent blocks. Based on the divided document, the instruction manual vector library is constructed. In this embodiment, through the standardized preprocessing of the electronic drug instruction manual and the use of fixed-length block division and overlapping area setting, the integrity and retrieval accuracy of the vector library construction are improved. The block division method ensures information coherence, avoids content breakage, and improves the accuracy and semantic matching effect of subsequent retrievals.

[0033] Step S106, when the user input information is a picture, perform medicine box detection on the picture, identify the target medicine box, extract cross-modal features based on the target medicine box to obtain a second feature representation, and perform cross-modal retrieval based on the second feature representation to generate medicine use consultation information for responding to the user input information.

[0034] 1) Perform medicine box detection on the picture, identify the target medicine box, and correct the target medicine box.

[0035] For example, identify the main body position of the target medicine box in the picture; when the rotation angle of the main body position deviates from the standard horizontal direction, use the affine transformation algorithm to correct the rotation angle of the main body position to the standard horizontal direction. In the embodiment of the present application, the rotation angle of the target medicine box is corrected by the affine transformation algorithm to align it to the standard horizontal direction, improving the accuracy of image normalization processing, ensuring the stability and accuracy of subsequent image recognition and information extraction, and being applicable to the image processing of medicine boxes photographed at different angles.

[0036] In some embodiments, AI can be used to identify and correct the medicine box. For example, as Figure 2 shown, the method for identifying and correcting a rotated medicine box based on AI includes the following steps:

[0037] Step S202, obtain a picture of the target medicine box to be identified.

[0038] Step S204, use the medicine box detection model to identify the picture of the target medicine box to be identified, and correct the rotation angle of the identified target medicine box to the standard horizontal direction.

[0039] In some embodiments, the medicine box detection model is obtained as follows: based on a medicine box data set with rotation angle annotations, train a neural network model to obtain the medicine box detection model. The specific training process can be as follows:

[0040] First, obtain medicine box sample data, where the medicine box sample data includes medicine box sample pictures.

[0041] Next, rotate angle annotation is performed on each medicine box sample image in the medicine box sample data to construct the medicine box data set with rotation angle annotation, and data augmentation is performed on the medicine box data set with rotation angle annotation, where the data augmentation includes at least one of the following: random angle rotation, scale transformation, and brightness and contrast adjustment. For example, the four vertex coordinates of the minimum bounding rectangle and the rotation angle relative to the horizontal axis of each medicine box sample image are annotated on each medicine box sample image to obtain the medicine box data set with rotation angle annotation; the medicine box data set is data-augmented by at least one of the following: performing the random angle rotation within the range of ±90°, performing the scale transformation between 0.8 and 1.2 times, and performing the brightness and contrast adjustment within the range of ±30°. In this embodiment, by performing rotation angle annotation on the medicine box sample image and constructing the medicine box data set with rotation angle information, the recognition ability of the model for medicine box images at different angles is improved. The data augmentation strategy further improves the generalization performance and robustness of the model, enabling it to stably recognize medicine box information under different shooting conditions, improving the adaptability and accuracy of image processing, and being applicable to automatic recognition and information extraction of drug packaging in complex scenarios.

[0042] Finally, based on the medicine box data set after data augmentation, the neural network model is trained to obtain the medicine box detection model. Specifically, using the oriented bounding box (OBB), redundant rotation detection boxes in each medicine box sample image in the medicine box data set after data augmentation are removed to obtain the target detection box. For example, for each medicine box sample image, when calculating the intersection over union (IoU) between the rotation detection boxes using the OBB, the calculation method of the rotated IoU is adopted, and the final overlap degree is obtained by dividing the intersection area of the polygons by the union area; the final overlap degree is used to filter out redundant rotation detection boxes. Then, affine transformation is used to correct the target detection box and adjust the target detection box to the 0° standard horizontal direction; the adjustment angle of the target detection box is compared with the annotation angle of each medicine box sample image, and the loss function of the neural network model is adjusted based on the comparison result to train the neural network model. In this application, the neural network model is trained using the medicine box data set after data augmentation, and the oriented bounding box (OBB) is used to optimize the rotation detection box to remove redundant boxes, improving the accuracy of target detection. The rotated IoU is used to calculate the overlap degree to ensure that the selected detection box is optimal. The affine transformation corrects the detection box to the standard horizontal direction, and the loss function is adjusted in combination with the annotation angle to optimize the model training effect. This method improves the stability and robustness of the medicine box detection model, making it more suitable for the medicine packaging recognition task under different shooting angles and complex environments.

[0043] 2) Perform cross-modal feature retrieval.

[0044] Based on the second feature representation, a cross-modal matching is performed on the drug image-text vector database using a multi-modal drug matching model to obtain drug information; wherein, the drug image-text vector database is constructed by at least one of the following: combining a brand and a drug name as the main text description and combining the key features of the drug to form complete text description information; performing at least one of the following data augmentation processes on drug sample images: random angular rotation within the range of ±30°, brightness and contrast adjustment, and horizontal and vertical flipping; performing at least one of the following data augmentation processes on training sample words: synonym replacement and brand-generic name interchange. In this application, a cross-modal matching is performed on the drug image-text vector database using a multi-modal drug matching model, improving the accuracy and comprehensiveness of drug information retrieval. Combining the brand and the drug name and the key features to construct a complete text description makes the matching result more targeted. At the same time, through various data augmentation processes on drug sample images and training text data, the robustness and generalization ability of the model are improved, ensuring accurate matching of drug information under different lighting, angles, and text variant conditions, and enhancing the adaptability and reliability of cross-modal retrieval.

[0045] In some embodiments, AI can be used for cross-modal feature retrieval. For example, as Figure 3 shown, the cross-modal retrieval matching method for drug information based on AI may include the following steps:

[0046] Step S302, obtain a target medicine box image to be recognized.

[0047] Step S304, use the multi-modal drug matching model to extract features from the target medicine box image to obtain a feature representation.

[0048] Step S306, based on the feature representation, use the multi-modal drug matching model to perform cross-modal matching to match drug information related to the target medicine box.

[0049] Among them, the multi-modal drug matching model is obtained through the following: based on the image-text pair training set in the drug field, perform domain adaptation training on the pre-trained multi-modal network model to obtain the multi-modal drug matching model. The specific training process is as follows:

[0050] First, obtain the text-image pair training set. Obtain drug sample data, where the drug sample data includes drug sample pictures and training text data; perform at least one of the following enhancement processes on the drug sample pictures: perspective transformation, lighting adjustment, and local magnification, and expand the training text data by means of synonym replacement and interchange of brand names and generic names; use the enhanced drug sample pictures and the expanded training text data as the text-image pair training set. This embodiment effectively improves the training effect of the cross-modal model, enabling it to more accurately and stably match and identify drug information in practical applications.

[0051] Next, based on the text-image pair training set, perform pre-training on the multi-modal network model in the drug domain to obtain the multi-modal drug matching model. Perform pre-training on the multi-modal network model on the general Chinese text-image dataset; based on the text-image pair training set, perform pre-training on the pre-trained multi-modal network model in the drug domain to obtain the multi-modal drug matching model. For example, based on the attention mechanism enhanced by drug knowledge, use the model to extract key visual features in the drug sample pictures in the text-image pair training set through an additional drug attribute supervision signal; through the attention mechanism enhanced by drug knowledge graph embedding, combined with the drug attribute classification loss function, use the multi-modal network model to extract key visual features; when encoding the training text data in the text-image pair training set, fuse the prior information of entity relationships in the drug knowledge graph; use the key visual features and the text encoding that has fused the prior information of entity relationships to perform the pre-training in the drug domain. In this application embodiment, by pre-training the multi-modal network model on the general Chinese text-image dataset and further pre-training it in combination with the text-image pair training set in the drug domain, the accuracy of the model in the drug matching task is improved. In addition, using the attention mechanism enhanced by drug knowledge and the drug attribute supervision signal optimizes the visual feature extraction ability; finally, fusing the prior information of entity relationships in the drug knowledge graph improves the semantic understanding ability of text encoding, thereby enhancing the accuracy and generalization ability of cross-modal matching and realizing efficient drug information retrieval and matching.

[0052] Finally, based on the feature representation, use the multimodal drug matching model for cross-modal matching to match the drug information related to the target medicine box. Specifically, based on the feature representation, use the multimodal drug matching model to perform cross-modal matching on the text-image vector database to obtain the drug information. For example, use the multimodal drug matching model to perform similarity retrieval on the text-image vector database and calculate the cosine similarity; sort based on the cosine similarity, and select the top K candidate results with the highest similarity as the drug information; wherein, the text-image vector database stores the image feature vectors of medicine box pictures, as well as the feature vectors of the corresponding drug names, brands, indications, dosages, and instruction manual texts; the drug information includes at least one of the following: the type, brand, indication, and instruction manual of the drug. This application realizes accurate drug information retrieval by performing cross-modal matching on the text-image vector database using the multimodal drug matching model.

[0053] An embodiment of the present invention provides an intelligent medicine box recognition and medication consultation system based on a multimodal large model, aiming to solve problems such as insufficient accuracy and inconvenient information acquisition in the processes of drug recognition, information query, and medication consultation. The system utilizes the pre-training in the drug field of the multimodal model, cross-modal retrieval, and the intelligent question-and-answer technology of the large model to provide an efficient and accurate drug information processing ability, and provide a comprehensive intelligent solution for users' medication management.

[0054] This system includes a pre-training and cross-modal retrieval module, and a database construction and intelligent medication consultation module.

[0055] The pre-training and cross-modal retrieval module is used for pre-training in the drug field and cross-modal retrieval of the multimodal large model. First, perform pre-training in the drug field. For example, based on data of Western medicines, traditional Chinese medicines, medical devices, and health products, perform drug field pre-training for the ChineseCLIP model in the drug field. The data covers drug pictures, names, and brands. By constructing text-image pairs for training, the cross-modal image and text understanding ability of the model in the drug field is improved, and efficient matching between medicine box pictures and drug information is achieved. Then, perform text-image vector database construction and cross-modal retrieval. For example, the system vectorizes the medicine box pictures and their corresponding drug names, brands, and other description texts and stores them in the text-image vector database. Through taking pictures of the medicine box or inputting the drug name by the user, the system quickly matches the corresponding drug information, including the type, brand, indication, and other details of the drug based on cross-modal retrieval technology, thus realizing accurate and convenient drug recognition and information query.

[0056] The construction of the database and the intelligent medication consultation module are used for the construction of the vector database of electronic drug instructions and intelligent medication consultation. First, the construction of the electronic instruction vector database is carried out. The electronic drug instructions are stored in blocks by paragraph, covering complete information such as the indications, usage and dosage, contraindications, adverse reactions, drug interactions, etc. of the drug. Semantic vectors of each paragraph are generated through the BGE vector model and stored in the instruction vector database. Then, intelligent question answering based on knowledge base and vector matching is carried out. A drug knowledge base is constructed by combining the structured information in the instructions with other authoritative data sources (such as pharmacopoeias, clinical guidelines) to form a comprehensive drug knowledge base. When the user asks questions (such as "What are the indications of this drug?", "Is it applicable to children?", "What are the adverse reactions?", etc.), the system first matches relevant information in the instruction vector database through the user query (query), and integrates it with the context semantic analysis technology and the knowledge base. The LLM large model generates accurate answers in natural language. The intelligent question answering function supports dynamic multi-round interaction and can generate coherent answers based on historical questions to provide users with complete drug consultation services. Cross-modal retrieval and intelligent question answering.

[0057] The user initiates a retrieval request by taking a picture of the medicine box, entering the medicine name, or voice query. The system retrieves all the information of the corresponding drug according to the matching result of the graphic and text vector database.

[0058] When the user queries specific questions such as the usage method, indications, contraindications, etc. of the drug, the system quickly matches relevant content through the electronic instruction vector database, and combines the context semantic analysis ability of the multi-modal large model to generate accurate and natural intelligent questions and answers to ensure comprehensive and accurate information.

[0059] The present invention applies the image and text understanding capabilities of the multi-modal large model to the scenarios of drug recognition and medication consultation, and combines the design for the elderly, significantly improving the efficiency of drug information query and the convenience of medication management, especially having wide application value in the elderly user group.

[0060] The embodiment of the present application also provides an elderly-friendly method for intelligent medicine box recognition and medication consultation of a multi-modal large model. This method is applied to the above-mentioned intelligent medicine box recognition and medication consultation system based on the multi-modal large model. This method is as Figure 4 shown and includes the following steps:

[0061] S402, data preprocessing and model pre-training.

[0062] The overall process of data preprocessing and model pre-training is as Figure 5 shown.

[0063] The construction of the drug vector library will be described in detail below.

[0064] The drug vector library constructed by the present invention is a large-scale multi-source heterogeneous data set, containing structured data records of multiple health products, medical devices, western medicines, and traditional Chinese medicines. Each record contains multiple data fields, covering basic information such as product name, brand, specification, scope of application, storage conditions, as well as regulatory information such as approval number, validity period, and manufacturer. At the same time, the record also contains clinical medication information such as main ingredients, precautions, dosage and administration, and contraindication symptoms, and provides medication guidance for special populations, such as pediatric medication, medication for pregnant and lactating women, etc. In addition, for different types of drugs, their specific professional information is also stored, such as the pharmacology and toxicology, pharmacokinetics of western medicines, and the structural composition, registration certificate number of medical devices, etc.

[0065] In the construction of multi-modal training samples, the present invention adopts an innovative graphic-text alignment strategy. First, the brand and drug name are combined as the main text description, and complete text description information is formed by combining key features such as specifications and properties. To improve the generalization ability and robustness of the model, the present invention performs comprehensive data augmentation processing on drug images, covering enhancement techniques such as random angle rotation, brightness and contrast adjustment, horizontal and vertical flipping within the range of 0° to 360°. At the same time, multi-dimensional enhancement processing such as synonym replacement and brand-generic name interchange is also performed at the text level, effectively expanding the diversity of training samples.

[0066] For the processing of drug instructions, the present invention adopts a fine-grained text chunking strategy, such as Figure 6 shown. First, all relevant fields are concatenated into a complete electronic instruction document in logical order. After standardized preprocessing, chunking is performed with 512 characters as the basic unit. To ensure the coherence and integrity of semantics, a 10%-15% overlapping area is set between adjacent chunks. This strategy not only ensures the independence of the content after chunking but also maintains the semantic association of the context. During the chunking process, special attention is paid to avoiding truncation of important information to ensure that each chunk has a complete semantic expression.

[0067] As Figure 7 shown, in the vector feature extraction and storage link, the present invention generates high-dimensional feature vectors for drug images, text descriptions, and instruction chunks respectively. The dimension of the image feature vector extracted by the pre-trained model is 768, the dimension of the text description vector is 768, and the dimension of the instruction chunk vector is 1024. These feature vectors are stored in the Milvus vector database, establishing an efficient multi-modal index structure to support fast similarity retrieval and approximate nearest neighbor search. At the same time, a perfect data quality control mechanism is implemented, including functions such as data integrity verification, outlier detection, and incremental update, ensuring the reliability and maintainability of the vector library.

[0068] The multimodal drug matching model will be described in detail below.

[0069] The present invention adopts the improved ChineseClip multimodal model as the core architecture, which is optimized specifically for the Chinese medicine field based on the original CLIP model. Figure 8 As shown in the figure, the model as a whole consists of two parts: image encoder and text encoder, which realizes the unified representation space mapping of image and text features through contrastive learning method.

[0070] Among them, the image encoder adopts the Vision Transformer (ViT) architecture, which contains 12 layers of Transformer blocks, each layer has 12 attention heads, and the input resolution is uniformly adjusted to 224×224 pixels; the text encoder is based on the RoBERTa architecture, consisting of 12 layers of bidirectional Transformer, the vocabulary expands the professional vocabulary in the pharmaceutical field, and the maximum sequence length is set to 128 tokens.

[0071] In the model pre-training stage, we first constructed a training set of image and text pairs using 142,307 pieces of drug data. Considering the particularity of drug images, we improved the standard data enhancement strategy: on the image side, we introduced drug-specific enhancement methods such as perspective transformation, lighting adjustment, and local magnification; on the text side, we expanded the training samples by synonym replacement and brand generic name conversion. The training process adopted a contrastive learning strategy, and used the InfoNCE loss function to optimize the model parameters. The batch size was set to 1024, and the learning rate adopted a cosine annealing strategy, which gradually decayed from 1e-4.

[0072] In order to improve the recognition accuracy of the model in the pharmaceutical field, the present invention designs a special training framework. First, pre-training is performed on a general Chinese image and text dataset, and then domain adaptability fine-tuning is performed using the collected pharmaceutical dataset. In the fine-tuning stage, an attention mechanism for pharmaceutical knowledge enhancement is introduced, and the model is guided to focus on key visual features in pharmaceutical images, such as packaging features, dosage form features, and identification features, through additional pharmaceutical attribute supervision signals. At the same time, the prior information of the pharmaceutical knowledge graph is incorporated into the text encoding, which enhances the model's ability to understand pharmaceutical terminology.

[0073] The training objective design of the model adopts a multi-task learning framework. In addition to the basic image-text comparison loss, it also includes drug classification loss and feature consistency loss. Among them, the classification loss helps the model learn the category information of the drug, and the feature consistency loss ensures that the feature representation of the same drug from different perspectives is consistent. Through this multi-objective optimization strategy, the model can take into account the two key tasks of image-text alignment and drug feature extraction at the same time.

[0074] The optimized multi-modal model demonstrates excellent feature extraction capabilities. In practical applications, the model can accurately understand the visual information of drug images and establish precise semantic associations with text descriptions, providing a reliable basis for feature representation in subsequent intelligent medicine box recognition and medication consultation. At the same time, through a knowledge-enhanced feature extraction mechanism, the model has strong discrimination ability for subtle differences in drugs and can effectively handle the recognition problems of similar drugs.

[0075] The medicine box detection model will be described below.

[0076] In view of the actual problem that there are diverse shooting angles for the medicine box pictures uploaded by users, the present invention innovatively proposes a rotation medicine box detection scheme based on OBB (Oriented Bounding Box). Although the samples have been augmented by angle rotation in the previous multi-modal training stage, due to the randomness and complexity of the medicine box placement angles in the actual application scenario, the recognition accuracy may still be insufficient. Therefore, the present invention deeply improves the traditional YOLO model, introduces a rotation target detection mechanism, and realizes the accurate positioning and angle correction of medicine boxes at any angle.

[0077] To improve the training effect of the model, the present invention specifically constructs a medicine box data set with annotated angles. Through a semi-automatic annotation tool, the four vertex coordinates of the minimum circumscribed rectangle and the main direction angle are annotated for each medicine box. At the same time, the training samples are augmented through data augmentation techniques, including random angle rotation, scale transformation, brightness and contrast adjustment, etc., to improve the adaptability of the model to various shooting conditions.

[0078] In the post-processing stage, in order to effectively filter redundant rotation detection boxes, the present invention designs a rotation non-maximum suppression algorithm (R-NMS). Different from the traditional non-maximum suppression (NMS) which is only applicable to horizontal rectangular boxes, R-NMS is optimized for the characteristics of rotation targets. When calculating the IoU between detection boxes, the calculation method of rotated IoU is adopted, and the final overlap degree is obtained by dividing the intersecting area of polygons by the union area. The traditional IoU calculation only considers the intersection and union areas of horizontal rectangles, while the present invention calculates the overlap degree between rotated detection boxes, that is, the rotated intersection over union (Rotated IoU, R-IoU). Specifically, the present invention adopts the following steps:

[0079] 1) Parametric representation of the rotation detection box.

[0080] Each detection box is represented by a five-element (x c , y c , w, h, θ) group, where {(x} c , y c) is the center coordinate of the box, w and h are the width and height of the box respectively, and θ is the main direction angle of the box (in radians, ranging from [-π / 2, π / 2]).

[0081] 2) Calculate the rotated IoU.

[0082] For two rotated detection boxes B i and B j , the formula for calculating R-IoU is:

[0083]

[0084] 3) Perform suppression.

[0085] Sort all detection boxes in descending order of confidence scores. Traverse the list of detection boxes. For the currently highest-scoring box B max , calculate its R-IoU with the remaining boxes. If R-IoU > τ (τ is a preset threshold, such as 0.3), then suppress the box with a lower score. Repeat the above steps until all detection boxes are processed.

[0086] Through the above method, R-NMS can effectively eliminate redundant rotated detection boxes, while retaining key targets and improving the accuracy of detection results. Compared with traditional NMS, R-NMS shows higher adaptability when dealing with rotated targets such as medicine boxes, especially in dense scenes.

[0087] In some other embodiments, redundant rotated detection boxes can also be filtered based on the following improved rotated non-maximum suppression (R-NMS):

[0088] 1) Preprocessing.

[0089] Input the original set of rotated detection boxes {B i = (x c , y c , w, h, θ, s)}, generate a direction vector based on the angle θ: v x = cosθ, v y = sinθ, and output the standardized parameter set {B i ′ = (x c , y c , w, h, v x , v y , s)}.

[0090] Compensate the confidence based on the visible area:

[0091]

[0092] The compensated confidence queue Q = {B i ′ | s′ in descending order}.

[0093] 2) Conduct stratified screening.

[0094] Based on the current highest confidence box B max With candidate box B j Perform fast collision detection. Input is B max , B j , apply the Separating Axis Theorem (SAT) to detect whether the projection is separated, and output a Boolean value {True / False} to indicate whether the calculation needs to continue.

[0095] 3) Calculate multi-precision R-IoU.

[0096] Based on center distance Select the calculation mode, when d>2×max(w max ,h max ), based on the Monte Carlo method The sampling points estimate the overlapping area to obtain an approximate R-IoU value. In some embodiments, the Greiner-Hormann algorithm can also be used to calculate the exact intersecting polygon vertices to obtain an accurate R-IoU value.

[0097] 4) Dynamic inhibition.

[0098] Based on the angle difference Δθ=|θ max -θ j | Adjust the threshold and calculate the angle-sensitive threshold based on the basic threshold τ0 and Δθ:

[0099] τ=τ0+0.1τ0(1-cosΔθ)

[0100] Based on local density Perform secondary adjustments, and calculate the final threshold based on the current threshold τ, density ρ, the number of neighboring boxes as the number of points in a circle with a radius of r, and ρ0 as the reference density:

[0101] τ final =τ×(1-0.2tanh(ρ / ρ0))

[0102] Based on R-IoU>τ final Determine whether to suppress and output the updated detection box queue Q′. Matrix parallel computing based on GPU. Input the rotation box parameter set and convert each box into a homogeneous coordinate matrix, where M i represents the homogeneous coordinate matrix of the i-th rotation frame, θ i represents the rotation angle of the i-th rotation frame, (x i ,y i ) represents the center coordinates of the i-th rotation frame. The calculation formula of Mi is as follows:

[0103]

[0104] Use CUDA thread blocks to calculate the projections of 16 pairs of boxes in parallel and output the batch-calculated R-IoU matrix.

[0105] Input candidate box pairs (B max , B j ), where F max represents the Fourier descriptor of candidate box B max , and F j represents the Fourier descriptor of candidate box B j . Calculate the Fourier descriptor distance, i.e., the shape similarity score S:

[0106]

[0107] Based on the overlapping region image patches, extract the shallow features of the pre-trained CNN and calculate the cosine similarity to obtain the texture similarity score T. Final suppression condition: (R-IoU > τ final ) ∧ (S < 0.2) ∧ (T > 0.7).

[0108] After detecting the rotation angle of the medicine box, the present invention further corrects the image using affine transformation to adjust the medicine box to the standard horizontal direction. During the correction process, considering the preservation of image quality, based on the geometric parameters of the detection box, combined with the bicubic interpolation algorithm for image resampling, the detailed information of the image is retained to the greatest extent. The image after angle correction will be input into the aforementioned multi-modal model for feature extraction and recognition, thereby significantly improving the overall recognition accuracy of the system. The specific steps are as follows:

[0109] 1) Construct the transformation matrix.

[0110] For the detected center (x c , y c ) of the medicine box and the rotation angle θ, construct an affine transformation matrix M for rotating the image to the horizontal direction. The transformation matrix is defined as:

[0111]

[0112] This matrix rotates the image counterclockwise by θ degrees around the center (xc, yc) to align the main direction of the medicine box with the horizontal axis.

[0113] 2) Map the coordinates.

[0114] For each pixel point (x, y) in the original image, calculate the corrected coordinates (x′, y′) through M

[0115]

[0116] 3) Optimize the bicubic interpolation.

[0117] During the calibration process, to maintain the image quality, a bicubic interpolation algorithm is used for image resampling. Here, W(t) represents the weight function.

[0118]

[0119] When a = -0.5, it provides the best edge preservation and detail reconstruction capabilities. t represents the pixel position offset during resampling, usually within the range of [-1, 1], and is used to calculate the interpolation weights.

[0120] The optimized rotating medicine box detection model can effectively handle the angular changes of the medicine box under complex shooting conditions, ensuring that the system can maintain high-precision recognition capabilities in different scenarios. Through this solution, the accuracy of medicine box detection and angle correction has been significantly improved, further enhancing the practicality and reliability of the intelligent medicine box recognition system.

[0121] The specific optimization effects are as Figure 9 shown. The optimized rotating medicine box detection model can effectively handle the angular changes of the medicine box under complex shooting conditions, ensuring that the system can maintain high-precision recognition capabilities in different scenarios. Through this solution, the accuracy of medicine box detection and angle correction has been significantly improved, further enhancing the practicality and reliability of the intelligent medicine box recognition system.

[0122] Step S404, user interaction.

[0123] As Figure 10 shown, the process of user interaction includes the following steps:

[0124] Step S1002, obtain user input.

[0125] Obtain user input information. For example, pictures and text. Users can upload pictures of the medicine box, and the system preprocesses the pictures, including size adjustment, illumination normalization, and noise elimination. Then, through the rotation angle detection algorithm, the main body position of the medicine box is identified and its tilt angle is corrected to ensure that the image is upright in subsequent processing. Users can also input text. For example, users input the name of the medicine, and the system performs cleaning, word segmentation, and standardization processing on the text.

[0126] Step S1004, determine whether the user input is text or a picture.

[0127] If the user input is a picture, execute step S1006; otherwise, execute step S1012.

[0128] Step S1006, use the medicine box detection model for medicine box recognition and correction.

[0129] Step S1008, cross-modal feature extraction.

[0130] In the case where the input is an image, use the pre-trained ChineseClip model (medicine box detection model) to extract the image feature vector. In the case where the input is text, convert the drug brand and name into a feature vector through the ChineseClip text encoder. Then, generate a unified feature representation of the feature vectors for subsequent retrieval.

[0131] Step S1010, cross-modal retrieval matching.

[0132] Use a vector database (such as Milvus or Faiss) for efficient similarity retrieval. Perform Top-K recall based on cosine similarity calculation to obtain the drug information most similar to the query and its corresponding instruction manual ID.

[0133] After cross-modal retrieval matching, the process can be ended to directly output the retrieval result, or it can jump to step S1016.

[0134] Step S1012, determine whether the intention recognition is a question.

[0135] If it is a question, execute step S1014; otherwise, execute step S1008.

[0136] Step S1014, QA feature extraction.

[0137] Step S1016, electronic instruction manual block retrieval.

[0138] Convert the user's specific consultation question into a query vector through the BGE model. Retrieve relevant content segments in the instruction manual vector database. Screen out the most relevant content paragraphs through semantic similarity ranking.

[0139] Step S1018, context information integration.

[0140] Integrate the basic drug information (such as name, specification, usage, etc.). Integrate the relevant instruction manual content (such as dosage and administration, contraindications, etc.). Construct a structured prompt template to provide complete context information for the large language model.

[0141] Input the integrated context information into the large language model (LLM). Guide the model to generate accurate and professional answers through optimized prompt engineering. Verify the reliability and integrity of the generated answers.

[0142] Finally, format the generated answer to make it more understandable. Necessary warning information and disclaimers can also be added. Finally, the final consultation result can be presented in an age-friendly manner (such as enlarged font, voice broadcast, etc.).

[0143] Specifically, the prompt words for the large model to generate medication results are as follows:

[0144]

[0145] The present invention adopts a large language model architecture based on retrieval enhancement in the medication consultation process. Through multi-level context information fusion and intelligent dialogue generation, it provides users with professional and accurate medication guidance. The system uses the medication instruction manual vector library as the knowledge basis and combines optimized prompt engineering to achieve intelligent medication consultation services.

[0146] In terms of prompt design, the present invention constructs a multi-level prompt template system. First, a clear role positioning is set, and the model is defined as a "professional medication consultant". This role positioning ensures that the model maintains professionalism and rigor when answering questions. Second, at the instruction level, the answering strategy is clarified: when the answer to a question can be found in the context, the model will generate an accurate answer based on the context; if the question exceeds the context scope, the model needs to answer politely and guide the user to ask answerable questions or suggest seeking medical advice beyond the scope of auxiliary medical treatment. This strategy design not only ensures the reliability of the answer but also reflects the service awareness of the system.

[0147] In the context construction process, the system adopts a multi-level retrieval strategy. First, it retrieves the most relevant medication information fragments to the user's question through vector similarity retrieval, and then ranks the importance of these fragments. The ranking considers multiple dimensions: semantic relevance to the question, importance of the content (such as the importance of contraindications being higher than general instructions), timeliness of the information, etc. The selected context content is organized in the logical order of "basic information - usage and dosage - precautions - professional information" to ensure that the large model can obtain complete and structured reference information.

[0148] To improve the quality of answers, the present invention designs a multiple constraint mechanism at the model output end. First is the reliability constraint, which requires the model to generate answers based on the given context to avoid generating false information; second is the integrity constraint, which ensures that the answers cover all aspects that users care about; finally is the security constraint, in issues related to medication safety, the model will emphasize the importance of referring to medical advice. In addition, the system also implements an intelligent answer expansion mechanism. In addition to directly answering users' questions, it will also supplement relevant medication precautions or possible risk warnings. The specific interface is as Figures 11 to 13 shown.

[0149] Through the above methods, the present invention can provide more accurate, professional, and safe medication consultation services, ensuring that users obtain correct guidance and suggestions during the medication process, thereby effectively improving the safety and accuracy of medication.

[0150] The present application also provides an intelligent medication consultation device based on a large model, including an acquisition module configured to acquire user input information; a text processing module configured to, when the user input information is text, perform feature extraction on the text to obtain a first feature representation, perform block retrieval on the electronic instruction manual based on the first feature representation, and perform context integration on the block retrieval results to generate medication consultation information for responding to the user input information; and an image processing module configured to, when the user input information is a picture, perform medicine box detection on the picture, identify a target medicine box, perform cross-modal feature extraction based on the target medicine box to obtain a second feature representation, and perform cross-modal retrieval based on the second feature representation to generate medication consultation information for responding to the user input information.

[0151] The present application also provides an intelligent medication consultation device based on a large model, including an acquisition module configured to acquire user input information; a text processing module configured to, when the user input information is text, perform feature extraction on the text to obtain a first feature representation, perform block retrieval on the electronic instruction manual based on the first feature representation, and perform context integration on the block retrieval results to generate medication consultation information for responding to the user input information; and an image processing module configured to, when the user input information is a picture, perform medicine box detection on the picture, identify a target medicine box, perform cross-modal feature extraction based on the target medicine box to obtain a second feature representation, and perform cross-modal retrieval based on the second feature representation to generate medication consultation information for responding to the user input information.

[0152] The present application also provides a cross-modal retrieval and matching device for drug information based on AI, including: an acquisition module configured to acquire a target medicine box picture to be identified; an extraction module configured to perform feature extraction on the target medicine box picture by using a multi-modal drug matching model to obtain a feature representation; and a matching module configured to perform cross-modal matching based on the feature representation by using the multi-modal drug matching model to match drug information related to the target medicine box; wherein, the multi-modal drug matching model is obtained by: performing domain adaptation training on a pre-trained multi-modal network model based on a graphic-text pair training set in the drug field to obtain the multi-modal drug matching model.

[0153] The present application also provides an AI-based rotating medicine box recognition and calibration device, including: an acquisition module configured to acquire a target medicine box image to be recognized; a recognition module configured to recognize the target medicine box image to be recognized by using a medicine box detection model and correct the rotation angle of the recognized target medicine box to the standard horizontal direction; wherein, the medicine box detection model is obtained through the following: based on a medicine box data set with rotation angle annotations, training a neural network model to obtain the medicine box detection model.

[0154] It should be noted that: for the device provided in the above embodiment, only the above division of each functional module is used for illustration. In actual application, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the corresponding method embodiment belong to the same concept, and the specific implementation process can be seen in the method embodiment, which will not be elaborated here.

[0155] Figure 14 The structural schematic diagram of an electronic device suitable for implementing the embodiments of the present disclosure is shown. It should be noted that Figure 14 The shown electronic device is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present disclosure.

[0156] As Figure 14 shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 into the random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other through a bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0157] The above is only the preferred embodiment of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. An intelligent medication consultation method based on a large model, characterized in that, Including: Obtain user input information; When the user input information is text, use the pre-trained language model BGE to extract features from the text to obtain a first feature representation. Based on the first feature representation, perform chunk retrieval on the electronic drug instructions, and perform context integration on the chunk retrieval results to generate drug use consultation information for responding to the user input information; When the user input information is a picture, perform medicine box detection on the picture, identify the target medicine box, and perform cross-modal feature extraction based on the target medicine box to obtain a second feature representation. Based on the second feature representation, perform cross-modal retrieval to generate drug use consultation information for responding to the user input information.

2. The method according to claim 1, characterized in that, The method of extracting features from the text to obtain a first feature representation, performing chunk retrieval on the electronic drug instructions based on the first feature representation, and performing context integration on the chunk retrieval results to generate drug use consultation information for responding to the user input information includes: When the text is recognized as a question, perform QA feature extraction on the text to obtain the first feature representation. Based on the first feature representation, perform chunk retrieval on the electronic drug instructions, and perform context integration on the retrieval results to generate drug use consultation information for responding to the user input information; When the text is recognized as non-question, perform cross-modal feature extraction on the text to obtain the first feature representation. Based on the first feature representation, perform cross-modal retrieval, match the associated medicine box image and text description from the drug image-text database to obtain the cross-modal retrieval result. Based on the cross-modal retrieval result, perform chunk retrieval on the electronic drug instructions, and perform context integration on the retrieval results to generate drug use consultation information for responding to the user input information.

3. The method according to claim 1, wherein After identifying the target medicine box, the method further includes: Identify the main position of the target medicine box in the picture; When the rotation angle of the main position deviates from the standard horizontal direction, use the affine transformation algorithm to correct the rotation angle of the main position to the standard horizontal direction.

4. The method according to claim 2, wherein The method of performing chunk retrieval on the electronic drug instructions based on the first feature representation and performing context integration on the chunk retrieval results to generate drug use consultation information for responding to the user input information includes: Based on the first feature representation, use the cosine similarity algorithm to retrieve relevant content segments in the instruction vector library generated based on the electronic drug instructions, and screen out the most relevant content paragraphs from the relevant content segments through semantic similarity ranking; Integrate the most relevant content paragraphs as context information, and based on the context information, use the large language model to generate drug use consultation information for responding to the user input information.

5. The method according to claim 4, characterized in that The instruction vector library is obtained through the following: Concatenate all relevant fields of the electronic drug instructions into a complete document in logical order, and perform standardization preprocessing on the concatenated document; For the preprocessed document, it is chunked with a preset number of characters as the basic unit, and an overlapping area with a preset overlap degree is set between adjacent chunks. Among them, the preset number of characters is 1024 characters, and the preset overlap degree is 10% of the number of characters in adjacent chunks; Based on the chunked document, the instruction vector library is constructed.

6. The method according to claim 1, wherein Performing cross-modal retrieval based on the second feature representation, including: Based on the second feature representation, using a multi-modal drug matching model to perform cross-modal matching on the drug text-image vector database to obtain drug information; among them, the drug text-image vector database is constructed through at least one of the following: Combining the brand and drug name as the main text description, and combining the key features of the drug to form complete text description information; Performing data augmentation processing on the drug sample pictures through at least one of the following: random angle rotation within ±30°, brightness and contrast adjustment, and horizontal and vertical flipping; Performing data augmentation processing on the training sample words through at least one of the following: synonym replacement, and interchange of brand and generic name.

7. An intelligent medication consultation device based on a large model, characterized in that, Including: An acquisition module configured to acquire user input information; A text processing module configured to, when the user input information is text, perform feature extraction on the text to obtain a first feature representation, perform chunk retrieval on the electronic instruction based on the first feature representation, and perform context integration on the chunk retrieval result to generate medication consultation information for responding to the user input information; A picture processing module configured to, when the user input information is a picture, perform medicine box detection on the picture, identify the target medicine box, perform cross-modal feature extraction based on the target medicine box to obtain a second feature representation, and perform cross-modal retrieval based on the second feature representation to generate medication consultation information for responding to the user input information.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that, Including: A memory and a processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program runs, it causes the processor to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images