Intelligent medicine information multi-dimensional identification method and device

Abstract

The invention relates to the technical field of intelligent medical equipment, and discloses an intelligent medicine information multi-dimensional identification method and device, and the method comprises the steps: obtaining a medicine identifier and user information, and enabling a cloud end to evaluate a risk according to the information, and generating a self-adaptive verification strategy; the terminal extracts the real-time visual features of the medicine, and performs weighted comparison with the standard features according to the strategy to obtain a comprehensive confidence score; judging whether the event is a deviation event or not according to the score, if yes, triggering a multi-mode alarm, and uploading a deviation event log to a cloud; and the cloud side updates the user personalized deviation graph and the global medical knowledge graph by using the log, so that closed-loop learning and continuous optimization of the system are realized. The device comprises an intelligent identification terminal with a built-in camera, a communication module and a man-machine interaction feedback unit; and the cloud server communicates with the terminal. According to the method, the drug safety is improved by dynamically evaluating the risk, adaptively adjusting the verification strategy and combining closed-loop learning and multi-mode barrier-free interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent medical devices, in particular to an intelligent medicine information multi-dimensional identification method and device. BACKGROUND

[0002] Ensuring the accuracy and safety of personal medication is a key link in daily health management, especially for chronic disease patients who need to take multiple medications for a long time. The current traditional method of relying on medicine packaging or instruction manual for information acquisition has inherent limitations. For the elderly or visually impaired people, the small font and complex professional terms on the medicine packaging constitute an insurmountable information barrier. This obstacle in information acquisition directly leads to a significant increase in medication safety risks, and patients are prone to misreading, misrecognition, and incorrect or missed medication, especially when the outer packaging of different medicines or the appearance of medicine tablets themselves are similar, the risk of confusion is higher.

[0003] In addition, existing medication assistance methods generally lack consideration of individual differences of users. They usually cannot provide personalized risk warnings and guidance by combining the user's specific health condition or past medication habits and deviation history, which makes it difficult to discover and avoid potential drug interactions or adverse reactions and other safety hazards in advance. Although some smartphone applications aimed at assisting medicine identification have appeared on the market, their operation methods have not fully met the needs of special groups. The complex interaction logic based on touch screens presents natural operational difficulties for visually impaired users, and for some elderly users, there is a certain threshold for learning and using these applications.

[0004] Therefore, the present application proposes an intelligent medicine information multi-dimensional identification method and device, which can provide a convenient operation, interactive friendly, and intelligent and personalized risk assessment capability of medicine information identification and safety verification service for special user groups such as the elderly and visually impaired people, to effectively make up for the shortcomings of the prior art. SUMMARY

[0005] In view of the shortcomings of the prior art, the present application provides an intelligent medicine information multi-dimensional identification method and device, which solves the problem that the medicine information identification method usually adopts a fixed identification strategy, lacks dynamic assessment capability of personalized risks caused by specific users and specific medicine combinations, and lacks a mechanism for adaptively adjusting the verification depth according to the identification risks, which leads to insufficient identification accuracy and safety hazards when facing easily confused medicines or high-risk medication scenarios.

[0006] To solve the above technical problems, the present application provides an intelligent medicine information multi-dimensional identification method, device and system.

[0007] The first aspect of the present application provides a multi-dimensional intelligent drug information recognition method. The method improves the accuracy and safety of drug recognition by dynamically evaluating risks and generating adaptive verification strategies. The method comprises the following steps:

[0008] Step S1, obtaining a drug identifier and calling corresponding standard visual feature descriptors according to the drug identifier, and obtaining user health information and historical deviation events associated with the current user.

[0009] In one embodiment, the image of the drug package is captured by the camera on the device, and the drug identifier is extracted through the image decoding algorithm. The standard visual feature descriptor is a data structure pre-established and stored in the cloud server visual feature library, which is used to define the standard appearance of the drug. The data structure can include geometric features, color features, size features, and surface mark features. At the same time, the health information of the user is loaded from the user database of the cloud server, and the historical deviation events of the user are loaded from the personalized deviation atlas of the user.

[0010] Step S2, based on the obtained drug identifier, user health information and historical deviation events, a comprehensive risk index is calculated, and an adaptive visual verification strategy is generated using the comprehensive risk index.

[0011] This step is one of the core innovations of the method of the present application. The comprehensive risk index is a quantitative value used to represent the risk level of confusion or error in the current medication scenario. The calculation process is as follows: first, from a global medical knowledge graph, the corresponding drug risk factor is retrieved according to the drug identifier, and the user risk factor is determined according to the user health information. Then, based on the historical deviation events, the historical deviation risk factor is determined. The comprehensive risk index is obtained by weighted summation of the three risk factors, and the calculation process can be represented by the following formula:

[0012]

[0013] In the formula, R comprehensive is the comprehensive risk index; are the normalized drug risk factor, user risk factor and historical deviation risk factor, respectively; w d ,w u ,w h are the preset weight coefficients corresponding to the three normalized risk factors, respectively.

[0014] Then, an adaptive visual verification strategy is generated using the calculated comprehensive risk index. The generation process of the strategy is as follows:

[0015] The adaptive visual verification strategy includes a set of features to be verified and a set of weights corresponding to each feature in the set. The generation of the strategy takes the comprehensive risk index as input. When the value of the comprehensive risk index is high, it means that the risk is high. At this time, the generated strategy will contain a larger number of features to be verified, and the weight value corresponding to the key identification feature (such as a specific mark or color) used to distinguish easily confused medicines will be increased.

[0016] Step S3, capturing a real-time image of the medicine entity by the camera, and extracting a real-time visual feature descriptor from the real-time image. The data structure of the real-time visual feature descriptor is consistent with that of the standard visual feature descriptor.

[0017] Step S4, according to the generated adaptive visual verification strategy, comparing the standard visual feature descriptor and the extracted real-time visual feature descriptor, thereby calculating the comprehensive confidence score.

[0018] The comparison process of this step is dynamically regulated by the adaptive visual verification strategy. Specifically, for each feature in the set of features to be verified specified in the strategy, the sub-item similarity of the standard visual feature descriptor and the real-time visual feature descriptor on this feature is calculated. Then, using the weight set specified in the strategy, the weighted sum of all calculated sub-item similarities is calculated to obtain the comprehensive confidence score. Its calculation process can be represented by the following formula:

[0019]

[0020] In the formula, S confidence is the final calculated comprehensive confidence score; F is the set of features to be verified specified by the adaptive visual verification strategy; i is a specific visual feature in the feature set F; w i is the weight specified by the adaptive visual verification strategy for feature i, which is determined by the comprehensive risk index; is the quantized value of feature i in the standard visual feature descriptor; is the quantized value of feature i in the real-time visual feature descriptor; g i is a similarity calculation function preset for the data type of feature i.

[0021] Step S5, when the comprehensive confidence score is lower than a preset confidence threshold, it is determined that a deviation event occurs, and a preset alarm is triggered via the loudspeaker. In an embodiment, the step of triggering the alarm also includes driving the vibration motor in the device to produce vibration to provide multi-modal warning. At the same time, the standard visual feature descriptor, the real-time visual feature descriptor and the adaptive visual verification strategy involved in this event are packaged as a structured deviation event log.

[0022] In a further embodiment, the method further comprises uploading the structured bias event log to a cloud server. After receiving the log, the cloud server uses it as an input to update the user personalized bias graph of the current user. In addition, the cloud server can also aggregate and analyze bias event logs collected from multiple users, and update the global medical knowledge graph with the information when a drug pair with high global confusion is identified. Through this feedback mechanism, the system can continuously learn and improve the accuracy of subsequent risk assessment.

[0023] The second aspect of the application provides an intelligent drug information multi-dimensional recognition device, which is designed to execute any of the above methods. The device comprises a shell, a main board body encapsulated in the shell, an integrated main control chip responsible for executing the method steps, and auxiliary electronic components such as an audio codec chip, a positioning module, a communication module, a charging management chip, and a vibration motor; a camera and a front-end integrated light supplementing device electrically connected to the main control chip for image acquisition; a microphone and a speaker electrically connected to the main control chip through the audio codec chip for voice interaction; a silicone Braille key and a Braille voice input key electrically connected to the main control chip through a microswitch for receiving user instructions; a display screen electrically connected to the main control chip for displaying information; a lithium battery and a charging data interface providing power to the entire device through the charging management chip.

[0024] The third aspect of the application provides an intelligent drug information multi-dimensional recognition system, which comprises the above-mentioned intelligent drug information multi-dimensional recognition device and a cloud server. In this system, the functions of each module are clearly defined: the information acquisition module and the image processing module in the device are responsible for collecting various types of raw data; the risk assessment module and the strategy generation module deployed on the cloud server are responsible for performing core computationally intensive tasks; the comparison and decision module and the alarm and log module in the device are responsible for comparison, decision-making, and interaction with the user. This terminal and cloud collaborative architecture can fully utilize the immediacy of the terminal and the powerful computing and storage capabilities of the cloud.

[0025] The application provides an intelligent drug information multi-dimensional recognition method and device. The following beneficial effects are achieved:

[0026] 1、The present application realizes the technical leap from static drug identification to dynamic risk perception by establishing a multi-dimensional risk assessment model. Instead of identifying the drug itself in isolation, the method fuses information from drug identifiers, user health information, and historical deviation events to calculate a quantitative comprehensive risk index. This mechanism enables the system to make personalized risk level judgments for each specific medication behavior, thereby predicting the potential for incorrect medication before identification and providing a decision basis for subsequent precise verification.

[0027] 2、The present application proposes an adaptive visual verification strategy generation mechanism based on risk index, improving the reliability and efficiency of drug visual verification. According to the level of comprehensive risk index, the mechanism dynamically adjusts the depth and breadth of visual feature comparison, specifically by adjusting the feature set to be verified and the feature weight set. When the risk is high, the system automatically adopts a more stringent and detailed comparison strategy; when the risk is low, it adopts a faster regular comparison, thereby allocating limited computing resources to the most critical verification link, achieving an optimal balance between identification accuracy and response speed.

[0028] 3、The present application constructs a deviation event-driven closed-loop learning and knowledge updating system, enabling the system to continuously optimize itself. When a deviation event occurs, the system not only records the event itself, but also uploads the structured deviation event log to the cloud. The cloud updates the user's personalized deviation graph by analyzing individual deviation event logs, and updates the global medical knowledge graph by aggregating and analyzing multiple user logs. This enables the system to learn from mistakes and continuously improve its cognitive accuracy for individual users and global drug confusion risks, ensuring the long-term effectiveness and accuracy of the identification model.

[0029] 4、The present application integrates camera, speaker, vibration motor, and silicone Braille key hardware modules functionally, constructing a complete, multi-modal, and barrier-free interaction system. This system does not rely solely on vision or complex operations, but through the coordinated work of visual acquisition, voice broadcast, tactile vibration, and Braille key input, it provides a complete and clear interaction path from information acquisition to result feedback for visually impaired or operationally challenged user groups. This greatly reduces the threshold for special groups to use intelligent devices for drug verification, fundamentally improving the safety and convenience of the medication process. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 Overall perspective of the device of the present application Figure 1 ;

[0031] Figure 2 Overall perspective of the device of the present application Figure 2 ;

[0032] Figure 3 is a schematic view of the side of the present application;

[0033] Figure 4 is a schematic view of the back of the present application;

[0034] Figure 5 is a schematic view of the internal structure of the present application;

[0035] Figure 6 is a hardware system block diagram of the present application;

[0036] Figure 7 is a whole flow chart of the present application;

[0037] Figure 8 is a schematic view of the functional modules and data interaction of the present application.

[0038] Wherein, 1, shell; 2, camera; 3, front-end integrated light supplementing device; 4, loudspeaker; 5, microphone; 6, silica gel Braille key; 7, Braille voice input key; 8, display screen; 9, charging data interface; 10, charging data interface; 11, mainboard main body; 12, main control chip; 13, audio codec chip; 14, positioning module; 15, communication module; 16, lithium battery; 17, charging management chip; 18, vibration motor; 19, micro switch. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the specification of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] Referring to the drawings Figures 1 to 4 It shows the physical form of the intelligent medicine information multi-dimensional identification device according to one embodiment of the present application. The present application provides an intelligent medicine information multi-dimensional identification system, which is physically composed of an intelligent medicine information multi-dimensional identification device and a cloud server. The intelligent identification terminal and the cloud server perform data interaction through a wireless communication network. The system aims to dynamically evaluate risks and adaptively adjust verification strategies to solve the technical problems of insufficient accuracy and safety caused by fixed identification strategies in the prior art.

[0041] Referring to the drawings Figure 7 The intelligent medicine information multi-dimensional identification method provided by the present application can include the following steps:

[0042] S1, obtain a drug identifier, and retrieve a corresponding standard visual feature descriptor according to the drug identifier, and obtain user health information and historical deviation events associated with the current user.

[0043] S2, based on the obtained drug identifier, user health information and historical deviation events, calculate a comprehensive risk index, and generate an adaptive visual verification strategy using the comprehensive risk index.

[0044] S3, collect real-time images of the drug entity through the camera 2, and extract real-time visual feature descriptors from the real-time images.

[0045] S4, according to the generated adaptive visual verification strategy, compare the retrieved standard visual feature descriptors with the extracted real-time visual feature descriptors, and calculate a comprehensive confidence score.

[0046] S5, when the comprehensive confidence score is lower than the preset confidence threshold, it is determined that a deviation event occurs, and a preset alarm is triggered via the loudspeaker 4.

[0047] In one embodiment, the method further comprises the steps of uploading the deviation event log to the cloud server, and the cloud server updating the user personalized deviation graph and the global medical knowledge graph using the log.

[0048] Referring to the drawings Figure 8 To implement the above method, the intelligent drug information multi-dimensional recognition system provided by the present application can be divided into several functional modules. The system includes: an information acquisition module 20 configured in the intelligent recognition terminal, used to acquire all initial data required for performing the recognition task, including the drug identifier, the user health information and the historical deviation events. An image processing module 21 configured in the intelligent recognition terminal, used to perform step S3, collect drug entity images and extract real-time visual feature descriptors. A risk assessment module 22 configured in the cloud server, used to perform the first half of step S2, i.e. calculating a comprehensive risk index based on multi-dimensional information. A strategy generation module 23 configured in the cloud server, used to perform the second half of step S2, i.e. generating an adaptive visual verification strategy based on the comprehensive risk index. A comparison and decision module 24 configured in the intelligent recognition terminal, used to perform step S4, receive the strategy and standard features issued by the cloud, compare and calculate a comprehensive confidence score. An alarm and log module 25 configured in the intelligent recognition terminal, used to perform step S5, judge according to the output result of the comparison and decision module 24, and trigger the alarm and generate the log when necessary. A system update module 26 configured in the cloud server, used to receive the deviation event log uploaded by the alarm and log module 25, and update the knowledge graph and user data in the cloud.

[0049] Embodiment one: hardware structure and circuit connection of intelligent medicine information multi-dimensional recognition device

[0050] Referring to the drawings Figures 1 to 5 The overall structure of the intelligent medicine information multi-dimensional recognition device in this embodiment is designed as a cuboid-shaped device that is easy to hold with one hand. The device includes a shell 1, which is integrally formed by an injection molding process of ABS engineering plastic, and the external dimensions can be 15 cm long, 3 cm wide, and 2.5 cm high. To increase the stability of holding, a non-slip texture with a depth of 0.3 mm is provided on the surface of the holding area of the shell 1.

[0051] Each external functional component of the device has a clear physical layout on the shell 1. At the front end of the shell 1, a camera 2 and a front-end integrated light supplement device 3 adjacent to the camera 2 are provided, and the lens part of the camera 2 can protrude 0.5 cm from the surface of the shell 1 to obtain a clear image acquisition field. On one side of the shell 1, a silica gel Braille key 6 and a Braille voice input key 7 are provided for users to input instructions, and the surface of the keys has standard Braille bumps to facilitate visually impaired users to locate and operate through touch.

[0052] On the front of the shell 1, a display screen 8 is provided to feed back visual information to the user. A speaker 4 opening hole for sound output and a microphone 5 opening hole for sound input are also provided at appropriate positions of the shell 1. At the end of the shell 1, a charging data interface 9 is provided, which can be a Type-C physical interface, for charging and data transmission.

[0053] Referring to the drawings Figure 5 The inside of the shell 1 is provided with a containing cavity for installing the internal electronic elements of the device. A mainboard body 11 is fixed in the containing cavity, which is a printed circuit board (PCB) with a size of 8 cm x 3 cm. The core integrated circuits of the device, such as the main control chip 12, the audio codec chip 13, the positioning module 14, the communication module 15, and the charging management chip 17, are all soldered on the mainboard body 11.

[0054] Other components inside the device are arranged around the mainboard body 11. A lithium battery 16 is fixed on one side of the mainboard body 11 and connected to the charging management chip 17 and the power management unit on the mainboard body 11 through wires. A vibration motor 18 is fixed to the inner wall of the shell 1 near the holding area and is electrically connected to the driving circuit on the mainboard body 11. The external components such as the camera 2, the display screen 8, and the keys are connected to the corresponding interfaces on the mainboard body 11 through flexible flat cables (FPC) or wires. Through the above arrangement, a compact and functionally integrated hardware whole is formed.

[0055] Referring to the drawings Figure 5 and the drawings Figure 6 , the drawingsFigure 6 Figure 1 is a hardware system block diagram of the intelligent medicine information multi-dimensional recognition device according to one embodiment of the present application. The core processing and control unit of the device is the main control chip 12, which is arranged on the mainboard body 11.

[0056] In one specific embodiment, the main control chip 12 can be selected as a microcontroller of ESP32-S3-WROOM-1-N16R8 model. The microcontroller of this model integrates a high-performance processor core, a memory, and 2.4GHz-Wi-Fi and Bluetooth 5.0 wireless communication functions.

[0057] The main control chip 12 undertakes the functions of data processing, task scheduling, and peripheral control in the entire device. In terms of functions, the main control chip 12 is responsible for running the built-in real-time operating system (such as FreeRTOS) and the upper-layer application program, processing data from various hardware modules, such as decoding image data obtained from the camera 2, and performing part of the local computing tasks of the comparison decision module 24 and the alarm and log module 25.

[0058] At the same time, the main control chip 12 is responsible for scheduling multiple concurrent tasks of the device. For example, when the user presses the key, the main control chip 12 is awakened through the GPIO interrupt, and performs the corresponding task according to the preset program logic, such as starting the camera 2 to perform image acquisition, or driving the speaker 4 to perform voice broadcast.

[0059] In addition, the main control chip 12 realizes direct control over all other hardware modules in the device through its rich general and special peripheral interfaces. Referring to the attached Figure 6 , the main control chip 12 is connected with the camera 2 through the digital camera interface (DCMI) to receive image data; is connected with the display screen 8 through the serial peripheral interface (SPI) to transmit display content; performs serial communication with the positioning module 14 through the universal asynchronous receiver-transmitter (UART) interface; receives the switch signal of the key through the general input-output (GPIO) pin, and controls the start-stop of the front-end integrated light supplementing device 3 and the vibration motor 18; cooperates with the communication module 15 through the internally integrated or externally connected Wi-Fi / Bluetooth controller to realize wireless data exchange with the cloud server; and communicates with the audio codec chip 13 through the integrated circuit built-in sound I 2S interface, thereby controlling the audio acquisition of the microphone 5 and the audio playback of the speaker 4.

[0060] Referring to the attached Figure 1 、 Figure 5 and Figure 6 , the multi-dimensional information acquisition unit of the device is composed of an image information acquisition part and a user instruction and environment information acquisition part, to obtain various types of raw data required for performing the recognition task.

[0061] The image information acquisition part includes a camera 2 and a front-end integrated light supplement device 3. In a specific embodiment, the camera 2 can be an OV2640 type CMOS image sensor, which supports 2 million pixels and UXGA (1632x1232) resolution, and can perform high-speed image data transmission with the main control chip 12 through its DCMI digital camera interface. The front-end integrated light supplement device 3 can be a high-brightness LED, which is connected to a GPIO pin of the main control chip 12 through a current-limiting resistor. When the ambient light is insufficient, the main control chip 12 can output a high-level signal through the GPIO pin to light up the LED, thereby supplementing the light for the shooting target. The image information acquisition part cooperates together to acquire the image of the medicine packaging in step S1 to decode the medicine identifier, and to acquire the real-time image of the medicine entity in step S3 to extract the real-time visual feature descriptor.

[0062] The user instruction and environment information acquisition part includes a microphone 5, a silica gel Braille key 6, and a Braille voice input key 7. The microphone 5 can be a high-sensitivity microphone, which is physically connected to the input end of the audio codec chip 13. After the audio codec chip 13 performs analog-to-digital conversion on the collected analog audio signal, it transmits the digital audio stream to the main control chip 12 through the I2S interface for processing to receive the user's voice instruction. The surfaces of the silica gel Braille key 6 and the Braille voice input key 7 are respectively provided with Braille codes in accordance with international standards, and the internal structure is that a micro switch 19 is arranged below each silica gel key. Each micro switch 19 is connected to a GPIO pin of the main control chip 12 through an independent circuit, which can include a pull-up resistor to provide a stable level signal. When the user presses a key, the corresponding micro switch 19 is closed, causing the level of the GPIO pin connected to it to jump, and the main control chip 12 detects this level jump to recognize the user's key instruction.

[0063] Referring to the accompanying drawings Figure 1 , Figure 5 and Figure 6 , the man-machine interaction feedback unit of the device is composed of a loudspeaker 4, a display screen 8, and a vibration motor 18. This unit is responsible for outputting the processing results and state information of the system to the user in multiple modalities such as voice, vision, and touch.

[0064] The speaker 4 can be a micro speaker configured to output a volume no less than 85 decibels to ensure the clarity of the broadcast content. The driving of the speaker 4 is not directly completed by the main control chip 12, but is completed by the audio codec chip 13. Specifically, the main control chip 12 synthesizes the text data to be broadcast into a digital audio stream, and sends it to the audio codec chip 13 through an I2S interface. The digital-to-analog converter (DAC) inside the audio codec chip 13 converts the digital audio stream into an analog audio signal, and then drives the speaker 4 to sound through the internal power amplifier. The speaker 4 is used to perform voice broadcast of drug information, system prompt sound, and trigger a preset alarm voice in step S5.

[0065] The display screen 8 can be a 2.4-inch IPS display screen with a physical resolution of 320x240 pixels. The display screen 8 is electrically connected to the main control chip 12 through a serial peripheral interface (SPI). The main control chip 12 sends instructions and pixel data to the driving integrated circuit of the display screen 8 through the SPI interface, thereby controlling the specific content displayed on the screen. The display screen 8 is used to provide visual feedback to the user, such as displaying the drug name, usage and dosage, comprehensive confidence score, or alarm information in large font mode.

[0066] The vibration motor 18 can be a flat vibration motor fixed to the inner wall of the shell 1. The vibration motor 18 is electrically connected to a GPIO pin of the main control chip 12 through a driving circuit. The driving circuit can be composed of a triode or MOSFET to provide the required current for driving the motor. The main control chip 12 controls the driving circuit to turn on and off by controlling the output of high and low level signals of the GPIO pin, thereby realizing the start and stop control of the vibration motor 18. The vibration motor 18 is used to provide tactile feedback, such as working in conjunction with the speaker 4 when triggering an alarm, or providing confirmation vibration during specific operations.

[0067] Referring to the accompanying drawings Figure 5 and 6 The device further includes a communication module 15, a positioning module 14, and a power management unit composed of a lithium battery 16 and a charging management chip 17 to ensure the networking, positioning, and long-lasting endurance of the device.

[0068] The communication module 15 is used to realize the data exchange between the device and the cloud server. In one embodiment, the functions of the communication module 15 can be realized by the 2.4 GHz Wi-Fi radio frequency circuit and protocol stack integrated in the main control chip 12. The main control chip 12 is connected to the wireless local area network through the integrated module, and then communicates with the cloud server through the Internet protocol. In another embodiment, in order to maintain the connection in the scene without Wi-Fi network coverage, the communication module 15 can also be a separate 5G / GSM communication module, which is electrically connected to the main control chip 12 through the UART interface. The communication module 15 is responsible for performing all data interaction tasks with the cloud server, including sending the request for obtaining user health information and historical deviation events in step S1, and uploading the structured deviation event log after the method is executed.

[0069] The positioning module 14 is used to obtain the current geographic position coordinates of the device. In a specific embodiment, the positioning module 14 can use a satellite navigation module of the ATGM332D-5NR32 type, which supports multiple satellite navigation systems such as Beidou and GPS. The positioning module 14 is arranged on the main board body 11 and is electrically connected to a group of GPIO pins (such as GPIO16 / GPIO17) of the main control chip 12 through its UART interface (TXD / RXD pins) for serial data communication at a baud rate of 9600. The main control chip (12) receives the NMEA-0183 format data frame output by the positioning module 14 through the interface, and parses the latitude and longitude position information therefrom. The obtained position information can be packaged into the deviation event log to provide geographical position context for subsequent data analysis.

[0070] The power management unit is responsible for providing stable and reliable power for the entire device. The unit includes a rechargeable lithium battery 16 and a charging management chip 17. The lithium battery 16 can be a 18650 type lithium ion battery with a capacity of 1500mAh, which serves as the main power source of the device. The charging management chip 17 can be an integrated circuit of the BQ24040DSQR type, which is connected to the charging data interface 9 outside the device at one end and to the lithium battery 16 and the main power supply circuit of the main board body 11 at the other end. The functions of the charging management chip 17 include: managing the current input through the charging data interface 9, safely charging the lithium battery 16 in constant current / constant voltage mode; providing overcharge, overdischarge and overcurrent protection functions; at the same time, it can provide battery voltage information to the analog-to-digital conversion (ADC) pin of the main control chip 12, so that the main control chip 12 can monitor the real-time power and trigger a low power prompt when the power is below the preset threshold.

[0071] Embodiment two: execution flow and core algorithm principle of intelligent medicine information multi-dimensional identification method

[0072] Refer to the attachedFigure 6 and Figure 7 This embodiment will describe in detail the specific execution flow of the intelligent drug information multidimensional identification method provided by the present invention, in conjunction with the hardware system of Embodiment 1.

[0073] This process corresponds to Appendix Figure 7 Step S1 aims to acquire all the initial data required for subsequent risk assessment and comparative decision-making. This process is performed by the information acquisition module 20 on the device.

[0074] This process is triggered by user operation. In one embodiment, the user points the device's camera 2 at the barcode or QR code on the medicine packaging and presses the silicone Braille button 6 or the Braille voice input button 7 to initiate a recognition request. After receiving the instruction signal from the microswitch 19, the main control chip 12 starts the information acquisition process.

[0075] First, the operation of obtaining the drug identifier is performed. The main control chip 12 drives the camera 2 to capture real-time images of the drug packaging. If the ambient light is insufficient, the main control chip 12 will simultaneously drive the front-end integrated supplementary lighting device 3 to provide supplementary lighting. The camera 2 transmits the captured image data stream to the main control chip 12 through the DCMI interface. The main control chip 12 internally runs a pre-built barcode decoding library (such as the ZBar decoding library) to process the received image data to identify and decode the barcode or QR code, thereby obtaining a string composed of numbers or characters, which is the drug identifier.

[0076] After successfully acquiring the drug identifier, the system then retrieves the user's health information and historical deviation events associated with the current user, and calls up the standard visual feature descriptor. The main control chip 12 establishes a secure network connection with the preset cloud server through its communication module 15. Subsequently, the main control chip 12 encapsulates the acquired drug identifier and a locally stored, unique device or user identifier into a data request packet, and sends it to the cloud server through the network connection.

[0077] Upon receiving the data request packet, the cloud server parses it. Based on the drug identifier, the server retrieves a matching, pre-structured standard visual feature descriptor from its internal visual feature library. Simultaneously, based on the device or user identifier, the server loads the user's associated health information (e.g., age, allergy history, diagnosed diseases, etc.) from its user database and loads the user's historical deviation events from the user's personalized deviation profile.

[0078] The cloud server encapsulates the standard visual feature descriptor, user health information, and historical deviation event data into a response data packet and returns it to the intelligent recognition terminal. The host chip 12 of the terminal receives and parses the response data packet through the communication module 15, and temporarily stores all the obtained information in the internal random access memory (RAM). At this point, step S1 is completed, and the system is ready for all the required input data for subsequent step S2.

[0079] Referring to the accompanying Figure 7 and Figure 8 , the present process corresponds to step S2 in the accompanying Figure 7 , which is one of the core parts of the technical solution of the present application. The process is executed on the cloud server and is cooperatively completed by the risk assessment module 22 and the strategy generation module 23. The purpose is to perform a quantitative risk assessment on the current medication behavior based on the multi-dimensional information obtained from step S1, and generate a customized verification strategy for subsequent comparison.

[0080] The risk assessment module 22 on the cloud server first performs the calculation of the comprehensive risk index. The module receives the drug identifier, user health information, and historical deviation event uploaded from the intelligent recognition terminal. Based on these inputs, the risk assessment module 22 calculates three sub-risk factors: drug risk factor, user risk factor, and historical deviation risk factor.

[0081] The calculation of the drug risk factor (R d ) is based on the inherent properties of the drug retrieved from the global medical knowledge graph based on the drug identifier. These properties can include whether the drug is defined as a high-alert drug, whether there are other drugs with known similar appearances that are easy to confuse, etc. The risk assessment module 22 scores and weights these properties according to the pre-set rule table to obtain the drug risk factor. Its calculation can be represented by the following formula:

[0082]

[0083] In the formula, R d is the drug risk factor; N d is the total number of pre-set drug risk attributes; c d,j is the pre-set score corresponding to the jth drug risk attribute; I d,j is an indicator function, I d,j = 1 when the retrieved drug has the jth risk attribute, otherwise 0.

[0084] The user risk factor R uThe calculation of the user risk factor is based on the correlation analysis of the user's health information and the current drug. The risk assessment module 22 compares the user's age, allergy history, past medical history, etc. with the drug's target population, contraindications, etc. If there is a potential risk, for example, the user's age falls within the cautious use range of the drug, or the user has a disease that is a contraindication of the drug, the user risk factor value will be increased accordingly. Its calculation can be represented by the following formula:

[0085]

[0086] In the formula, R u is the user risk factor; N u is the total number of preset user risk correlation rules; c u,k is the preset score corresponding to the kth user risk correlation rule; I u,k is an indicator function, which is 1 when the user's health information triggers the kth risk correlation rule, and 0 otherwise. u,k

[0087] The calculation of the historical deviation risk factor (R h ) is based on the historical deviation events obtained from the user's personalized deviation map. The risk assessment module 22 analyzes whether the user has ever had a medication deviation event related to the current drug (e.g. the drug itself, similar drugs or drugs with similar appearance). The value of this factor can be associated with the frequency, severity and time distance of the historical deviation events.

[0088] After calculating the above three sub-risk factors, the risk assessment module 22 obtains the final comprehensive risk index by weighted summation. Its calculation is represented by the following formula:

[0089]

[0090] In the formula, R comprehensive is the comprehensive risk index; are the normalized drug risk factor, user risk factor and historical deviation risk factor respectively, whose value range is [0, 1]; w d , w u , w h are the preset weight coefficients corresponding to the three normalized risk factors respectively, and satisfy w d + w u + w h = 1.

[0091] The strategy generation module 23 receives the comprehensive risk index R comprehensive ​and generate an adaptive visual verification strategy based on the value of the index. The strategy is a data structure that specifies which visual features need to be verified in the subsequent comparison step S4, and the importance of each feature in the final decision.

[0092] In particular, the adaptive visual verification strategy includes a feature set F to be verified, and a weight set W corresponding to each feature in the set F. The strategy generation module 23 internally stores a mapping table that maps different ranges of the comprehensive risk index R comprehensive to different feature sets F and weight sets W.

[0093] For example, when R comprehensive is in the low-risk range, the generated feature set F can only contain easily identifiable macro features, such as {color, shape}. When R comprehensive is in the high-risk range, the generated feature set F is expanded to include more detailed features, such as {color, shape, size, surface markings}.

[0094] Meanwhile, the weight values w i in the weight set W are also dynamically adjusted based on R comprehensive . For the i-th feature in the feature set F, the adjusted weight can be calculated as follows:

[0095] w′ i = w i,base ·(1 + a i · R comprehensive );

[0096] where w′ i is the adjusted non-normalized weight of feature i; w i,base is a pre-set base weight value for feature i; a i is the risk sensitivity coefficient of feature i, which is a pre-set value indicating the degree of importance of feature i increasing with risk; and R comprehensive is the comprehensive risk index. After calculating the non-normalized weights of all the features to be verified, the strategy generation module 23 normalizes them so that the sum of all the weights in the weight set W is 1.

[0097] Finally, the cloud server sends the generated adaptive visual verification strategy (i.e., the feature set F and the weight set W) back to the intelligent recognition terminal, along with the standard visual feature descriptors obtained in step S1, for use in step S4. At this point, step S2 is complete.

[0098] Referring to Figures 1 Figure 6 , Figure 7 , and Figure 8 , the present process corresponds to Figure 1 Figure 7Steps S3 and S4 in FIG. 1 are executed locally on the smart identification terminal. This flow is accomplished by the image processing module 21 and the comparison decision module 24 on the terminal, aiming to obtain the real-time visual features of the drug entity and make comparison according to the adaptive strategy received from the cloud, and finally obtain a quantitative confidence score.

[0099] This flow is started after the terminal receives the adaptive visual verification strategy and the standard visual feature descriptor issued by the cloud. First, the system prompts the user to place a single dose of the drug entity (e.g., a tablet or a capsule) in front of the camera 2 through the speaker 4 or the display screen 8.

[0100] Subsequently, Step S3 is executed, i.e., the real-time visual features are extracted. The image processing module 21 on the terminal drives the camera 2 to capture the real-time image of the drug entity. The captured image data is sent to the main control chip (12) for analysis. The image processing module 21 pre-processes the image, such as image segmentation to separate the drug entity from the background, and then for each feature required by the feature set F in the adaptive visual verification strategy, the corresponding extraction algorithm is executed. For example, when extracting color features, the color histogram of the drug region can be calculated; when extracting shape features, edge detection and contour analysis algorithms can be used to calculate the contour descriptor; when extracting size features, the pixel size in the image can be calculated; when extracting surface mark features, optical character recognition (OCR) or template matching algorithms can be run. The extracted results are integrated into a structured real-time visual feature descriptor.

[0101] Next, Step S4 is executed, i.e., adaptive comparison and decision are made. The comparison decision module 24 on the terminal receives three inputs: the standard visual feature descriptor obtained in Step S1, the adaptive visual verification strategy (including the feature set F and the weight set W) generated in Step S2, and the real-time visual feature descriptor extracted in Step S3.

[0102] The core task of the comparison decision module 24 is to calculate a comprehensive confidence score. The calculation is divided into two stages: item similarity calculation and weighted summation.

[0103] For each to-be-verified feature i specified in the feature set F of the adaptive visual verification strategy, the comparison decision module 24 first calculates the item similarity. This similarity is used to quantify the matching degree of the real-time visual features and the standard visual features in a single dimension. Its calculation process can be represented by the following formula:

[0104]

[0105] In the formula, sim i is the item similarity of feature i, whose value range is [0, 1]; is the quantized value of feature i in the standard visual feature descriptor; is the quantized value of feature i in the real-time visual feature descriptor; i is a similarity calculation function preset for the data type of feature i. For example, for numerical features such as size, the function can be a Gaussian function.

[0106] After the partial similarity of all features in the feature set F is calculated, the comparison decision module 24 sums up all the partial similarities according to the weight set W in the adaptive visual verification strategy, so as to obtain the final comprehensive confidence score. The score represents the degree of confidence of the matching between the real-time drug and the standard drug under the current risk assessment level. The calculation process is represented by the following formula:

[0107] S confidence =∑ i∈F w i ·sim i ;

[0108] In the formula, S confidence is the final comprehensive confidence score obtained by calculation; F is the feature set to be verified specified by the adaptive visual verification strategy; i is a specific visual feature in the feature set F; w i is the weight specified by the adaptive visual verification strategy for feature i, which is determined by the comprehensive risk index, and ∑ i∈F w i =1; sim i is the partial similarity of feature i.

[0109] The comprehensive confidence score S confidence obtained by calculation will be passed to the alarm and log module 25 as the direct basis for decision-making in the subsequent step S5. Thus, steps S3 and S4 are completed.

[0110] Referring to the accompanying Figure 7 and Figure 8 , the present process corresponds to steps S5, S6, S7 and S8 in the accompanying Figure 7 , and constitutes the response and optimization closed loop of the technical solution of the present application. The process describes how the system responds according to the decision result after completing the comparison decision, and records the event and drives the system to learn and update itself when there is a deviation.

[0111] The present process first executes step S5, which is executed locally at the intelligent identification terminal and is responsible to the alarm and log module 25. The alarm and log module 25 receives the comprehensive confidence score S confidence obtained by calculation from the comparison decision module 24. Subsequently, the alarm and log module 25 compares the comprehensive confidence score S confidence with a preset confidence threshold T confidenceS confidence <T confidence If S confidence ≥ T confidence , the system determines that the current recognition result is consistent with the standard, and the process proceeds to step S6.

[0112] When the process proceeds to step S6, the system determines that the current drug check is correct. The alarm and log module 25 provides clear and positive feedback to the user through the human-computer interaction feedback unit. In an embodiment, the main control chip 12 drives the loudspeaker 4 to play a voice that the drug check is correct, and displays a green confirmation mark on the display screen 8. After the positive feedback is completed, the current recognition process ends normally.

[0113] After determining that a deviation event occurs in step S5, the alarm and log module 25 immediately performs step S7, that is, generates a structured deviation event log locally. The log is a data packet that contains complete and traceable records of the current deviation event. The log can include the following fields: a unique identifier of the device or user, a timestamp of the event occurrence, a drug identifier that triggered the current process, a comprehensive risk index R comprehensive calculated by the cloud, an adaptive visual verification strategy (including a feature set F and a weight set W) issued by the cloud, a comprehensive confidence score S confidence calculated locally, a standard visual feature descriptor, a real-time visual feature descriptor extracted locally, and geographical position coordinates obtained from the positioning module 14.

[0114] After the log is generated, the alarm and log module 25 calls the communication module 15 to upload the structured deviation event log to the cloud server through a secure network connection.

[0115] The system update module 26 on the cloud server is responsible for performing step S8. The module continuously listens to and receives deviation event logs uploaded from the terminal. Whenever a log is received, the system update module 26 parses the log and uses the information in the log to update two core databases in the cloud, thereby realizing closed-loop learning of the system.

[0116] One of the update paths is to update the user's personalized bias graph. The system update module 26 locates the user's node in the graph according to the user identifier in the log, and associates the details of the current bias event (such as drug identifier, time, risk index, etc.) as a new event node with it. This operation will directly affect the calculation of the historical bias risk factor R h of the user in the future risk assessment, so that the system has a more accurate understanding of the specific risk pattern of the user.

[0117] The second update path is to update the global medical knowledge graph. The system update module 26 aggregates and analyzes the bias event logs from different users. When the analysis finds that a specific drug identifier appears significantly more frequently than the statistical average in bias events, or two different drugs are frequently misidentified in each other's identification process, the system can determine that there is a high risk of confusion between these drugs. At this time, the system update module 26 will add or strengthen the appearance similar or high risk attribute labels for the nodes of these drugs in the global medical knowledge graph. This update will affect the calculation of the drug risk factor R d of all users in the future when identifying these drugs, thereby improving the system's ability to predict universal risks.

[0118] Through the above closed-loop process of bias event handling, log uploading and cloud knowledge graph dual updating, the system of the present application can obtain information from each identification error, continuously optimize its risk assessment model and verification strategy, and thereby continuously improve the accuracy and safety of identification.

[0119] Referring to the accompanying Figures 1 to 8, In order to more completely illustrate how the intelligent medicine information multi-dimensional recognition system and method provided by the present application work in cooperation, a specific application scenario will be described below. The scenario is set as follows: a visually impaired user needs to take a new medicine named felodipine extended-release tablets for the first time, and the appearance of the medicine is similar to that of the user's commonly used medicine. The user holds the intelligent recognition device of the present application, presses the Braille voice input key 7 to start the process. After the device prompts, the user points the camera 2 at the medicine box, the system performs step S1 to obtain the medicine identifier and the user identifier, and interacts with the cloud server to obtain the standard visual features of the medicine and the health information of the user. The risk assessment module 22 of the cloud server performs step S2. After analysis, the system identifies the risks of similar appearance of the medicine and first-time taking of the user, and calculates a high comprehensive risk index. According to this, the strategy generation module 23 generates a high-level adaptive visual verification strategy, which requires comparison of multiple fine features including the surface mark of the tablet, and sets the weight of the surface mark as the highest. After the device receives the strategy, it prompts the user to compare the tablets. If the user mistakenly places the old medicine with a similar appearance in front of the camera, the comparison decision module 24 will calculate a comprehensive confidence score far below the preset threshold when performing steps S3 and S4 due to the extremely low similarity of the surface mark feature and the extremely high weight of the feature. The alarm and log module 25 then performs step S5 to trigger a strong multi-modal alarm of sound, light and vibration, effectively preventing medication errors. At the same time, the system performs steps S7 and S8 to upload the deviation event log for updating the user's personalized deviation atlas, realizing closed-loop learning.

[0120] After the user replaces the correct felodipine extended-release tablets according to the alarm prompt, the system re-identifies and verifies successfully, performs step S6, and broadcasts through the loudspeaker 4 that the medicine is correct. At this time, to solve the user's medication doubts, the system automatically activates the personalized medication guidance function and asks the user if there are any questions. The user can ask questions through voice: what are the side effects of this medicine, and the system will call the cloud medical knowledge base to answer clearly in the form of voice.

[0121] In addition, the present application also integrates two other key services to constitute a complete medication safety guarantee. After successfully identifying the medicine, the system can automatically establish a medication reminder for the user, ensuring that the user takes the medicine on time and in the right amount through timed broadcasting and vibration. During medication, if the user feels any discomfort, the user can activate the emergency call function of the device at any time, and the system will automatically dial the preset emergency contact phone or connect to the medical service center, providing comprehensive technical support for the user's independent safe medication.

[0122] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A method for multi-dimensional identification of intelligent drug information, characterized in that, Includes the following steps: S1. Obtain the drug identifier and retrieve the corresponding standard visual feature descriptor based on the drug identifier, while obtaining the user health information and historical deviation events associated with the current user. S2. Based on the acquired drug identifier, user health information and historical deviation events, calculate the comprehensive risk index, and use the comprehensive risk index to generate an adaptive visual verification strategy. S3. Acquire real-time images of the drug entity through the camera (2) and extract real-time visual feature descriptors from the real-time images; S4. Based on the generated adaptive visual verification strategy, the retrieved standard visual feature descriptor is compared with the extracted real-time visual feature descriptor to calculate the comprehensive confidence score. S5. When the overall confidence score is lower than the preset confidence threshold, a deviation event is determined to have occurred, and a preset alarm is triggered via the speaker (4).

2. The intelligent drug information multidimensional recognition method according to claim 1, characterized in that, Step S1, which involves obtaining a drug identifier and retrieving the corresponding standard visual feature descriptor based on the drug identifier, while simultaneously obtaining user health information and historical deviation events associated with the current user, includes the following steps: The image of the drug packaging is captured by the camera (2), and the drug identifier is decoded from the image; Based on the decoded drug identifier, the standard visual feature descriptor is retrieved from the visual feature library of the cloud server. The standard visual feature descriptor defines the geometric features, color features, size features and surface imprint features of the drug. User health information is loaded from the user database on the cloud server. The user health information defines the user's age, allergy history, and diagnosed diseases. Historical deviation events are loaded from the user-personalized deviation map of the cloud server.

3. The intelligent drug information multidimensional recognition method according to claim 1, characterized in that, Step S2, which involves calculating the comprehensive risk index based on the acquired drug identifier, user health information, and historical deviation events, includes: Retrieve drug risk factors associated with the drug identifier and user health information from the global medical knowledge graph; Determine user risk factors based on the user's health information; Based on the aforementioned historical deviation events, historical deviation risk factors are determined; The comprehensive risk index is obtained by weighting and summing the drug risk factor, user risk factor, and historical deviation risk factor with preset weights.

4. The intelligent drug information multidimensional recognition method according to claim 1, characterized in that, In step S2, the step of generating an adaptive visual verification strategy using the comprehensive risk index includes: The adaptive visual verification strategy includes a set of features to be verified and a set of weights corresponding to each feature in the feature set. The adaptive visual verification strategy is generated based on a comprehensive risk index. The higher the comprehensive risk index, the more features the feature set to be verified contains, and the higher the weight of the key identification features used to distinguish easily confused drugs in the weight set.

5. The intelligent drug information multidimensional recognition method according to claim 1, characterized in that, In step S4, the step of comparing the retrieved standard visual feature descriptor with the extracted real-time visual feature descriptor according to the generated adaptive visual verification strategy to calculate the comprehensive confidence score includes: Based on the set of features to be verified and the corresponding set of weights specified in the adaptive visual verification strategy, the sub-item similarity between the standard visual feature descriptor and the real-time visual feature descriptor on each feature in the feature set is calculated, and all the sub-item similarities are weighted and summed to obtain the comprehensive confidence score.

6. The intelligent drug information multidimensional recognition method according to claim 1, characterized in that, In step S5, when the overall confidence score is lower than a preset confidence threshold, a deviation event is determined to have occurred, and a preset alarm is triggered via the speaker (4). An alarm voice is broadcast through the speaker (4), and the vibration motor (18) is driven to generate vibration; The standard visual feature descriptor, real-time visual feature descriptor, and adaptive visual verification strategy are then encapsulated into a structured deviation event log.

7. The intelligent drug information multidimensional recognition method according to claim 1, characterized in that, The method further includes: Upload the aforementioned deviation event logs to the cloud server; The cloud server uses the received structured deviation event logs as input to update the current user's personalized deviation profile.

8. The intelligent drug information multidimensional recognition method according to claim 7, characterized in that, The cloud server also performs aggregated analysis on structured deviation event logs received from multiple users. When a globally highly confusing drug pair is identified in the aggregated analysis, the globally highly confusing drug pair is used as the update content to update the global medical knowledge graph.

9. A smart drug information multidimensional recognition device, applied to the method described in any one of claims 1-8, characterized in that, The device includes: Shell (1); The main board body (11) is encapsulated in the housing (1). The main board body (11) integrates a main control chip (12), an audio codec chip (13), a positioning module (14), a communication module (15), a charging management chip (17), and a vibration motor (18). The camera (2) and the front-end integrated supplementary lighting device (3) are arranged adjacent to the camera (2), and the camera (2) is electrically connected to the main control chip (12); The microphone (5) and speaker (4) are electrically connected by an audio codec chip (13) and a main control chip (12); The silicone Braille button (6) and the Braille voice input button (7) are electrically connected to the main control chip (12) by a micro switch (19). Display screen (8), which is electrically connected to the main control chip (12); A lithium battery (16) is provided with power to the device by a charging management chip (17); Charging data interface (9) is electrically connected to charging management chip (17).

10. A multi-dimensional intelligent drug information recognition system, applied to the method described in any one of claims 1-8, characterized in that, The system includes: The information acquisition module is configured to acquire drug identifiers, user health information associated with the current user, and historical deviation events; The image processing module is configured to acquire real-time images of the drug entity and extract a real-time visual feature descriptor from the real-time image; The risk assessment module, configured on a cloud server, is used to calculate a comprehensive risk index based on the drug identifier, user health information, and historical deviation events. The strategy generation module, configured on the cloud server, is used to generate an adaptive visual verification strategy using the comprehensive risk index and to retrieve the standard visual feature descriptor corresponding to the drug identifier. The comparison decision module is configured in the intelligent recognition terminal to receive the adaptive visual verification strategy and the standard visual feature descriptor, and compare the standard visual feature descriptor with the real-time visual feature descriptor according to the adaptive visual verification strategy, thereby calculating a comprehensive confidence score. An alarm and log module is configured in the intelligent recognition terminal to trigger a preset alarm and generate a structured deviation event log to be uploaded to the cloud server when the overall confidence score is lower than a preset confidence threshold.

Images