Hospital intelligent consumable management cabinet system based on multi-modal visual large model

Abstract

The application relates to the technical field of consumable management, in particular to a hospital intelligent consumable management cabinet system based on a multimodal visual large model, which collects video data and weight change values between the opening and closing of the management cabinet, and screens out invalid video data through the duration of the video data, so as to improve the recognition accuracy in the application. In addition, the application extracts optical flow information in the video data, and uses the optical flow information to identify the motion trajectory of the object in the video, so as to decompose the entire data into image frames in the putting stage when the user's hand enters and image frames in the taking stage when the hand is drawn out. By identifying the image frames in the two or more stages, the put-in article category and the taken-out article category are obtained. Combined with the motion trajectory and the weight change value, the application can accurately identify the put-in article category quantity and the taken-out article category and quantity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of consumable management, and particularly to a hospital intelligent consumable management cabinet system based on a multimodal visual large model. BACKGROUND

[0002] At present, medical consumables are widely used in medical institutions, and their management is an important part of hospital material management. Traditional medical consumable management mainly relies on manual checking for operation. For example, medical staff or management personnel record the warehouse entry, warehouse exit, inventory quantity and use of consumables through paper registration books or electronic spreadsheets (such as Excel).

[0003] Although manual checking is still used in some primary or small medical institutions, with the rapid growth of medical consumable types and usage, and the improvement of medical quality and safety management requirements, this management method has gradually exposed many drawbacks. Therefore, image recognition technology is used in the prior art to identify warehouse consumables. However, for medical consumables, different medical consumable outer packages have high similarity, so simply using image recognition patterns is prone to type misidentification and quantity misidentification. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a hospital intelligent consumable management cabinet system based on a multimodal visual large model to solve the problems in the background art.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] The hospital intelligent consumable management cabinet system based on a multimodal visual large model of the present application comprises:

[0007] A data acquisition module, configured to start timing, acquire initial weights of a plurality of storage units, and acquire videos when a cabinet door of a consumable management cabinet is opened, and stop timing, stop video acquisition, and acquire final weights of the plurality of storage units when the cabinet door is closed, to obtain a time length of a consumable taking period, video data, and weight change values of the plurality of storage units; and video data of a consumable taking period with a time length greater than a preset time length threshold is taken as valid video data;

[0008] An image extraction module, configured to frame from the valid video data to obtain a plurality of key frames;

[0009] A division module, configured to extract optical flow information of the plurality of key frames, and construct object motion trajectories in the key frames using the optical flow information; and divide the plurality of key frames into put-in stage image frames and take-out stage image frames based on the object motion trajectories;

[0010] The category identification module is configured to identify the put-in stage image frame and the take-out stage image frame respectively to obtain a put-in article category, a preliminary quantity of the put-in article, a take-out article category and a preliminary quantity of the take-out article.

[0011] The quantity identification module is configured to perform quantity verification on the preliminary quantity of the put-in article and the preliminary quantity of the take-out article based on the weight change value of the plurality of storage units, the object motion trajectory in the key frame, the put-in article category and the take-out article category to obtain a put-in article quantity and a take-out article quantity.

[0012] The management module is configured to construct cabinet inventory change information based on the put-in article category, the put-in article quantity, the take-out article category and the take-out article quantity, and perform consumable management based on the cabinet inventory change information.

[0013] In an embodiment of the present application, a plurality of key frames are obtained by frame extraction from the effective video data, including:

[0014] The effective video data is subjected to interval frame extraction to obtain a plurality of initial frames;

[0015] The plurality of initial frames are subjected to grayscale conversion to obtain a plurality of grayscale image frames;

[0016] Adjacent grayscale image frames are subjected to difference operation to obtain a difference image, and the difference image is subjected to binaryzation processing to obtain a binaryzation image;

[0017] A motion region in the binaryzation image is extracted, and based on a proportion of the motion region, when the proportion of the motion region is greater than a set threshold, a latter image in the adjacent grayscale image frames is determined as a dynamic frame, and the dynamic frame is taken as a key frame.

[0018] In an embodiment of the present application, optical flow information of the plurality of key frames is extracted, and the object motion trajectory in the key frame is constructed by using the optical flow information, including:

[0019] Key feature points in the key frame are extracted, wherein the key feature points include hand joint feature points and corner points;

[0020] The optical flow field in the key frame is calculated based on the key feature points to obtain a motion vector of each key feature point , wherein, represents a displacement amount of the key feature point along the x-axis, represents a displacement amount of the key feature point along the y-axis.

[0021] Based on the motion vector The key feature points of each key frame are subjected to density clustering to obtain one or more key feature point clusters.

[0022] based on all motion vectors in the key feature point cluster calculating an average value to obtain an average vector ; and calculating average coordinates of the key feature points in the key feature point cluster wherein, represents an average displacement along the x-axis, represents an average displacement along the y-axis, represents an average coordinate of the x-axis, represents an average coordinate of the y-axis;

[0023] converting the average vector from a two-axis displacement representation into a direction angle and a one-way displacement representation, and combining the average coordinates obtaining a target vector wherein:

[0024]

[0025]

[0026] wherein, is a direction angle, is a displacement amount;

[0027] based on a plurality of key frame target vectors constructing a motion trajectory of one or more objects.

[0028] In an embodiment of the present application, based on a plurality of key frame target vectors constructing a motion trajectory of one or more objects, comprising:

[0029] obtaining a background image with the same perspective as the video data;

[0030] mapping the plurality of key frame target vectors into the background image to obtain a motion feature image;

[0031] in the motion feature image, taking a starting position of each target vector as an origin to define a matching range with a radius of wherein, , is an empirical proportion parameter;

[0032] based on the matching range, matching any two adjacent target vectors, wherein when an end of one of the target vectors falls within the matching range of the adjacent target vector, the two adjacent target vectors are determined to be matched;

[0033] Fitting the start points of all matched target vectors to obtain the motion trajectory of the object.

[0034] In an embodiment of the present application, the plurality of key frames are divided into the putting-in stage image frames and the taking-out stage image frames based on the motion trajectory of the object, including:

[0035] Calculating the distance between the start points of any two adjacent target vectors in the motion trajectory of the object , and based on the time interval between adjacent key frames Calculating the object motion speed of the plurality of trajectory segments , ;

[0036] Based on the direction angle of each key frame , the object motion speed , and the acquisition time point , constructing the stage feature vector of each key frame ;

[0037] Constructing a time axis and mapping the stage feature vectors of the plurality of key frames to the time axis;

[0038] Sliding the sliding window constructed in advance along the time axis, and at each sliding time, calculating the average value of the direction angle and the average value of the speed within the sliding window; when the average value of the direction angle falls within a preset first angle range and the average value of the speed is greater than or equal to a preset speed threshold, determining that the segment within the sliding window is a putting-in segment; when the average value of the direction angle falls within a preset second angle range and the average value of the speed is greater than or equal to a preset speed threshold, determining that the segment within the sliding window is a taking-out segment; when the average value of the speed is less than a preset speed threshold, determining that the segment within the sliding window is an operation segment;

[0039] Merging the continuous putting-in segments and taking-out segments to obtain the time interval of the putting-in stage and the time interval of the taking-out stage, and extracting the putting-in stage image frames from the time interval of the putting-in stage and the taking-out stage image frames from the time interval of the taking-out stage.

[0040] In an embodiment of the present application, the putting-in stage image frames and the taking-out stage image frames are respectively identified to obtain the putting-in object category, the preliminary quantity of the putting-in object, the taking-out object category, and the preliminary quantity of the taking-out object, including:

[0041] Removing the non-motion background regions of the putting-in stage image frames and the taking-out stage image frames to obtain input images;

[0042] ​inputting the input image into a pre-constructed recognition model to obtain the put-in article category, the preliminary quantity of the put-in articles, the taken-out article category and the preliminary quantity of the taken-out articles.

[0043] In an embodiment of the present application, the method for constructing the recognition model comprises:

[0044] obtaining a sample image and performing extension processing on the sample image to obtain an extended sample, wherein the extension processing comprises image scaling, rotation, cropping, dark light enhancement, de-smearing and dynamic anti-reflection processing;

[0045] and labeling the sample image to obtain a training sample;

[0046] performing back propagation training on an artificial neural network based on the training sample to obtain a recognition model.

[0047] In an embodiment of the present application, the preliminary quantity of the put-in articles and the preliminary quantity of the taken-out articles are quantity-verified based on the weight change values of the plurality of storage units, the object motion trajectory in the key frame, the put-in article category and the taken-out article category to obtain the quantity of the put-in articles and the quantity of the taken-out articles, comprising:

[0048] obtaining a pre-constructed consumable weight data table, wherein the consumable weight data table comprises weight references of a plurality of consumables;

[0049] determining a target storage unit that causes a weight change based on the object motion trajectory in the key frame;

[0050] extracting the weight change value of the target storage unit from the weight change values of the plurality of storage units ;

[0051] obtaining a reference weight of the put-in article category or the taken-out article category from the consumable weight data table, and when the reference weight is less than a minimum weight threshold, taking the preliminary quantity of the put-in articles and the preliminary quantity of the taken-out articles as the quantity of the put-in articles and the quantity of the taken-out articles respectively; when the reference weight is greater than or equal to the minimum weight threshold, calculating a reference change value based on the preliminary quantity of the put-in articles and the reference weight of the put-in article category, or calculating a reference change value based on the preliminary quantity of the taken-out articles and the reference weight of the taken-out article category;

[0052] calculating a deviation rate of the weight change value from the reference change value , ;

[0053] when the deviation rate when the deviation rate is less than or equal to a preset deviation rate threshold, the preliminary quantity of the put-in articles and the preliminary quantity of the taken-out articles are taken as the put-in article quantity and the taken-out article quantity respectively; when the deviation rate is greater than the preset deviation rate threshold, the weight change value is calculated when the deviation rate is greater than the preset deviation rate threshold, the weight change value is calculated a ratio of the weight change value to a reference weight of the target article category wherein the target article category is the put-in article category or the taken-out article category

[0054] when a non-integer part of the ratio is less than a target value, the ratio is taken as the estimated quantity, and the estimated quantity and the effective video data are sent to the target object, and after receiving the verification information from the target object, the put-in article quantity and the taken-out article quantity are obtained; when the non-integer part of the ratio is greater than or equal to the target value, error information and the effective video data are sent to the target object, and after receiving the verification information from the target object, the put-in article quantity and the taken-out article quantity are obtained.

[0055] In an embodiment of the present application, further comprising:

[0056] The identity verification module is configured to, when receiving the identity information from the user, verify the identity information, and when the verification is passed, unlock the consumable management cabinet.

[0057] The present application has the following advantages: the hospital intelligent consumable management cabinet system based on the multi-modal visual large model can improve the recognition accuracy in the present application by collecting the video data and the weight change value of the management cabinet from the opening to the closing of the door, and screening out invalid video data by the duration of the video data. In addition, the present application can extract the optical flow information in the video data, and use the optical flow information to identify the motion trajectory of the object in the video, so as to divide the entire data into put-in stage image frames when the user's hand enters and taken-out stage image frames when the hand is withdrawn. By identifying the image frames of two or more stages, the put-in article category and the taken-out article category are obtained. Combined with the motion trajectory and the weight change value, the present application can accurately identify the put-in article category quantity and the taken-out article category and quantity. The present application has the advantages of accurate recognition and high information updating efficiency, and greatly improves the management efficiency of hospital consumables. BRIEF DESCRIPTION OF DRAWINGS

[0058] The present application will be further described below in conjunction with the drawings and embodiments:

[0059] Figure 1 is a use scene diagram of the hospital intelligent consumable management cabinet system based on the multi-modal visual large model shown in an embodiment of the present application;

[0060] Figure 2 is a practical hardware structure diagram of the hospital intelligent consumable management cabinet system based on a multimodal visual large model shown in an embodiment of the present application;

[0061] Figure 3 is a structural diagram of the hospital intelligent consumable management cabinet system based on a multimodal visual large model shown in an embodiment of the present application;

[0062] Figure 4 is a hand motion feature schematic diagram in an embodiment of the present application;

[0063] Figure 5 is a motion feature image schematic diagram in an embodiment of the present application;

[0064] Figure 6 is a flowchart of the hospital intelligent consumable management method based on a multimodal visual large model shown in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The present application will be described in more detail by the following specific examples. Other advantages and effects of the present application can be easily understood by those skilled in the art from this disclosure. The present application can also be implemented or applied in other different specific embodiments, and various modifications or changes can be made to the details in this specification based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0066] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only show the layers related to the present application in the diagrams, not the number of layers, shapes and sizes in actual implementation. The actual implementation of each layer may be a random change in pattern, number and proportion, and the layer layout pattern may also be more complex.

[0067] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application, however, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details.

[0068] Figure 1 is a practical hardware structure diagram of the hospital intelligent consumable management cabinet system based on a multimodal visual large model shown in an embodiment of the present application; Figure 1As shown, the management cabinet of the present application is divided into multiple storage units by multiple partitions, the bottom of each storage unit is provided with a gravity sensing module, and a photoelectric sensor is arranged at the joint of the cabinet door 110 and the cabinet body to sense the opening and closing state of the cabinet door. A camera 120 is arranged above the cabinet body and shoots downward. An RFID sensing module 130 is arranged in front of the cabinet body, and the cabinet and the cabinet door 110 are locked by an electromagnetic lock 140. When a user needs to open the cabinet door 110 to take out the required items, the identity information can be verified by swiping the RFID sensing module 130 with a hospital staff card. When the RFID sensing module 130 senses the hospital staff card, the system automatically reads the card information and verifies the identity of the staff. After the identity is verified, the related equipment will automatically unlock the cabinet door.

[0069] Figure 2 The actual hardware structure diagram of the hospital intelligent consumable management cabinet system based on a multi-modal visual large model shown in an embodiment of the present application is as shown in Figure 2 From the hardware structure of the system, the present application further includes a microcontroller and an analysis device. When the cabinet door 110 is opened, the photoelectric sensor transmits the door opening information to the microcontroller, and the microcontroller controls the camera 120 to start shooting. When the cabinet door 110 is closed, the photoelectric sensor transmits the door closing information to the microcontroller, and the microcontroller controls the camera 120 to stop shooting. The video data shot by the camera 130 is transmitted to the analysis device for screening, processing and analysis to obtain consumable change information. In addition, the electromagnetic door lock of the management cabinet is also controlled by the microcontroller. When the RFID sensing module senses the pre-registered radio frequency identification information, the microcontroller will control the electromagnetic door lock to be unlocked and opened through the related signal amplification circuit.

[0070] Figure 3 The structure diagram of the hospital intelligent consumable management cabinet system based on a multi-modal visual large model shown in an embodiment of the present application is as shown in Figure 3 As shown: the hospital intelligent consumable management cabinet system based on a multi-modal visual large model of the present embodiment includes:

[0071] The identity verification module 310 is used for verifying the identity information when receiving the identity information from the user, and unlocking the consumable management cabinet when the verification is passed.

[0072] Firstly, if the user has the access demand of the consumables, the identity information needs to be verified through the staff card, and after the verification is passed, the cabinet door is automatically unlocked.

[0073] The data acquisition module 320 is configured to start timing, acquire initial weights of the plurality of storage units, and acquire video when the cabinet door of the consumable management cabinet is opened, and stop timing, stop video acquisition, and acquire final weights of the plurality of storage units when the cabinet door of the consumable management cabinet is closed, to obtain a time length of a consumable taking period, video data, and weight change values of the plurality of storage units; and take the video data of the consumable taking period with a time length greater than a preset time length threshold as valid video data.

[0074] In the normal state of the management cabinet, the cabinet door is kept closed by a spring hinge. The photoelectric sensor includes a transmitter and a receiver, which are usually integrated in one device. The working principle is that the transmitter emits light (usually infrared), which is reflected back by the cabinet door and received by the receiver when the cabinet door is closed. Once the cabinet door is opened, the reflection path of the light is broken, and the receiver cannot receive enough reflected light, thereby triggering a state change of the sensor. Thus, the state change of the cabinet door can be automatically sensed.

[0075] When the state of the cabinet door is switched from closed to open, the microcontroller automatically controls the camera to start video recording, and simultaneously acquires the weight values of the plurality of storage units at the starting time point When the cabinet door is closed, the microcontroller automatically controls the camera to end video recording, and simultaneously acquires the weight values of the plurality of storage units at the ending time point At this time, the weight change value of the first storage unit .

[0076] In order to avoid unnecessary processing of video data generated when the cabinet door is mistakenly opened (when it is quickly opened and closed), the video with a duration less than 2 seconds is set as invalid (it is generally considered that the operation cannot be normally completed within 2 seconds). The video exceeding 2 seconds is valid video data. Only the valid video data is subjected to subsequent processing.

[0077] The image extraction module 330 is configured to extract frames from the valid video data to obtain a plurality of key frames.

[0078] In the present application, the frame rate of the valid video data is 60, that is, there are 60 frames of video data per second, therefore, key frames need to be extracted therefrom, so as to save subsequent analysis and calculation time, improve analysis efficiency, and the frame extraction process includes:

[0079] (1) interval frame extraction is performed on the valid video data to obtain a plurality of initial frames;

[0080] First, frames are extracted at a fixed interval (for example, one frame is extracted every 10 frames), which significantly reduces the total amount of video data, thereby reducing storage requirements and calculation load.

[0081] (2) Perform gray scale conversion on the plurality of initial frames to obtain a plurality of gray scale image frames;

[0082] Then, the color image is converted into a gray scale image (only the brightness information is reserved) to reduce the computational overhead of the color channel.

[0083] (3) Perform difference operation on adjacent gray scale image frames to obtain a difference image; and perform binarization processing on the difference image to obtain a binarized image;

[0084] By calculating the absolute difference value of adjacent gray scale frames (inter-frame difference method), the pixel change between frames is detected to locate the motion region. The difference image is converted into a binary image (black and white image) to highlight the difference between the foreground (motion region) and the background.

[0085] (4) Extract the motion region in the binarized image, and based on the proportion of the motion region, when the proportion of the motion region is greater than a set threshold, determine that the latter image in the adjacent gray scale image frames is a dynamic frame, and take the dynamic frame as a key frame.

[0086] By counting the pixel proportion of the motion region in the binarized image, it is determined whether the current frame contains significant motion. When the proportion of the motion region exceeds a set threshold, it is determined that the latter frame is a dynamic frame, which is kept as a key frame for subsequent analysis or storage.

[0087] The division module 340 is configured to extract optical flow information of a plurality of key frames, construct an object motion trajectory in the key frames by using the optical flow information, and divide the plurality of key frames into a put-in stage image frame and a take-out stage image frame based on the object motion trajectory.

[0088] After extracting the key frames obtained by shooting a dynamic object, the application analyzes the object motion direction and motion trajectory by using optical flow information, so as to divide the key frames into different motion stages, and thus perform object recognition on the image frames in different motion stages respectively. The process of extracting optical flow information of a plurality of key frames and constructing an object motion trajectory in the key frames by using the optical flow information includes the following steps:

[0089] (1) Extract key feature points in the key frames, wherein the key feature points include hand joint feature points and corner points.

[0090] By extracting hand joint feature points (such as fingers and wrists) and image corner points (such as edge intersection points), the region that may contain dynamic information in the video is focused. Only the key feature points are processed instead of all image pixels, which significantly reduces the subsequent computational complexity.

[0091] The corner point detection is realized by using a response function (such as Harris). The hand joint feature points can be extracted by using a model such as OpenPose.

[0092] (2) calculating an optical flow field in the key frame based on the key feature points, to obtain a motion vector of each key feature point wherein, represents a displacement amount of the key feature point along the x-axis, represents a displacement amount of the key feature point along the y-axis;

[0093] Based on sparse optical flow, the application calculates a displacement vector of each key feature point between adjacent frames by an optical flow algorithm (such as Lucas-Kanade), describes a motion trajectory thereof, and obtains an optical flow field. The optical flow vector directly reflects the spatial motion of the feature point, and provides a data basis for subsequent clustering and trajectory analysis.

[0094] (3) based on the motion vector performing density clustering on the key feature points of each key frame, to obtain one or more key feature point clusters;

[0095] If there are multiple moving objects (such as multiple people operating in cooperation) in the scene, clustering can group the feature points thereof, to avoid trajectory confusion. Since the optical flow field can come from one or more objects, in the application, different motion feature optical flow vectors are separated by density clustering, and similar motion feature optical flow vectors are gathered together, to form key feature point clusters. Each key feature point cluster represents a moving object.

[0096] (4) based on all the motion vectors in the key feature point cluster calculating an average value to obtain an average vector ; and calculating average coordinates of the key feature points in the key feature point cluster wherein, represents an average displacement amount along the x-axis, represents an average displacement amount along the y-axis, represents an average coordinate of the x-axis, represents an average coordinate of the y-axis;

[0097] Since one or more key feature point clusters are calculated in the foregoing, in order to facilitate analysis of the motion trajectory of an independent object, average value calculation is performed on all the motion vectors in each key feature point cluster, and average value calculation is performed on each key feature point, so that the average coordinates represent the position of the independent object, and the average vector represents the motion feature of the independent object.

[0098] (5) converting the average vector from a two-axis displacement representation form into a direction angle and one-way displacement representation form, and combining the average coordinates to obtain a target vector wherein:

[0099]

[0100]

[0101] In the formula, It is the direction angle. It is the displacement;

[0102] To facilitate the representation of motion of independent objects, this application converts the displacements along the x-axis and y-axis into the form of direction angles and displacements. Figure 4 This is a schematic diagram of hand movement features in one embodiment of this application. The converted hand movement features are as follows: Figure 4 As shown.

[0103] (6) Target vector based on multiple keyframes Constructing the motion trajectory of one or more objects, specifically including:

[0104] (6-1) Obtain a background image with the same viewpoint as the video data;

[0105] In this embodiment, a background image is obtained by directly using a camera and a fill light to capture the inside of the cabinet from the same angle.

[0106] (6-2) The target vectors of the multiple keyframes The motion feature image is obtained by mapping it onto the background image;

[0107] During the mapping process, As the starting point of the vector Indicates the direction of the vector. Indicates the length of the vector. Figure 5 This is a schematic diagram of a motion feature image in one embodiment of this application. The motion feature image obtained by mapping vectors is as follows: Figure 5 As shown.

[0108] (6-3) In the motion feature image, with each target vector starting position Define a radius of for the origin. The matching range, where, , This is an empirical proportional parameter;

[0109] For a single feature point, adjacent optical flow vectors can generally be directly concatenated. However, to represent the overall motion characteristics of an object, clustering and mean calculation have been performed as described above. Therefore, the resulting overall vector may not be able to be concatenated. To accurately represent the motion trajectory of a single object and avoid confusion between different objects, this application addresses this issue at each starting position. The matching range is demarcated, and the radius of the matching range is determined by the object motion speed. The faster the speed, the greater the possible deviation, and thus a greater matching range is needed, so the radius , The value of R can be 0.1.

[0110] (6-4) Matching any two adjacent target vectors based on the matching range, where when the end of one target vector falls within the matching range of the adjacent target vector, the two adjacent target vectors are determined to be matched;

[0111] (6-5) Fitting the start points of all matched target vectors to obtain the object motion trajectory.

[0112] After obtaining the object motion trajectory, the key frames can be divided according to the object motion trajectory, thereby obtaining the image frames of each motion stage. Identifying the image frames of each motion stage can intuitively obtain what the user put in and took out, and the specific process includes:

[0113] (1) Calculating the distance between the start points of any two adjacent target vectors in the object motion trajectory , and calculating the object motion speed of the multi-segment trajectory based on the time interval between adjacent key frames ,

[0114] (2) Constructing the stage feature vector of each key frame based on the direction angle of each key frame, the object motion speed , and the collection time point ;

[0115] The present application determines the stage based on the direction angle of each key frame and the object motion speed , and thus the stage feature vector includes the direction angle , the object motion speed , and the collection time point .

[0116] (3) Constructing a time axis and mapping the stage feature vectors of multiple key frames to the time axis;

[0117] ​​​(4) sliding along the time axis based on a pre-constructed sliding window, and at each sliding, calculating the average value of the direction angle and the average value of the speed within the sliding window; when the average value of the direction angle falls into a preset first angle range and the average value of the speed is greater than or equal to a preset speed threshold, determining that the section within the sliding window is a put-in section; when the average value of the direction angle falls into a preset second angle range and the average value of the speed is greater than or equal to the preset speed threshold, determining that the section within the sliding window is a take-out section; when the average value of the speed is less than the preset speed threshold, determining that the section within the sliding window is an operation section;

[0118] In the above process, the overall direction and overall speed of the object (such as the hand and the consumable) in a period of time (such as 0.5 seconds) are collected through the sliding window. If the average value of the direction angle falls into a preset first angle range and the average value of the speed is greater than or equal to a preset speed threshold, it indicates that the object (such as the hand and the consumable) is moving towards the management cabinet at this time, and the speed is fast, so it is determined that the hand of the user is in the put-in section at this time. On the contrary, if the average value of the direction angle falls into a preset second angle range and the average value of the speed is greater than or equal to a preset speed threshold, it indicates that the object is moving away from the management cabinet at this time, and the speed is fast, so it is determined that the hand of the user is in the take-out section at this time.

[0119] If the movement speed is small, it is determined that the user is currently arranging, homing, etc. in front of the cabinet. When performing these operations, the recognized consumables are meaningless, so the image frames corresponding to this stage are filtered out.

[0120] (5) merging the continuous put-in section and take-out section to obtain the time interval of the put-in stage and the time interval of the take-out stage, and extracting the put-in stage image frames from the time interval of the put-in stage and the take-out stage image frames from the time interval of the take-out stage.

[0121] Finally, since the put-in section and the take-out section extracted by the sliding window are scattered, the hand of the user is determined to be in the put-in section at this time. The stages with a duration less than 1S are removed, and the stages with a longer duration are taken as the final put-in stage or take-out stage, and the corresponding image frames are extracted from the corresponding time interval.

[0122] The category identification module 350 is configured to identify the put-in stage image frames and the take-out stage image frames respectively to obtain the put-in article category, the preliminary quantity of the put-in article, the take-out article category, and the preliminary quantity of the take-out article.

[0123] After extracting the image frames of different action stages of the user, the image frames of different stages are recognized, so as to obtain what consumables the user operates at different action stages. The application is recognized based on AI technology, and the process is as follows:

[0124] (1) removing the non-motion background area of the put-in stage image frame and the take-out stage image frame to obtain an input image;

[0125] In the foregoing, the motion part of the key frame is extracted by the frame difference method, so only the motion part is retained and reconstructed into an input image.

[0126] (2) inputting the input image into a pre-constructed recognition model to obtain the put-in article category, the preliminary quantity of the put-in article, the take-out article category and the preliminary quantity of the take-out article.

[0127] In the application, the recognition model preliminarily identifies the consumable category and quantity in the image, and the construction method of the recognition model includes:

[0128] Obtain a sample image and perform extension processing on the sample image to obtain an extended sample, wherein the extension processing includes image scaling, rotation, cropping, dark light enhancement, de-smearing and dynamic anti-reflection processing;

[0129] and labeling the sample image to obtain a training sample;

[0130] Based on the training sample, the artificial neural network is trained by back propagation to obtain a recognition model.

[0131] First, collect multi-angle article images covering different angles, light, and background scenes. Label the images (such as frame the object position, classification label), and use weak supervision learning technology to reduce the dependence on labeling and train a high-precision model with a small amount of labeled data.

[0132] Image processing: image scaling, rotation, cropping, etc. to enhance the generalization ability of the model. For low light, blur, and reflection, use dark light enhancement, de-smearing, and dynamic anti-reflection technology to improve image quality.

[0133] The core model mainly uses a convolutional neural network to automatically extract local features such as edges and textures and global features such as shapes and structures from images through multiple layers of convolution. The training process optimizes parameters through back propagation to enable the model to distinguish different objects. Dynamic multi-target recognition: compress the number of model layers to achieve fast identification of articles on the terminal.

[0134] The target detection model not only identifies the category but also locates the object position (outputs the bounding box). In a multi-target scene, the tracking algorithm SLAM technology is combined to maintain the stability of the object in consecutive frames and avoid shaking.

[0135] Basic models (such as ResNet and MobileNet) extract common features such as color and texture.

[0136] To address product differences (such as different specifications of consumable packaging), an attention mechanism is introduced to focus on local details (such as packaging text and logo).

[0137] Coarse classification (consumable categories) → fine classification (e.g., "gauze, scissors"). Accuracy is improved by combining cross-modal learning with multi-source information such as images.

[0138] The quantity recognition module 360 ​​is used to verify the initial quantity of the put-in items and the initial quantity of the taken-out items based on the weight change value of the multiple storage units, the motion trajectory of the object in the key frame, the type of put-in items, and the type of taken-out items, so as to obtain the quantity of put-in items and the quantity of taken-out items.

[0139] After obtaining the recognition results, voltage change values, and object motion trajectories within keyframes, logical reasoning can be performed based on these results to obtain change information, specifically including:

[0140] (1) Obtain a pre-built consumable weight data table, wherein the consumable weight data table includes weight references for various consumables, and an example of the consumable weight data table is shown in the table below.

[0141] Table 1. Example of Consumable Weight Data

[0142]

[0143] (2) Determine the target storage unit that produces the weight change based on the object's motion trajectory within the key frame;

[0144] When the end of the motion trajectory is within the coverage area of ​​a storage unit, that storage unit is used as the target storage unit.

[0145] (3) Extract the weight change value of the target storage unit from the weight change values ​​of the plurality of storage units. ;

[0146] weight change value The extraction process is as described above and will not be repeated here.

[0147] (4) Obtain the reference weight of the type of item put in or the type of item taken out from the consumable weight data table, and when the reference weight is less than the minimum weight threshold, take the initial quantity of the items put in and the initial quantity of the items taken out as the quantity of items put in and the quantity of items taken out, respectively; when the reference weight is greater than or equal to the minimum weight threshold, calculate the reference change value based on the initial quantity of the items put in and the reference weight of the type of items put in. , or, a reference change value is calculated based on the preliminary quantity of the taken-out article and the reference weight of the taken-out article category ;

[0148] For a light category of consumables, such as consumables with a weight less than 20 grams, the change of the total weight change value caused by the change is small, and if the quantity is large, the total weight change error can also easily cause the calculation quantity to deviate. Therefore, for consumables with a weight less than 20 grams, the visual recognition quantity result is taken as the final true result.

[0149] And for consumables with a weight greater than or equal to 20 grams, the gravity change value is used to correct the image recognition result, so that the result is more accurate.

[0150] The reference change value is the product of the article quantity and the article reference weight.

[0151] (5) Calculate the weight change value and the deviation rate of the reference change value , ;

[0152] (6) When the deviation rate is less than or equal to a preset deviation rate threshold, the preliminary quantity of the put-in article and the preliminary quantity of the taken-out article are taken as the put-in article quantity and the taken-out article quantity respectively; when the deviation rate is greater than the preset deviation rate threshold, the ratio of the weight change value to the reference weight of the target article category is calculated , wherein the target article category is the put-in article category or the taken-out article category;

[0153] If the deviation rate is small, it means that the visual recognition result is correct. If the deviation rate is large, there is a large probability that the visual recognition quantity has an error, so it can be considered that there may be an error in the quantity recognition result. In order to verify this possibility, the ratio of the weight change value to the reference weight of the target article category is calculated .

[0154] (7) When the non-integer part of the ratio is less than a target value, the ratio is taken as the estimated quantity, and the estimated quantity and the effective video data are sent to the target object. After receiving the verification information from the target object, the put-in article quantity and the taken-out article quantity are obtained; when the non-integer part of the ratio ​When the non-integer part of the ratio is greater than or equal to the target value, error information and the valid video data are sent to the target object, and after receiving the verification information from the target object, the put-in article quantity and the take-out article quantity are obtained.

[0155] If the ratio is an integer or the decimal part is small (less than 0.1), it is inferred that the quantity recognized by the vision may have a problem, but the recognized category does not have a problem. Therefore, the ratio is taken as the inferred quantity, and the inferred quantity and the valid video data are sent to the administrator for verification.

[0156] If the decimal part is large, it is possible that the recognized category also has a problem, in which case an error is reported and the valid video data is sent to the administrator for verification.

[0157] The management module 370 is configured to construct cabinet inventory change information based on the put-in article category, the put-in article quantity, the take-out article category and the take-out article quantity, and to perform consumable management based on the cabinet inventory change information.

[0158] Finally, the identity information of the user, the put-in article category, the put-in article quantity, the take-out article category and the take-out article quantity are obtained to construct the cabinet inventory change information, and the inventory change information is added to the ledger, and the inventory of the corresponding article is automatically updated.

[0159] The hospital intelligent consumable management cabinet system based on the multi-modal vision large model provided by the application improves the recognition accuracy in the application by collecting the video data and the weight change value of the management cabinet from opening to closing, and screening out invalid video data by the duration of the video data. In addition, the application extracts the optical flow information in the video data and uses the optical flow information to identify the motion trajectory of the object in the video, so as to divide the entire data into put-in stage image frames when the user's hand enters and take-out stage image frames when the hand is withdrawn. By identifying the image frames of two or more stages, the put-in article category and the take-out article category are obtained. Combined with the motion trajectory and the weight change value, the application can accurately identify the put-in article category quantity and the take-out article category and quantity. The application has the advantages of accurate recognition and high information updating efficiency, and greatly improves the management efficiency of hospital consumables.

[0160] As shown in Figure 6 The application also provides a hospital intelligent consumable management method based on a multi-modal vision large model, which comprises:

[0161] S610, starting timing, collecting initial weights of the plurality of storage units and video acquisition when the cabinet door of the consumable management cabinet is opened, and stopping timing, stopping video acquisition and collecting final weights of the plurality of storage units when the cabinet door of the consumable management cabinet is closed, obtaining the time length, video data and weight change value of the plurality of storage units of a consumable taking period, and taking the video data of the consumable taking period with a time length greater than a preset time length threshold as valid video data;

[0162] S620, frame extraction is performed from the valid video data to obtain a plurality of key frames;

[0163] S630, light flow information of the plurality of key frames is extracted, and the object motion trajectory in the key frame is constructed by using the light flow information; and the plurality of key frames are divided into put-in stage image frames and take-out stage image frames based on the object motion trajectory;

[0164] S640, the put-in stage image frames and the take-out stage image frames are respectively identified to obtain a put-in article category, a preliminary quantity of the put-in article, a take-out article category and a preliminary quantity of the take-out article;

[0165] S650, the preliminary quantity of the put-in article and the preliminary quantity of the take-out article are quantity checked based on the weight change value of the plurality of storage units, the object motion trajectory in the key frame, the put-in article category and the take-out article category to obtain a put-in article quantity and a take-out article quantity;

[0166] S660, cabinet inventory change information is constructed based on the put-in article category, the put-in article quantity, the take-out article category and the take-out article quantity, and consumable management is performed based on the cabinet inventory change information.

[0167] The hospital intelligent consumable management method based on the multi-modal visual large model of the application can improve the recognition accuracy in the application by collecting the video data and the weight change value between the opening and closing of the management cabinet, and screening out invalid video data by the duration of the video data. In addition, the application can extract the light flow information in the video data, and use the light flow information to identify the motion trajectory of the object in the video, so as to divide the entire data into the put-in stage image frames when the user's hand enters and the take-out stage image frames when the hand is taken out. By identifying the two or more stage image frames, the put-in article category and the take-out article category are obtained. Combined with the motion trajectory and the weight change value, the application can accurately identify the put-in article category quantity and the take-out article category and quantity. The application has the advantages of accurate recognition and high information updating efficiency, and greatly improves the management efficiency of the hospital consumables.

[0168] The embodiment also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any of the methods in the embodiment.

[0169] The embodiment also provides an electronic terminal, which comprises a processor and a memory.

[0170] The memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, so that the terminal executes any of the methods in the embodiment.

[0171] The computer readable storage medium in the embodiment can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by the hardware of the computer program. The computer program mentioned above can be stored in a computer readable storage medium. The program is executed to perform the steps of the above-mentioned method embodiments; and the storage medium mentioned above includes ROM, RAM, magnetic disk or optical disk and various storage medium that can store program codes.

[0172] The electronic terminal provided in the embodiment comprises a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected with the processor and the transceiver and complete communication between each other. The memory is used for storing a computer program, the communication interface is used for communication, and the processor and the transceiver are used for running the computer program, so that the electronic terminal executes each step of the method.

[0173] In the embodiment, the memory can include random access memory (RAM) and can also include non-volatile memory, for example, at least one disk memory.

[0174] The processor mentioned above can be a general processor, including a central processing unit (CPU), a network processor (NP) and the like; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0175] While the application has been described in terms of particular embodiments with reference to specific apparatus, it is to be understood that the disclosure is not limited to the specifics of the embodiments. Rather, it is meant to cover various alternatives, modifications, and equivalents, as would be apparent to one ordinarily skilled in the art having the benefit of this disclosure. Embodiments of the application are intended to embrace all such alternatives, modifications, and variations that fall within the scope of the appended claims.

[0176] The above-described embodiments are merely illustrative for the principles of the application and the efficacy thereof, and are not intended to limit the application. Any modification or change made by any person skilled in the art without departing from the spirit and scope of the application shall be covered by the appended claims.

Claims

1. A hospital intelligent consumable management cabinet system based on a multimodal visual large model, characterized in that, The method comprises the following steps: a data collection module, for starting timing, collecting initial weights of a plurality of storage units, and collecting video when the cabinet door of the consumable management cabinet is opened, and stopping timing, stopping video collection, and collecting final weights of a plurality of storage units when the cabinet door of the consumable management cabinet is closed, obtaining the time length of a consumable taking period, video data, and weight change values of a plurality of storage units; and taking the video data of the consumable taking period with a time length greater than a preset time length threshold as valid video data; an image extraction module, for extracting a plurality of key frames from the valid video data; a division module, for extracting optical flow information of the plurality of key frames, and constructing object motion trajectories in the key frames by using the optical flow information; and dividing the plurality of key frames into put-in stage image frames and take-out stage image frames based on the object motion trajectories; a category identification module, for identifying the put-in stage image frames and the take-out stage image frames respectively, obtaining put-in article categories, a preliminary quantity of put-in articles, take-out article categories, and a preliminary quantity of take-out articles; a quantity identification module, for quantity checking the preliminary quantity of put-in articles and the preliminary quantity of take-out articles based on the weight change values of the plurality of storage units, object motion trajectories in key frames, put-in article categories, and take-out article categories, obtaining put-in article quantities and take-out article quantities; a management module, for constructing cabinet inventory change information based on the put-in article categories, the put-in article quantities, the take-out article categories, and the take-out article quantities, and performing consumable management based on the cabinet inventory change information.

2. The hospital intelligent consumable management cabinet system based on a multimodal visual large model according to claim 1, characterized in that, extracting a plurality of key frames from the valid video data, comprising: interval frame extraction is performed on the valid video data to obtain a plurality of initial frames; gray scale conversion is performed on the plurality of initial frames to obtain a plurality of gray scale image frames; difference operation is performed on adjacent gray scale image frames to obtain difference images; and binaryzation processing is performed on the difference images to obtain binaryzation images; motion regions in the binaryzation images are extracted, and based on the proportion of the motion regions, when the proportion of the motion regions is greater than a set threshold, it is determined that a latter image in adjacent gray scale image frames is a dynamic frame, and the dynamic frame is taken as a key frame.

3. The multi-modal vision-based large model hospital intelligent consumable management cabinet system according to claim 1, wherein, extracting optical flow information of the plurality of key frames, and constructing object motion trajectories in key frames by using the optical flow information, comprising: extracting key feature points in the key frames, wherein the key feature points include hand joint feature points and corner points; Calculate an optical flow field in the key frame based on the key feature points, to obtain a motion vector of each key feature point wherein, denotes a displacement amount of the key feature point along the x-axis, denotes a displacement amount of the key feature point along the y-axis; Motion vector based Density clustering is performed on the key feature points of each key frame to obtain one or more key feature point clusters. based on all motion vectors in the cluster of key feature points calculating the average to obtain an average vector ; and calculating average coordinates of the key feature points in the cluster of key feature points wherein, denotes the average displacement along the x-axis, denotes the average displacement along the y-axis, denotes the average coordinate of the x-axis, denotes the average coordinate of the y-axis; converting the average vector from a two-axis displacement representation into a representation of a direction angle and a single direction displacement, and combining the average coordinates obtaining a target vector wherein: wherein is the directional angle, is the displacement amount; Target vector based on multiple keyframes Constructing a motion trajectory of one or more objects.

4. The hospital intelligent consumable management cabinet system based on a multimodal visual large model according to claim 3, characterized in that, Target vector based on multiple keyframes Constructing a motion trajectory of one or more objects, comprising: obtaining a background image with the same view angle as the video data; The target vector of the multiple key frames The motion feature image is obtained by mapping it onto the background image; In the motion feature image, with each target vector starting position Define a radius of for the origin. The matching range, where, , This is an empirical proportional parameter; matching any two adjacent target vectors based on the matching range, wherein when the end of one of the target vectors falls within the matching range of the adjacent target vector, it is determined that the two adjacent target vectors are matched; fitting the starting points of all matched target vectors to obtain the motion trajectory of the object.

5. The hospital intelligent consumable management cabinet system based on a multimodal visual large model according to claim 4, characterized in that, dividing the plurality of key frames into put-in stage image frames and take-out stage image frames based on the object motion trajectories, comprising: calculate a distance between start points of any two adjacent target vectors in the object motion trajectory , and based on a time interval between adjacent key frames calculate an object motion speed of the multi-segment trajectory , ; based on a direction angle of each key frame , object motion speed and acquisition time point constructing a stage feature vector to which each key frame belongs ; constructing a timeline and mapping stage characteristic vectors of a plurality of keyframes into the timeline constructing a timeline and mapping stage characteristic vectors of a plurality of keyframes into the timeline Sliding along the time axis based on a pre-constructed sliding window, and at each sliding, calculating the average of the direction angle and the average of the speed within the sliding window; when the average of the direction angle falls into a pre-set first angle range and the average of the speed is greater than or equal to a pre-set speed threshold, determining that the section within the sliding window is a putting-in section; when the average of the direction angle falls into a pre-set second angle range and the average of the speed is greater than or equal to a pre-set speed threshold, determining that the section within the sliding window is a taking-out section; when the average of the speed is less than a pre-set speed threshold, determining that the section within the sliding window is an operation section; Merging the continuous putting-in sections and taking-out sections to obtain a time interval of the putting-in stage and a time interval of the taking-out stage, and extracting putting-in stage image frames from the time interval of the putting-in stage and taking-out stage image frames from the time interval of the taking-out stage.

6. The multimodal vision-based large model hospital intelligent consumable management cabinet system according to claim 1, wherein, Respectively identifying the putting-in stage image frames and the taking-out stage image frames to obtain the putting-in article category, the preliminary quantity of the putting-in articles, the taking-out article category and the preliminary quantity of the taking-out articles, including: Removing the non-motion background area of the putting-in stage image frames and the taking-out stage image frames to obtain input images; Inputting the input images into a pre-constructed identification model to obtain the putting-in article category, the preliminary quantity of the putting-in articles, the taking-out article category and the preliminary quantity of the taking-out articles.

7. The multi-modal vision-based large model hospital intelligent consumable management cabinet system according to claim 6, wherein, The construction method of the identification model includes: Obtaining sample images and performing expansion processing on the sample images to obtain expanded samples, wherein the expansion processing includes image scaling, rotation, cropping, dark light enhancement, de-smearing and dynamic anti-reflection processing; And labeling the sample images to obtain training samples; Based on the training samples, the artificial neural network is trained by back propagation to obtain an identification model.

8. The multimodal vision-based large model hospital intelligent consumable management cabinet system according to claim 1, wherein, Based on the weight change values of the plurality of storage units, the object motion trajectories in the key frames, the putting-in article category and the taking-out article category, the preliminary quantity of the putting-in articles and the preliminary quantity of the taking-out articles are quantity checked to obtain the quantity of the putting-in articles and the quantity of the taking-out articles, including: Obtaining a pre-constructed consumable weight data table, wherein the consumable weight data table includes weight references of a plurality of consumables; Determining the target storage unit that causes the weight change based on the object motion trajectories in the key frames; extracting a weight change value of a target storage unit from weight change values of the plurality of storage units ; acquire a reference weight of the put-in article category or the take-out article category from the consumable weight data table, and when the reference weight is less than a minimum weight threshold, take the preliminary number of put-in articles and the preliminary number of take-out articles as the put-in article number and the take-out article number respectively; when the reference weight is greater than or equal to the minimum weight threshold, calculate a reference change value based on the preliminary number of put-in articles and the reference weight of the put-in article category , or calculate a reference change value based on the preliminary number of take-out articles and the reference weight of the take-out article category . calculating a weight change value deviation rate from a reference change value deviation rate from a reference change value , ; when the deviation rate is less than or equal to a preset deviation rate threshold value, the preliminary number of the put-in articles and the preliminary number of the taken-out articles are respectively taken as the number of the put-in articles and the number of the taken-out articles when the deviation rate is greater than the preset deviation rate threshold value, the weight change value is calculated, and a ratio of the weight change value to a reference weight of a target article category is calculated wherein the target article category is the put-in article category or the taken-out article category In the case where the non-integer part of the ratio is smaller than a target value, the ratio is taken as the estimated number, and the estimated number and the valid video data are sent to the target object, and after receiving the collation information from the target object, the number of put-in articles and the number of taken-out articles are obtained. In the case where the non-integer part of the ratio is greater than or equal to the target value, error information and the valid video data are sent to the target object, and after receiving the collation information from the target object, the number of put-in articles and the number of taken-out articles are obtained.

9. The multi-modal vision-based large model hospital intelligent consumable management cabinet system according to claim 1, wherein, Further comprising: An identity verification module for, when receiving identity information from a user, verifying the identity information, and when the verification is passed, unlocking the consumable management cabinet.

Images