Intraoperative high-value consumable non-sensing billing method and system based on multi-modal perception
By combining multimodal perception technology with visual, weight, and vital sign data, high-value consumable usage events are identified and automatically recorded, solving the accuracy problem of high-value consumable identification in the operating room environment and achieving the reliability and synchronization of high-value consumable consumption records.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- YANCHENG DAFENG PEOPLES HOSPITAL
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to accurately identify and record the use of high-value consumables in the complex environment of an operating room, leading to risks of misjudgment and omission, which affects the reliability of operating room management.
By employing a multimodal perception method that combines visual data, weight data, and vital sign monitoring data, high-value consumable usage events are identified through temporal localization and cross-modal analysis. Automatic accounting and self-calibration are then performed, and a reverse causal verification logic is constructed to filter out environmental interference.
It improves the accuracy and reliability of high-value consumables accounting, ensures that consumption records are synchronized with the surgical process, reduces the frequency and cost of manual intervention, and has the ability to self-optimize to adapt to equipment aging and environmental changes.
Smart Images
Figure CN122135905A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical information technology and relates to a non-intrusive billing method for high-value intraoperative consumables based on multimodal perception. Background Technology
[0002] In modern surgery, the use of high-value medical consumables is becoming increasingly widespread, accounting for a significant proportion of surgical costs. Intraoperative, unobtrusive accounting for high-value consumables refers to the automatic and accurate recording of every high-value consumable consumed during surgery without interfering with the normal surgical procedure; it is a key element in achieving intelligent management of the operating room.
[0003] Currently, the main technical means to achieve contactless billing for high-value consumables during surgery include radio frequency identification (RFID) technology and machine vision recognition alone. However, while RFID technology can achieve non-contact identification, it is easily interfered with by liquids and metal instruments in the operating room environment, and there are problems with multiple tag reading conflicts; while relying solely on visual recognition methods is easily affected by factors such as personnel obstruction, changes in lighting, and the high similarity of packaging for high-value consumables.
[0004] Therefore, in the dynamic and complex environment of the operating room with strong interference, the robustness and accuracy of existing technologies for identification are difficult to guarantee. Even if multiple physical senses such as vision and weight are simply combined, the data source is often affected by the same physical interference source, and there is a lack of an independent and objective verification dimension to confirm the final clinical use of high-value consumables. This results in a high risk of misjudgment and omission at key identification points, and the reliability of the accounting results is insufficient. Summary of the Invention
[0005] In view of this, in order to solve the problems mentioned in the background technology, a method and system for seamless billing of high-value consumables during surgery based on multimodal perception is proposed.
[0006] The objective of this invention can be achieved through the following technical solution: The first embodiment of this invention provides a method for seamless billing of high-value intraoperative consumables based on multimodal perception, comprising: S1. Simultaneously acquire visual data sequences and weight data sequences from the storage space of high-value consumables, and correspondingly obtain the patient's vital signs monitoring data sequences during the surgical process.
[0007] S2. Identify physiological response pattern characteristic signals that match the physiological response pattern of high-value consumables from the vital signs monitoring data sequence, and determine one or more key time points of high-value consumable use through time-series localization algorithm.
[0008] S3. Delineate a retrospective time window based on the key time points of high-value consumable usage. Within the retrospective time window, perform cross-modal feature extraction and logical correlation analysis on visual data sequences and weight data sequences to generate analysis results indicating the convergence of multi-source data. The analysis results include complete convergence or incomplete convergence.
[0009] S4. When the analysis result is fully converged, a high-value consumable usage event is determined to have occurred, and an automatic accounting operation is performed based on the high-value consumable identity information contained in the event, binding the consumption status of the high-value consumable with the corresponding surgical order in real time.
[0010] S5. Extract the standard weight change parameters from the high-value consumable usage event as the true value and feed them back to the force sensing unit to calibrate the force sensor used to collect the weight data sequence.
[0011] The second embodiment of the present invention provides a non-intrusive billing system for intraoperative high-value consumables based on multimodal perception, including: a multi-dimensional perception module, a physiological signal triggering module, a multi-source convergence verification module, an automatic billing operation module, and a nested self-calibration module.
[0012] The multi-dimensional perception module is connected to the physiological signal triggering module, the physiological signal triggering module is connected to the multi-source convergence verification module, the multi-source convergence verification module is connected to the automatic accounting operation module, and the automatic accounting operation module is connected to the nested self-calibration module.
[0013] The multi-dimensional perception module simultaneously collects visual data sequences and weight data sequences from the storage space of high-value consumables, and correspondingly obtains the patient's vital signs monitoring data sequences during the surgical process.
[0014] The physiological signal triggering module identifies physiological response pattern characteristic signals that match the physiological response pattern of high-value consumables from the vital signs monitoring data sequence, and determines one or more key time points of high-value consumable use through a time-series localization algorithm.
[0015] The multi-source convergence verification module defines a backtracking time window based on the key time points of high-value consumable usage. Within the backtracking time window, it performs cross-modal feature extraction and logical correlation analysis on visual data sequences and weight data sequences, generating analysis results that indicate the convergence of multi-source data. The analysis results include complete convergence or incomplete convergence.
[0016] The automatic accounting module determines a high-value consumable usage event when the analysis result is fully converged, and performs automatic accounting based on the high-value consumable identity information contained in the event, binding the consumption status of the high-value consumable with the corresponding surgical order in real time.
[0017] The nested self-calibration module extracts the standard weight change parameters from high-value consumable usage events as the true value and feeds them back to the force sensing unit to calibrate the force sensor used to collect weight data sequences.
[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention introduces patient vital signs data as an independent objective verification anchor point for high-value consumables usage events and constructs reverse causal verification logic, which can distinguish between real clinical usage behavior and accidental physical movement, effectively filter out strong interference factors such as physical obstruction and personnel movement in the complex environment of the operating room, and help improve the accuracy of high-value consumables accounting.
[0019] (2) This invention automatically collects, analyzes and determines the convergence of visual and weight data. After confirming the use of high-value consumables, it automatically performs identity information extraction, inventory deduction and surgical order binding operations. No manual intervention is required throughout the process, ensuring that the high-value consumable consumption record is strictly synchronized with the surgical process.
[0020] (3) The present invention constructs an intelligent self-calibration closed loop, which uses the high confidence accounting results generated by itself to perform dynamic and real-time calibration on the force sensor that is prone to drift; it has the ability to self-optimize, and can actively compensate for the perception deviation caused by equipment aging and environmental changes, ensuring long-term operational stability and data accuracy, and reducing the frequency and cost of manual maintenance. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a schematic diagram illustrating the implementation steps of the method of the present invention.
[0023] Figure 2 This is a schematic diagram of the module connection of the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] Please see Figure 1As shown, the first embodiment of the present invention provides a method for seamless billing of high-value intraoperative consumables based on multimodal perception, the specific steps of which are as follows: S1. Simultaneously acquire visual data sequences and weight data sequences from the storage space of high-value consumables, and correspondingly obtain the patient's vital signs monitoring data sequences during the surgical process.
[0026] To establish a unified, time-aligned foundation for multimodal data acquisition, the data process in step S1 is as follows: Image sensors deployed inside and outside the storage cabinet continuously capture dynamic images of the high-value consumables storage space. The dynamic images are decoded and frames are extracted by the edge computing gateway. Noise reduction and contrast enhancement are performed on each frame. A deep learning object detection algorithm is used to identify the outer contours of all high-value consumables targets in the frame and extract their bounding box coordinates, category confidence, and visual feature vectors. The recognition results of each frame are serialized according to the timestamp, forming a structured visual data sequence containing timestamps, a list of high-value consumables targets, and their respective visual features.
[0027] The load stress state is monitored by a force sensing matrix embedded in the bottom of the storage compartment. The force sensing matrix consists of multiple force sensor units embedded in the bottom of the storage compartment in a two-dimensional grid, with each unit corresponding to an independent storage compartment. By periodically collecting the output value of each force sensor unit, a Kalman filter-based algorithm is used to denoise and smooth the signals. Based on the physical layout of the compartments, the processed weight data of each unit is aggregated to calculate the total weight currently borne by the storage compartment, and the weight data sequence that changes continuously over time is recorded.
[0028] By subscribing to the physiological parameter streams sent by the intraoperative monitor in real time through the medical monitoring network, vital sign monitoring data sequences, including heart rate, arterial blood pressure, and blood oxygen saturation, can be obtained.
[0029] The visual data sequence, weight data sequence, and vital sign monitoring data sequence are synchronized and aligned using a global clock reference signal, which originates from a network time protocol server deployed in the operating room.
[0030] The synchronization alignment process involves synchronizing the local clocks of the image sensor, the signal processing unit of the force perception matrix, and the vital signs data receiving terminal with the time server via a network. When generating each data sequence, a timestamp in a uniform format is applied based on the synchronized clock. In subsequent processing, the frame events of the visual data sequence, the weight value changes in the weight data sequence, and the physiological parameter sampling points of the vital signs data sequence are time-aligned according to the timestamps.
[0031] S2. Identify physiological response pattern characteristic signals that match the physiological response pattern of high-value consumables from the vital signs monitoring data sequence, and determine one or more key time points of high-value consumable use through time-series localization algorithm.
[0032] In a preferred embodiment of the present invention, the key timing points for the use of high-value consumables are determined by the following method: a knowledge base for the use of high-value consumables and physiological responses is constructed, and the knowledge base stores standardized physiological response timing templates caused by different types of high-value consumables after implantation or access to the human body.
[0033] It should be noted that the construction of the high-value consumables usage-physiological response knowledge base is based on an offline process of retrospective analysis and feature learning of clinical medical data. Its purpose is to establish a mapping relationship between the use of specific high-value consumables and the resulting changes in typical physiological parameters. The specific construction steps are as follows: Extract surgical case data from the historical anesthesia monitoring record database and surgical material management system. Data on the use of high-value consumables, such as stent release and pacing electrode insertion, with clear records are extracted. The surgical case data includes raw time-series data of physiological parameters such as arterial blood pressure waveform, heart rate, and blood oxygen saturation, as well as verified logs of high-value consumable use events.
[0034] For each surgical case, using the time of use of its high-value consumables as the time origin, the physiological signals within the defined time window before and after the procedure are quantitatively analyzed, and key features are extracted to construct a multi-dimensional feature vector: A. Arterial blood pressure: The original arterial blood pressure waveform is low-pass filtered to remove high-frequency noise. The first derivative of the systolic blood pressure time series is calculated, and the maximum value of the first derivative is taken as the maximum upward slope. The delay time between the occurrence of the maximum upward slope and the use time of high-value consumables is recorded simultaneously.
[0035] B. Heart rate: Record the amplitude of heart rate changes before and after the use of high-value consumables and the duration of reaching the peak of the change. At the same time, perform fast Fourier transform or wavelet transform on the RR interval sequence to calculate the ratio of high-frequency power to low-frequency power as an indicator of changes in autonomic nerve tension.
[0036] C. Blood oxygen saturation: The blood oxygen saturation sequence is fitted to calculate the average slope.
[0037] It should be further explained that the specific analytical dimensions for arterial blood pressure, heart rate, and blood oxygen saturation mentioned above are merely illustrative examples, intended to illustrate the basic principle of extracting quantitative features from physiological signals to construct response patterns.
[0038] Those skilled in the art should understand that, based on the unique physiological mechanisms of action of different high-value consumables (such as electrical stimulation, mechanical dilation, drug release, etc.) and feedback from actual clinical applications, other effective analytical dimensions can be introduced for the same physiological parameter. For example, morphological analysis of blood pressure waveforms can be performed to extract waveform characteristic coefficients, or a more refined frequency domain division can be performed on heart rate variability. Other relevant physiological parameters, such as central venous pressure and end-tidal carbon dioxide partial pressure, can also be included for analysis.
[0039] For each surgical case, the aforementioned feature extraction process targeting multiple physiological parameters generates a multidimensional feature vector sequence, which is time-aligned to the usage time of the high-value consumables. By aggregating a large number of cases involving similar high-value consumables, existing statistical methods such as cluster analysis and principal component analysis are used to retrieve case groups with similar feature combinations. Each case group represents a potential typical physiological response pattern.
[0040] A summary textual description of the common changing trends within each case group is provided to generate a qualitative description template for the response pattern, such as a rapid rise in blood pressure accompanied by a slight increase in heart rate or a slow decrease in blood oxygen saturation accompanied by fluctuations in blood pressure.
[0041] The original physiological parameter sequences of all cases in the case group before and after the use of high-value consumables are re-aligned on the time axis. For each aligned time point, the mean and standard deviation of each physiological parameter value of all cases are calculated.
[0042] Therefore, for each physiological parameter, a typical reference sequence is generated with time as the horizontal axis and the parameter mean as the vertical axis, and the confidence interval of the physiological parameter value is defined by the mean ± N times the standard deviation, where N can be 1.96 for example. The typical reference sequence and the corresponding confidence interval together constitute a quantitative time series template of the response pattern.
[0043] The qualitative description template is associated with the quantitative timing template and stored together with the corresponding consumable identifier to form a standardized physiological response timing template.
[0044] Therefore, the high-value consumables usage-physiological response knowledge base is essentially a structured database derived from objective clinical big data statistical summarization. Its effectiveness is based on clear medical principles: many high-value consumables, when applied to the human body, produce immediate, regular, and detectable physiological responses through mechanical, electrophysiological, or pharmacological pathways. This provides an objective technical foundation for intraoperative event identification based on pre-established data-driven models.
[0045] Based on the current surgical type and the list of backup high-value consumables, at least one expected physiological response pattern is retrieved from the high-value consumables usage-physiological response knowledge base.
[0046] A sliding time window is set to analyze the real-time changing trends of various physiological parameters within the vital signs monitoring data sequence, and the matching degree is calculated with the expected physiological response pattern. The matching degree calculation process is as follows: Within a sliding time window, feature extraction operations are performed on each physiological parameter stream within the vital signs monitoring data sequence at the same time as the knowledge base construction time, generating real-time multidimensional feature vectors.
[0047] The algorithm analyzes whether the multidimensional feature vectors conform to the qualitative description template corresponding to the expected physiological response pattern. If they do not conform, the matching degree is directly assigned to 0. Otherwise, the extracted real-time feature vectors are compared one by one with the feature confidence intervals within the quantitative time-series template corresponding to the expected physiological response pattern preloaded in the knowledge base.
[0048] The proportion of values in each dimension of the real-time feature vector that fall into the corresponding feature confidence interval is calculated to obtain the final calculated matching degree.
[0049] When the matching degree is greater than the preset response threshold, the physiological response pattern characteristic signal is identified, and the starting jump time of the physiological response pattern is marked as the key time point for the use of high-value consumables.
[0050] S3. Delineate a retrospective time window based on the key time points of high-value consumable usage. Within the retrospective time window, perform cross-modal feature extraction and logical correlation analysis on visual data sequences and weight data sequences to generate analysis results indicating the convergence of multi-source data. The analysis results include complete convergence or incomplete convergence.
[0051] In a preferred embodiment of the present invention, generating analysis results indicating the convergence of multi-source data includes: within a backtracking time window, extracting visual recognition events of high-value consumable removal and corresponding first consumable identifiers from the visual data sequence, extracting weight step-down events and corresponding weight change values from the weight data sequence, and retrieving second consumable identifiers corresponding to the weight change values according to a preset high-value consumable attribute table.
[0052] The visual recognition event and the weight step-down event are time-stamped and aligned. If the visual recognition event and the weight step-down event are temporally related, and the first consumable identifier and the second consumable identifier point to the same high-value consumable code, an analysis result indicating complete convergence of multi-source data is generated; otherwise, an analysis result indicating incomplete convergence of multi-source data is generated.
[0053] Specifically, extracting the visual recognition events and corresponding first consumable identifiers of high-value consumables from the visual data sequence involves: performing frame-by-frame analysis of the visual data sequence to identify and locate multiple high-value consumable targets existing in the current high-value consumable storage space.
[0054] Track the position and orientation of each high-value consumable target in the image coordinate system to serve as the image features of each high-value consumable target.
[0055] When tracking of the image features of a high-value consumable target fails in consecutive frames, a high-value consumable retrieval candidate event is recorded.
[0056] The system continuously analyzes and records the visual data sequence within a preset time period after a candidate event for high-value consumable removal. If it is confirmed that the high-value consumable target cannot be tracked again, a visual recognition event of high-value consumable removal is determined to have occurred, and the item code is extracted from the image features of the high-value consumable target as the first consumable identifier.
[0057] Extracting weight step-down events and corresponding weight changes from a weight data series: The total weight data of the force perception matrix output during the static and inactive period of the storage partition is statistically analyzed. The average value and standard deviation of the total weight data during this period are calculated and used as the current steady-state reference value and noise standard deviation, respectively.
[0058] The weight data sequence is processed sequentially, and the difference between consecutive readings is calculated to obtain the instantaneous change. When the instantaneous change is continuously negative and its absolute value is greater than K1 times the noise standard deviation (K1 is a constant greater than 1), the time corresponding to the first reading that meets the condition is marked as the starting point of the descent.
[0059] After the starting point of the descent, data processing continues. When the absolute value of the instantaneous changes of a preset number of consecutive quantities is less than the noise standard deviation, the descent trend is determined to have stopped. Subsequently, the average value and standard deviation of the aggregated total weight data within the preset time period are calculated. If the standard deviation is less than the noise standard deviation at this time, the average value at this time is recorded as the new steady-state value of the interval, and its starting time is marked as the end point of the descent.
[0060] Calculate the average value of the previously confirmed steady-state interval preceding the descent start point as the old interval steady-state value. Calculate the difference between the old interval steady-state value and the new interval steady-state value. If the difference is greater than K2 times the noise standard deviation (K2 is a constant greater than K1), then a weight step descent event has been confirmed between the descent start point and the descent end point, and the difference is the corresponding weight change value.
[0061] The temporal correlation between visual recognition events and weight step descent events is determined by the following method: obtaining the moment when the high-value consumable leaves the storage location in the visual recognition event as the start time of high-value consumable retrieval, verifying whether the start time of high-value consumable retrieval is earlier than the key time point of high-value consumable use, and whether the time interval is within the preset reasonable physical path duration of the surgical operation.
[0062] It should be noted that the preset reasonable physical path duration is determined by reviewing a large number of past surgical videos offline, manually or automatically marking the typical walking path from the consumable storage cabinet to the main operating table, and statistically analyzing the average time and standard deviation required for medical staff to complete the material retrieval path while holding the consumables. The sum of the average time and three times the standard deviation is taken as the preset reasonable physical path duration.
[0063] The moment when the weight begins to decay linearly during the weight step-down event is obtained as the weight descent start moment. The absolute time difference between the weight descent start moment and the high-value consumable retrieval start moment is verified to be less than the preset response delay tolerance of the force sensor. The preset response delay tolerance is determined by measuring the physical delay time from when the high-value consumable is picked up to when the weight data of the storage partition begins to fall. It is usually set to 200 milliseconds to 500 milliseconds to cover the mechanical response time of different sensors.
[0064] If both the start time of high-value consumable retrieval and the start time of weight decrease are verified, it is determined that there is a temporal correlation between the visual recognition event and the weight step-down event.
[0065] S4. When the analysis result is fully converged, a high-value consumable usage event is determined to have occurred, and an automatic accounting operation is performed based on the high-value consumable identity information contained in the event, binding the consumption status of the high-value consumable with the corresponding surgical order in real time.
[0066] In a preferred embodiment of the present invention, the automatic accounting operation is performed in the following manner: The specifications, batch number, and unit price information of the high-value consumables corresponding to the first consumable identifier are logically bound to the key time points of high-value consumable usage and encapsulated into a high-value consumable usage event data packet with a unique identifier.
[0067] Data packets are pushed to the hospital's supplies management terminal and surgical billing terminal via an asynchronous communication interface, triggering a real-time inventory deduction process and accounting process for the use of high-value consumables.
[0068] S5. Extract the standard weight change parameters from the high-value consumable usage event as the true value and feed them back to the force sensing unit to calibrate the force sensor used to collect the weight data sequence.
[0069] In a preferred embodiment of the present invention, calibrating a force sensor used to collect weight data sequences includes: retrieving the corresponding high-value consumable material file from high-value consumable usage events and extracting the theoretical weight value of the high-value consumable under standard conditions.
[0070] The measured weight change value generated by the force sensor at the start of the corresponding high-value consumable usage is extracted synchronously.
[0071] Based on the deviation between the theoretical weight value and the measured weight change value, the formula is used. Calculate the calibration compensation parameters. Among them, C_new is the new calibration parameter obtained from this calculation, namely the gain compensation coefficient. It is the currently stored calibration parameters from the last time. These are the measured weight changes obtained during this high-value consumable usage incident. This is the theoretical weight of the high-value consumables used in this experiment under standard conditions. This is a preset smoothing factor, typically ranging from 0.9 to 0.99, used to control the smoothness of the calibration process. The calibration compensation parameters are written into the signal processing unit of the force sensor to update the analytical benchmark for subsequent measurement data.
[0072] In a preferred embodiment of the present invention, calibrating the force sensor used to collect weight data sequences further includes: setting an intraoperative observation period, and statistically analyzing the triggering frequency of high-value consumable usage events within the intraoperative observation period, marking them as a first frequency.
[0073] The frequency at which the first and second consumable identifiers point to different high-value consumable codes within the statistical observation period is marked as the second frequency.
[0074] Calculate the ratio of the second frequency to the first frequency, and based on the trend of the ratio's change, dynamically adjust the update strategy for the calibration compensation parameters, specifically as follows: Calculate the moving average and slope of the ratio of the second frequency to the first frequency over multiple consecutive observation periods. When the moving average continuously exceeds a first preset threshold (which can be set to 5% for example), and its slope remains positive over consecutive statistical periods, it is determined that there is a decreasing trend in calibration reliability.
[0075] When a decreasing trend in calibration reliability is detected, and the baseline noise level of the force sensor output is found to exceed its nominal range, the calibration strategy is automatically switched.
[0076] The routine successive compensation based on single-event deviations is suspended, and a deep calibration process based on batch analysis of historical data is initiated. Specifically, a calibration dataset is constructed by extracting data from the event log of the most recent high-value consumable usage events, a predetermined number N (N≥20). Each event data includes: the theoretical weight value retrieved from the material file based on the consumable's identity information, and the corresponding measured weight change value extracted from the weight data sequence. The theoretical weight value and the measured weight change value are then paired together.
[0077] The relative deviation of each data pair in the calibration dataset is calculated, and outlier data points with relative deviations exceeding three standard deviations are removed for preprocessing. The least squares method is then used to perform linear regression fitting on all preprocessed data pairs to obtain the fitted equation. ,in This is the gain compensation coefficient. This is the zero-point offset compensation amount. This represents the calculated weight change after calibration.
[0078] Using the fitted results and The parameters are used to perform reverse verification of the measured values in the calibration dataset and calculate the verification error.
[0079] If the root mean square value of the verification error is lower than the preset accuracy threshold, the new gain compensation coefficient and zero-point offset compensation amount are written into the signal processing unit of the force sensor to replace the original parameters and complete the update of the response curve. The preset accuracy threshold is determined based on the weight accuracy required for accounting, and can be set, for example, to ±2% of the theoretical weight value.
[0080] In the subsequent observation period, the system will enter verification and monitoring mode to closely track the changes between the first and second frequencies in order to confirm the effectiveness of the calibration.
[0081] In addition, the present invention also includes an abnormal arbitration step: when the first consumable identifier and the second consumable identifier are inconsistent, but the time correlation verification passes, the abnormal arbitration logic is initiated.
[0082] Image segments before and after key time points in the visual data sequence are retrieved, and secondary feature extraction of the consumable packaging is performed using an enhanced fine-grained classification model.
[0083] It should be noted that the aforementioned enhanced fine-grained classification model refers to a fine-grained convolutional neural network model based on an attention mechanism, specifically designed to distinguish highly similar visual categories. Its architecture includes at least:
[0084] Backbone Feature Extraction Network: A convolutional neural network used to extract multi-level visual features from an input image, such as ResNet or VGGNet.
[0085] Attention Mechanism Module: This module processes the feature maps extracted by the backbone network, automatically learns and focuses on key local areas on the outer packaging of consumables that are distinctive, such as specific trademarks, model text or pattern textures.
[0086] Fine-grained feature fusion and classification layer: It fuses focused features from different regions or levels and maps them to a high-dimensional feature space through a fully connected layer, finally outputting the identity identifier corresponding to the specific consumable category.
[0087] The enhanced fine-grained classification model is derived from supervised training on a large amount of packaging image data labeled with accurate high-value consumable identification information. Its training objective is to minimize the difference between the model's predicted consumable identity and the actual identity label. The model architecture (such as attention mechanisms and feature fusion methods) and its training methods are all well-known techniques in the field of computer vision and will not be elaborated upon here.
[0088] If the secondary feature extraction result supports the first consumable identification, then visual recognition shall be used as the standard and the force sensor shall be marked as needing to be forced to zero-point calibration immediately. If the secondary feature extraction result is still ambiguous, then a verification request shall be sent to the manual management terminal and the convergence analysis logic shall be corrected according to the manual feedback.
[0089] Reference Figure 2 As shown, the second embodiment of the present invention provides a non-intrusive billing system for intraoperative high-value consumables based on multimodal perception, including: a multi-dimensional perception module, a physiological signal triggering module, a multi-source convergence verification module, an automatic billing operation module, and a nested self-calibration module.
[0090] The multi-dimensional perception module is connected to the physiological signal triggering module, the physiological signal triggering module is connected to the multi-source convergence verification module, the multi-source convergence verification module is connected to the automatic accounting operation module, and the automatic accounting operation module is connected to the nested self-calibration module.
[0091] The multi-dimensional perception module simultaneously collects visual data sequences and weight data sequences from the storage space of high-value consumables, and correspondingly obtains the patient's vital signs monitoring data sequences during the surgical process.
[0092] The physiological signal triggering module identifies physiological response pattern characteristic signals that match the physiological response pattern of high-value consumables from the vital signs monitoring data sequence, and determines one or more key time points of high-value consumable use through a time-series localization algorithm.
[0093] The multi-source convergence verification module defines a backtracking time window based on the key time points of high-value consumable usage. Within the backtracking time window, it performs cross-modal feature extraction and logical correlation analysis on visual data sequences and weight data sequences, generating analysis results that indicate the convergence of multi-source data. The analysis results include complete convergence or incomplete convergence.
[0094] The automatic accounting module determines a high-value consumable usage event when the analysis result is fully converged, and performs automatic accounting based on the high-value consumable identity information contained in the event, binding the consumption status of the high-value consumable with the corresponding surgical order in real time.
[0095] The nested self-calibration module extracts the standard weight change parameters from high-value consumable usage events as the true value and feeds them back to the force sensing unit to calibrate the force sensor used to collect weight data sequences.
[0096] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications and additions should fall within the protection scope of the present invention.
Claims
1. A method for seamless intraoperative high-value consumables billing based on multimodal perception, characterized in that: include: S1. Simultaneously acquire visual data sequences and weight data sequences from the storage space of high-value consumables, and correspondingly obtain the patient's vital signs monitoring data sequences during the surgical process; S2. Identify physiological response pattern characteristic signals that match the physiological response pattern of high-value consumables from the vital signs monitoring data sequence, and determine one or more key time points of high-value consumables use through time-series localization algorithm; S3. Delineate a retrospective time window based on the key time point of high-value consumables use, and perform cross-modal feature extraction and logical association analysis on visual data sequences and weight data sequences within the retrospective time window to generate analysis results indicating the convergence of multi-source data. The analysis results include complete convergence or incomplete convergence. S4. When the analysis result is fully converged, it is determined that a high-value consumable usage event has occurred, and an automatic accounting operation is performed based on the high-value consumable identity information contained in the event, and the consumption status of the high-value consumable is linked to the corresponding surgical order in real time. S5. Extract the standard weight change parameters from the high-value consumable usage event as the true value and feed them back to the force sensing unit to calibrate the force sensor used to collect the weight data sequence.
2. The method for seamless intraoperative high-value consumables billing based on multimodal perception according to claim 1, characterized in that, The simultaneous acquisition of visual and weight data sequences from the storage space of high-value consumables, and the corresponding acquisition of vital sign monitoring data sequences of the patient during the surgical process, includes: The image sensors deployed inside and outside the storage cabinet continuously collect dynamic images of the high-value consumable storage space, which are then processed by the edge computing gateway to form a structured visual data sequence. The load stress state is monitored by a force sensing matrix embedded in the bottom of the storage partition, and a continuous weight data sequence is formed by fusing multiple signals. By subscribing to the physiological parameter streams sent by the intraoperative monitor in real time through the medical monitoring network, vital sign monitoring data sequences including heart rate, arterial blood pressure, and blood oxygen saturation can be obtained; The visual data sequence, the weight data sequence, and the vital signs monitoring data sequence are synchronized and aligned using a global clock reference signal.
3. The method for seamless intraoperative high-value consumables billing based on multimodal perception according to claim 1, characterized in that, The critical timing for the use of the high-value consumables is determined in the following way: Construct a knowledge base for the use and physiological response of high-value consumables, which stores standardized physiological response time sequence templates caused by different types of high-value consumables after implantation or access to the human body; Based on the current surgical type and the list of spare high-value consumables, at least one expected physiological response mode is retrieved from the high-value consumables usage-physiological response knowledge base; A sliding time window is set to analyze the real-time changing trend of each physiological parameter flow within the vital signs monitoring data sequence, and the matching degree with the expected physiological response pattern is calculated. When the matching degree is greater than the preset response threshold, the physiological response pattern characteristic signal is identified, and the starting jump time of the physiological response pattern is marked as the key time point for the use of high-value consumables.
4. The method for seamless intraoperative high-value consumables billing based on multimodal perception according to claim 1, characterized in that, The analysis results that indicate the convergence of multi-source data include: Within the retrospective time window, visual recognition events of high-value consumables being retrieved and corresponding first consumable identifiers are extracted from the visual data sequence, and weight step-down events and corresponding weight change values are extracted from the weight data sequence. The second consumable identifier corresponding to the weight change value is retrieved according to the preset high-value consumable attribute table. The visual recognition event and the weight step-down event are time-stamped and aligned. If the visual recognition event and the weight step-down event are temporally related, and the first consumable identifier and the second consumable identifier point to the same high-value consumable code, an analysis result indicating complete convergence of multi-source data is generated; otherwise, an analysis result indicating incomplete convergence of multi-source data is generated.
5. The method for seamless intraoperative high-value consumables billing based on multimodal perception according to claim 4, characterized in that, The visual recognition event for extracting high-value consumables from the visual data sequence includes: Frame-by-frame analysis of visual data sequences is performed to identify and locate multiple high-value consumable targets within the current high-value consumable storage space; Track the position and orientation of each high-value consumable target in the image coordinate system to serve as the image features of each high-value consumable target; When tracking of the image features of a high-value consumable target fails in consecutive frames, a high-value consumable retrieval candidate event is recorded. The system continuously analyzes and records the visual data sequence within a preset time period after a candidate event of high-value consumable removal. If it is confirmed that the high-value consumable target cannot be tracked again, a visual recognition event of high-value consumable removal is determined to have occurred.
6. The method for seamless intraoperative high-value consumables billing based on multimodal perception according to claim 4, characterized in that, The temporal correlation between the visual recognition event and the weight ladder descent event is determined by the following method: The moment when a high-value consumable leaves its storage location during a visual recognition event is obtained as the start time for high-value consumable retrieval. The start time for high-value consumable retrieval is verified to be earlier than the critical time point for high-value consumable use, and the time interval is within the preset reasonable physical path duration of the surgical operation. Obtain the moment when the weight begins to decay linearly during the weight step-down event, and use it as the starting moment of weight descent. Verify whether the absolute time difference between the starting moment of weight descent and the starting moment of high-value consumable use is less than the preset response delay tolerance of the force sensor. If both the start time of high-value consumable retrieval and the start time of weight decrease are verified, it is determined that there is a temporal correlation between the visual recognition event and the weight step-down event.
7. The method for seamless intraoperative high-value consumables billing based on multimodal perception according to claim 4, characterized in that, The automatic accounting operation is performed in the following manner: Logically bind the specifications, batch and unit price information of the high-value consumables corresponding to the first consumable identifier with the key time points of high-value consumable use, and encapsulate them into a high-value consumable use event data packet with a unique identifier; Data packets are pushed to the hospital's supplies management terminal and surgical billing terminal via an asynchronous communication interface, triggering a real-time inventory deduction process and accounting process for the use of high-value consumables.
8. The method for seamless intraoperative high-value consumables billing based on multimodal perception according to claim 4, characterized in that, The calibration of the force sensor used to acquire weight data sequences includes: From the high-value consumable usage events, retrieve the corresponding high-value consumable material files and extract the theoretical weight value of the high-value consumables used under standard conditions; Simultaneously extract the measured weight change value generated by the force sensor at the start time of the corresponding high-value consumable use; Based on the deviation between the theoretical weight value and the measured weight change value, calibration compensation parameters are calculated and written into the signal processing unit of the force sensor to update the analytical benchmark for subsequent measurement data.
9. The method for seamless intraoperative high-value consumables billing based on multimodal perception according to claim 8, characterized in that, The calibration of the force sensor used to acquire weight data sequences also includes: Set an intraoperative observation period, and count the frequency of high-value consumable usage events within the intraoperative observation period, marking them as the first frequency; The frequency at which the first and second consumable identifiers point to different high-value consumable codes within the statistical observation period is marked as the second frequency; The ratio of the second frequency to the first frequency is calculated, and the update strategy for the calibration compensation parameters is dynamically adjusted based on the trend of the ratio change.
10. A non-contact billing system for high-value intraoperative consumables based on multimodal perception, characterized in that: The system for implementing the method as described in any one of claims 1 to 9 comprises: The multi-dimensional perception module simultaneously collects visual data sequences and weight data sequences from the storage space of high-value consumables, and correspondingly obtains the patient's vital signs monitoring data sequences during the surgical process. The physiological signal triggering module identifies physiological response pattern characteristic signals that match the physiological response pattern of high-value consumables from the vital signs monitoring data sequence, and determines one or more key time points of high-value consumable use through a time-series localization algorithm; The multi-source convergence verification module defines a backtracking time window based on the key time point of high-value consumables usage. Within the backtracking time window, it performs cross-modal feature extraction and logical correlation analysis on visual data sequences and weight data sequences, and generates analysis results indicating the convergence of multi-source data. The analysis results include complete convergence or incomplete convergence. The automatic accounting module determines that a high-value consumable usage event has occurred when the analysis result is fully converged, and performs automatic accounting based on the high-value consumable identity information contained in the event, binding the consumption status of the high-value consumable with the corresponding surgical order in real time. The nested self-calibration module extracts the standard weight change parameters from high-value consumable usage events as the true value and feeds them back to the force sensing unit to calibrate the force sensor used to collect weight data sequences.