A method and system for automatically extracting vital sign information based on medical devices

Abstract

The application relates to a medical equipment-based vital sign information automatic extraction method and system, belonging to the technical field of medical information processing. The extraction method comprises the following steps: receiving a medical equipment display image collected by a user and performing pretreatment; performing image feature extraction according to the pretreated medical equipment display image to obtain an image feature set; extracting equipment aggregation features from the image feature set; identifying the equipment type of the medical equipment based on the equipment aggregation features, and matching the vital sign recognition model corresponding to the equipment type; extracting a correlation feature set based on the vital sign recognition model; identifying the parameter value at the feature position associated with each vital sign parameter type in the image feature set to obtain a vital sign parameter set and perform structured processing, and outputting the structured vital sign information. The application can automatically identify and extract effective vital sign information in the diversity of various equipment images and the complexity of image shooting environments.

Description

Technical Field

[0001] This application relates to the field of medical information processing technology, and in particular to a method and system for automatically extracting vital signs information based on medical devices. Background Technology

[0002] In modern medical monitoring and treatment, various medical devices (such as patient monitors, ventilators, anesthesia machines, and blood analyzers) are widely used in hospital wards, emergency rooms, and ICUs to continuously monitor patients' vital signs. These devices provide medical staff with real-time monitoring of patients' health status by displaying key information such as heart rate, blood pressure, respiratory rate, and blood oxygen saturation. This data is of great significance for medical diagnosis, treatment decisions, and emergency response.

[0003] Traditionally, healthcare professionals have relied on manual observation and recording of vital signs data from these devices, transcribing it into electronic medical records or patient monitoring records. However, due to the heavy workload of healthcare workers, especially during high-intensity emergency or intensive care situations, manually recording vital signs data is not only inefficient but also makes it difficult to guarantee the comprehensiveness and accuracy of the data. The manual recording process also limits the real-time nature of vital signs data, making it difficult to meet the needs for continuous monitoring and timely analysis of patient vital signs data.

[0004] To address the efficiency issues in medical data entry, Optical Character Recognition (OCR) technology has been increasingly applied in the medical field in recent years to extract textual information from documents such as medical bills and medical records. However, existing OCR technologies are mostly suitable for document images with relatively fixed formats and structures. When recognizing images displayed on medical devices, OCR technology faces numerous technical challenges. Images from different medical devices vary significantly in display layout due to differences in brand and model, resulting in poor performance of existing OCR technologies when recognizing these images. Furthermore, image data on medical device displays is usually captured by handheld devices, making it susceptible to external factors such as shooting angle, distance, and lighting conditions, leading to problems such as image quality degradation and content misalignment, making it difficult for OCR algorithms to accurately extract the required numerical information.

[0005] Furthermore, medical device images often display numerical values ​​for multiple vital signs. Even if OCR technology can recognize the textual information of these values, it still struggles to understand their meaning within the device image. For example, "98" on one device image might represent blood oxygen saturation, while on another device it might represent heart rate. The actual semantic information of the values ​​requires specialized analysis based on the specific structure of the device image, a need that current OCR technology struggles to meet.

[0006] Therefore, how to automatically identify and extract effective vital signs information amidst the diversity of images from various devices and the complexity of image shooting environments has become an urgent technical problem to be solved. Summary of the Invention

[0007] To improve the efficiency and accuracy of medical information acquisition, this application provides a method and system for automatically extracting vital sign information based on medical devices.

[0008] Firstly, this application provides a method for automatically extracting vital sign information based on medical devices, employing the following technical solution:

[0009] An automatic extraction method for vital sign information based on medical devices, the extraction method comprising:

[0010] Receive and preprocess images of medical devices collected by the user;

[0011] Image feature set is obtained by extracting image features from the preprocessed medical device display image;

[0012] Based on the device identifier feature library, device aggregate features are extracted from the image feature set;

[0013] Based on the aggregated features of the devices, the device type of the medical device is identified, and the corresponding vital sign recognition model is matched with the device type;

[0014] Based on the vital sign recognition model, the vital sign parameter types corresponding to the device type and the feature positions associated with the vital sign parameter types are extracted to obtain an associated feature set;

[0015] Based on the associated feature set, the parameter values ​​at the feature locations associated with each vital sign parameter type are identified in the image feature set to obtain the vital sign parameter set;

[0016] The set of vital signs parameters is processed into a structured format to output structured vital signs information.

[0017] By adopting the above technical solution, a systematic and comprehensive scheme for extracting vital sign information from medical devices is formed, encompassing image preprocessing, image feature extraction, device identification, parameter association, numerical extraction, and finally, structured output. This technical solution can efficiently and accurately transform image information from various medical devices into structured data, making automated extraction of vital sign parameters possible, improving the efficiency and accuracy of medical information acquisition, and possessing broad applicability.

[0018] Optionally, the preprocessing step includes:

[0019] The image displayed by the medical device is processed into a grayscale image.

[0020] The grayscale image is denoised using Gaussian filtering.

[0021] Edge detection is performed on the denoised grayscale image;

[0022] Based on the edge detection results, the foreground boundary region of the grayscale image is determined and extracted to obtain the corresponding foreground image.

[0023] By adopting the above technical solution, medical device images gradually become clearer and more focused from complex raw data. The final generated foreground image can accurately highlight the content of the area and reduce background interference, which is beneficial for subsequent text recognition and parameter extraction, and improves the accuracy and efficiency of the image processing process.

[0024] Optionally, the step of identifying the device type of the medical device based on the device aggregation features and matching the vital sign recognition model corresponding to the device type includes:

[0025] Based on the aggregated feature matching library, the corresponding device type is determined according to the aggregated features of the device.

[0026] Determine whether a vital sign recognition model corresponding to the device type exists;

[0027] If yes, proceed to the next step; otherwise, send a feedback message indicating that the device type is not supported to the user terminal.

[0028] By adopting the above technical solution, the device type is identified by comparing aggregated device features with a matching library, and model adaptation is performed using a model library. If the device type is not supported, feedback is provided, thus forming a complete closed loop of identification and model matching. This technical solution ensures that the system can intelligently adapt to different device types, improving the accuracy and adaptability of identification, while providing timely feedback and effectively enhancing the user experience.

[0029] Optionally, the system also includes a process for constructing the vital sign recognition model, the specific steps of which include:

[0030] Acquire a sample image dataset and perform preprocessing; wherein, the sample image dataset includes sample image data from various medical devices;

[0031] Image features are extracted from the preprocessed sample image dataset to obtain the sample image feature set;

[0032] Based on the equipment model labeling information, the equipment types are clustered, and the equipment aggregate feature set is extracted from the feature set of the sample images;

[0033] Based on the device aggregation feature set and the corresponding device type, a device identification feature library is constructed;

[0034] Extract the corresponding device type's characteristic parameter layout model from the device identification feature library;

[0035] Match each feature location in the sample image feature set with its location in the vital sign parameter layout model to determine the type of vital sign parameter corresponding to each feature location;

[0036] Based on the correlation between the feature location and the vital sign parameter type, a parameter location association set is obtained;

[0037] Based on the parameter location association set and the sample image feature set, the pre-constructed neural network model is trained to obtain the trained vital sign recognition model.

[0038] By adopting the above technical solution, a vital sign recognition model with multi-device adaptability was constructed. This model can automatically extract vital sign parameters according to the device type, improving recognition accuracy and versatility, and providing reliable technical support for the diversified recognition needs of medical devices.

[0039] Optionally, the step of training a pre-built neural network model based on the parameter location association set and the sample image feature set to obtain a trained vital sign recognition model includes:

[0040] The sample image feature set is labeled based on the parameter location association set to construct a labeled dataset;

[0041] The labeled dataset is divided into a training set, a validation set, and a test set;

[0042] The training set is input into a pre-built convolutional neural network model for initial training, and the model parameters are optimized and validated based on the validation set and test set to obtain the trained vital sign recognition model.

[0043] By adopting the above technical solutions, a complete model training process is formed. The final generated vital sign recognition model can accurately identify the types and locations of vital sign parameters in medical device images, providing highly accurate and generalizable technical support for the automated extraction of vital sign parameters from multiple devices.

[0044] Optionally, the steps of inputting the training set into a pre-built convolutional neural network model for initial training, and performing validation and model parameter optimization based on the validation and test sets include:

[0045] The training set is input into a pre-built convolutional neural network model for iterative training. The model parameters are optimized and the loss function of the model is calculated until the loss function meets the preset conditions, thus obtaining the initially trained convolutional neural network model.

[0046] The convolutional neural network model after initial training is validated based on the validation set, the performance of the convolutional neural network model is evaluated, and the model parameters are adjusted according to the validation results.

[0047] The predictive ability of the adjusted convolutional neural network model is tested based on the test set to obtain the trained vital sign recognition model.

[0048] By adopting the above technical solution, the training process of the vital sign recognition model was gradually completed through iterative training on the training set, model validation and optimization on the validation set, and final performance evaluation on the test set. The final generated vital sign recognition model can accurately identify the types and locations of vital sign parameters in medical device images, and has good generalization ability and stability, providing reliable technical support for the automated identification and extraction of vital sign information from multiple devices.

[0049] Secondly, this application provides an automatic vital sign information extraction system based on medical devices, which adopts the following technical solution:

[0050] An automatic vital sign information extraction system based on medical devices, the extraction system comprising:

[0051] Image receiving module, used to receive images collected by the user and displayed on the medical device;

[0052] The preprocessing module is used to preprocess the images displayed by the medical device;

[0053] The image feature extraction module is used to extract image features based on the preprocessed medical device display image to obtain an image feature set;

[0054] The device aggregation feature extraction module is used to extract device aggregation features from the image feature set based on the device identification feature library;

[0055] The device type identification module is used to identify the device type of the medical device based on the device aggregation features.

[0056] The model matching module is used to match the vital sign recognition model corresponding to the device type;

[0057] The associated feature set generation module is used to extract the vital sign parameter type corresponding to the device type and the feature position associated with the vital sign parameter type based on the vital sign recognition model, so as to obtain the associated feature set;

[0058] The parameter value recognition module is used to identify the parameter value at the feature position associated with each vital sign parameter type in the image feature set based on the associated feature set, so as to obtain the vital sign parameter set;

[0059] The structured processing module is used to perform structured processing on the set of vital sign parameters and output structured vital sign information.

[0060] Optionally, the extraction system further includes:

[0061] A sample image acquisition module is used to acquire a sample image dataset; wherein, the sample image dataset includes sample image data from various medical devices;

[0062] The preprocessing module is also used to preprocess the sample image dataset;

[0063] The image feature extraction module is also used to extract image features from the preprocessed sample image dataset to obtain a sample image feature set;

[0064] The clustering module is used to cluster equipment types based on equipment model labeling information and extract the equipment aggregate feature set from the sample image feature set;

[0065] The identification feature library construction module is used to construct a device identification feature library based on the device aggregate feature set and the corresponding device type;

[0066] The layout model extraction module is used to extract the corresponding device type's characteristic parameter layout model from the device identification feature library;

[0067] The location matching module is used to match each feature location in the feature set of the sample image with the location in the vital sign parameter layout model to determine the type of vital sign parameter corresponding to each feature location;

[0068] The association set generation module is used to obtain a parameter location association set based on the association relationship between the feature location and the vital sign parameter type;

[0069] The vital sign recognition model training module is used to train a pre-constructed neural network model based on the parameter location association set and the sample image feature set to obtain a trained vital sign recognition model.

[0070] Thirdly, this application provides a computer device, which adopts the following technical solution:

[0071] A computer device includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to perform the steps of the method as described in the first aspect.

[0072] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0073] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as in any of the methods in the first aspect.

[0074] In summary, this application includes at least one of the following beneficial technical effects: by efficiently and accurately extracting and structuring vital sign parameter information from images displayed by medical devices, it not only effectively reduces the errors and workload of manual recording and improves the efficiency and real-time nature of vital sign data collection, but also enhances the accuracy and standardization of medical data, making the automated collection and application of data from various devices possible, and providing reliable data support and technical guarantee for scenarios such as clinical monitoring and telemedicine. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the first process of an automatic extraction method for vital signs information based on medical devices, according to one embodiment of this application.

[0076] Figure 2 This is a schematic diagram of the second process of an automatic extraction method for vital signs information based on medical devices, according to one embodiment of this application.

[0077] Figure 3 This is a schematic diagram of the third process of an automatic extraction method for vital signs information based on medical devices, according to one embodiment of this application.

[0078] Figure 4 This is a schematic diagram of the fourth process of an automatic extraction method for vital signs information based on medical devices, according to one embodiment of this application.

[0079] Figure 5 This is a schematic diagram of display images of different types of monitors according to one embodiment of this application.

[0080] Figure 6 This is a schematic diagram of vital sign parameters on a monitor image according to one embodiment of this application.

[0081] Figure 7 This is a schematic diagram of the fifth process of an automatic extraction method for vital signs information based on medical devices, according to one embodiment of this application.

[0082] Figure 8 This is a schematic diagram of the sixth process of an automatic extraction method for vital signs information based on medical devices, according to one embodiment of this application. Detailed Implementation

[0083] To make the purpose, technical solution, and advantages of this application clearer, the following description is provided in conjunction with the appendix. Figure 1-8 The present application will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the application.

[0084] This application discloses a method for automatically extracting vital sign information based on medical devices.

[0085] Reference Figure 1 An automatic extraction method for vital sign information based on medical devices, the extraction method includes:

[0086] Step S101: Receive the medical device display image collected by the user and perform preprocessing;

[0087] Medical equipment includes, but is not limited to, monitors, ventilators, anesthesia machines, blood analyzers, etc. Since user-collected images (such as images of monitors or ventilators taken with handheld devices) are affected by factors such as light, shooting angle, and distance, preprocessing steps are needed to eliminate these interferences and improve image quality to facilitate subsequent recognition.

[0088] For example, in a pre-processed image containing a monitor screen, only the data display area is retained, while background interference is eliminated. This makes the text and graphic information in the image stand out more, which helps the subsequent feature extraction process to be carried out on higher quality image data and improves recognition accuracy.

[0089] Step S102: Extract image features based on the preprocessed medical device display image to obtain an image feature set;

[0090] Specifically, visual features in the preprocessed image can be extracted using feature extraction algorithms, including text features, icon features, and their location information, to construct an image feature set containing each element. Among these, text features include the numerical form of vital sign parameters in the image, and icon features include information such as the device model or manufacturer identification of the monitor.

[0091] For example, optical character recognition (OCR) can be used to recognize text in an image and convert the displayed values ​​into readable text; shape feature extraction can also be used to extract shape features unique to the device, such as icons, dividing lines, and numerical frames, to retain information such as device model and manufacturer identification.

[0092] Understandably, by extracting textual and graphic features, a multi-dimensional feature set of the image is constructed, providing basic data for device type identification and parameter location.

[0093] Step S103: Extract device aggregate features from the image feature set based on the device identifier feature library;

[0094] Since different medical devices may have specific icons, layouts, or text styles, a pre-built device identification feature library is used to filter device aggregate features from the image feature set. These aggregate features describe different device types, laying the foundation for loading the corresponding vital sign recognition model in subsequent steps. Specifically, the device identification feature library can be pre-built based on specific identifier information such as manufacturer logos and device model text in the image.

[0095] Step S104: Based on the device aggregation features, identify the device type of the medical device and match the corresponding vital sign recognition model for the device type;

[0096] Specifically, by filtering out information such as device model or manufacturer identifier from the image feature set as specific aggregated features, and then using the aggregated features to match the device model already in the library, the device type can be confirmed.

[0097] It is understandable that different device types have different parameter layouts and labeling methods. By identifying the device type, a suitable vital sign recognition model can be selected to match the image content and extract the corresponding vital sign parameters.

[0098] The vital sign recognition model is pre-trained and includes recognition rules for vital sign parameters (such as heart rate and blood oxygen) at specific locations. By matching device types with the model, the system supports different device types, giving it strong device versatility and scalability.

[0099] Step S105: Based on the vital sign recognition model, extract the vital sign parameter types corresponding to the device type and the feature locations associated with the vital sign parameter types to obtain the associated feature set;

[0100] After identifying the device type, the location of vital sign parameters in the image is located using a vital sign recognition model to extract these parameters and their associated location information, which facilitates accurate value positioning and parameter extraction.

[0101] For example, by using the parameters defined in the model to associate positions, the numerical regions of specific parameters (such as heart rate in the upper left and blood oxygen in the lower right) in the image can be identified. By calibrating the values ​​and positions, the specific location and corresponding value of each vital sign parameter type can be located in the image feature set. For instance, in the vital sign recognition model corresponding to a certain device, the heart rate (HR) value is usually located below the "HR" label in the upper left corner of the image, so the value at that location is identified as the heart rate parameter.

[0102] Understandably, extracting the parameter-related feature locations ensures the accurate identification of various vital sign parameters in the image, which helps reduce errors and improve data accuracy.

[0103] Step S106: Based on the associated feature set, identify the parameter values ​​at the feature locations associated with each vital sign parameter type in the image feature set to obtain the vital sign parameter set;

[0104] Specifically, based on the location of associated features, numerical information of each parameter type is extracted from the feature information in the image feature set to form a set of vital sign parameters, thus realizing the complete integration of device screen display data.

[0105] For example, the set of vital signs parameters includes, but is not limited to, a set of values ​​for parameters such as heart rate, blood pressure, respiratory rate, and blood oxygen saturation. For example, it may include the following data: heart rate = 78 beats / min, blood pressure = 120 / 80 mmHg, blood oxygen = 98%, respiratory rate = 18 breaths / min, etc.

[0106] Step S107: The vital signs parameter set is processed into a structured form, and structured vital signs information is output.

[0107] The identified vital sign parameters are structured into a standardized data format so that they can be called and used by upper-level systems or applications.

[0108] Specifically, the parameter set can be converted into a standard format based on a predefined data structure (such as JSON or XML), with each parameter containing explicit key-value pairs, such as timestamp, parameter name and its corresponding value.

[0109] For example, the final output of structured vital sign information includes the following:

[0110] "time":"2024-08-27 16:09:16";

[0111] "SpO2":"97";

[0112] "pr":"91";

[0113] "temp":"37.2";

[0114] "sys":"144";

[0115] "dia":"87";

[0116] "resp":"37".

[0117] Understandably, the structured data format described above facilitates direct reading and analysis by the upper-level medical management system, thereby improving the standardization and efficiency of data flow.

[0118] The above embodiments form a systematic and comprehensive solution for extracting vital sign information from medical devices, encompassing image preprocessing, image feature extraction, device identification, parameter association, numerical extraction, and finally, structured output. This technical solution can efficiently and accurately convert image information from various medical devices into structured data, enabling automated extraction of vital sign parameters, improving the efficiency and accuracy of medical information acquisition, and possessing broad applicability.

[0119] Reference Figure 2 As one implementation of step S101, the preprocessing step includes:

[0120] Step S201: Perform grayscale processing on the image displayed by the medical device to obtain a grayscale image;

[0121] Specifically, the original medical device display image is converted from a color image to a grayscale image to reduce the number of channels and the amount of data, allowing the image to focus on brightness information. This step reduces processing complexity and effectively extracts the main features of the image by merging the RGB three channels of the color image into one channel.

[0122] Step S202: Denoise the grayscale image based on Gaussian filtering;

[0123] Gaussian filtering, a smoothing filtering method based on the Gaussian distribution, aims to remove random noise caused by shooting conditions from images while preserving edge information as much as possible. This step improves image smoothness by reducing noise interference, providing more stable input data for subsequent edge detection.

[0124] For example, Gaussian filtering uses a Gaussian kernel to perform a convolution operation on an image. Each element in the Gaussian kernel is generated according to the Gaussian distribution formula, with the central element typically having the highest weight and decreasing towards the edges. In a practical implementation, a suitable Gaussian kernel (such as 3x3 or 5x5 in size) can be set to smooth the grayscale image, thereby removing random noise.

[0125] Step S203: Perform edge detection on the denoised grayscale image;

[0126] Edge detection is used to identify regions with significant grayscale variations in an image, thereby accurately extracting important contour information from the denoised image. Edge detection typically finds image edges by identifying regions of pixel gradient changes. These edges are often the boundaries of a device display or the frames that display numerical values, helping to separate the foreground from the background. Taking the Canny edge detection algorithm as an example, it first calculates the gradient values ​​and directions of the image, and then uses a double-threshold detection method to filter out significant edge regions.

[0127] Step S204: Based on the edge detection results, determine and extract the foreground boundary region of the grayscale image to obtain the corresponding foreground image.

[0128] Specifically, based on the edge detection results, the system performs contour analysis on the image to determine the foreground boundary region. The foreground boundary region mainly contains the content of the medical device's display screen, while the background region may contain environmental information from when the image was taken handheld and is not part of the image to be identified. For example, by identifying the innermost closed bounding box as the foreground region, cropping the grayscale image to the size of this region, and retaining the content of the monitor or medical device's display screen as the foreground image, background interference can be effectively removed, ensuring that subsequent recognition is only for the device display area.

[0129] In the above embodiments, by gradually clarifying and focusing the medical device image from complex raw data, the final generated foreground image can accurately highlight the content of the area and reduce background interference, which is beneficial for subsequent text recognition and parameter extraction, and improves the accuracy and efficiency of the image processing process.

[0130] Reference Figure 3 As one implementation of step S104, the steps of identifying the device type of medical devices based on device aggregation features and matching the corresponding vital sign recognition model for the device type include:

[0131] Step S301: Based on the aggregated feature matching library, determine the corresponding device type according to the device aggregated features;

[0132] The device aggregation features include identification information for different device types (such as device model, manufacturer brand, etc.), which reflects the unique layout and parameter display format of a specific device. By comparing with the aggregation feature matching library, which stores predefined correspondences between features and device types, the system can determine the specific device type corresponding to the image by finding the entry that best matches the device feature.

[0133] Step S302: Determine if there is a vital sign recognition model corresponding to the device type; if yes, continue to step S105; if no, proceed to step S303.

[0134] Since different device types have different layouts for displaying vital signs parameters, after identifying the device type, the system checks whether there is a vital signs recognition model of that type in the system to ensure that the feature extraction process of the device image conforms to its specific layout and display characteristics.

[0135] Step S303: Send a feedback message indicating that the device type is not supported to the user terminal.

[0136] Specifically, if a vital sign recognition model for that device type already exists in the model library, the image of that device can continue to be processed; otherwise, the user is redirected to the feedback prompt step to avoid continuing to extract invalid parameters, thereby saving system resources and informing the user that the current device type cannot be recognized.

[0137] In the above implementation, device aggregation features are compared with a matching library to identify device types, and model adaptation is performed using a model library. If the device type is not supported, a prompt is provided, thus forming a complete closed loop of identification and model matching. This technical solution ensures that the system can intelligently adapt to different device types, improving the accuracy and adaptability of identification, while providing timely feedback and effectively enhancing the user experience.

[0138] Reference Figure 4 In a further embodiment of this application, a process for constructing a vital sign recognition model is also included, the specific steps of which include:

[0139] Step S401: Obtain the sample image dataset and perform preprocessing;

[0140] The sample image dataset includes sample image data from various medical devices.

[0141] Specifically, in order to build a model that can identify vital signs information from various medical devices, it is necessary to collect sample image data of various medical devices (such as monitors, ventilators, anesthesia machines, etc.) and different models of devices, and ensure that the sample data covers common types of medical devices and the various vital signs parameters they display (such as heart rate, blood pressure, blood oxygen, etc.).

[0142] Reference Figure 5 The images shown are from two different models of patient monitors. Due to the wide variety of brands and types of patient monitors, the displayed information varies greatly. For example, the layout of vital sign information differs between the images. As shown in Figures A and B, which are images from two different monitors, in Figure A, Spo2 is above Resp, and Resp is above the NIBP parameter, while in Figure B, NIBP is above Spo2, demonstrating a significant difference in information layout. Furthermore, the types of vital sign parameters also differ between patient monitors. For instance, real-time monitoring data typically includes four standard vital sign parameters: heart rate, blood pressure, respiratory rate, and blood oxygen saturation, but some devices also include parameters such as systolic blood pressure and diastolic blood pressure.

[0143] As one implementation of the preprocessing step, the preprocessing step includes grayscale processing, Gaussian filtering for noise reduction, and foreground image extraction. By unifying the format and quality of the sample data, subsequent feature extraction and training can be carried out on the same basis. The content of preprocessing the medical device display image can be referred to in steps S201-S204 above.

[0144] Step S402: Extract image features from the preprocessed sample image dataset to obtain the sample image feature set;

[0145] In this process, feature extraction is used to obtain text and graphic features of the device display area from the preprocessed sample images, forming a sample image feature set. This feature set contains information such as numerical values, text labels, and device identifiers from the sample images, providing a data foundation for subsequent recognition.

[0146] Step S403: Cluster the equipment types based on the equipment model labeling information and extract the equipment aggregate feature set from the sample image feature set;

[0147] The device aggregation feature set is used to distinguish different device types, ensuring that the features of different devices are classified and identified during model training. Cluster analysis identifies feature sets of similar devices within the sample dataset.

[0148] Specifically, based on the equipment model labeling information, image data of the same model are grouped into one category. Clustering algorithms are applied to classify the equipment identification features in the sample image feature set to form aggregated features. Common features of equipment types are extracted from each category to generate an aggregated feature set of equipment.

[0149] Step S404: Construct a device identification feature library based on the device aggregation feature set and the corresponding device type;

[0150] The device identification feature library is built based on the device aggregate feature set and is used to store feature information of different device types. This feature library provides data support for subsequent device identification and parameter matching.

[0151] Specifically, the identification information (such as brand, model, etc.) of each equipment type in the equipment aggregation feature set is stored in the equipment identification feature library. The equipment identification feature library records the aggregation feature information of different equipment types and their correspondence with the vital sign parameter layout model.

[0152] Step S405: Extract the corresponding device type's characteristic parameter layout model from the device identification feature library;

[0153] Among them, the vital sign parameter layout model reflects the specific layout and position of each vital sign parameter in different equipment types, ensuring that the type of vital sign parameter corresponding to each value can be correctly identified during image processing.

[0154] Specifically, by recording the layout positions of vital signs parameters (such as heart rate and blood pressure), the layout model of each type of equipment is labeled to clarify the relative positions of different parameters in the image, ensuring that the values ​​and labels in subsequent images are correctly classified into the corresponding vital signs parameter types.

[0155] Step S406: Match each feature location in the feature set of the sample image with its location in the vital sign parameter layout model to determine the type of vital sign parameter corresponding to each feature location;

[0156] In this process, by matching the feature locations in the sample image with the layout model, the type of vital sign parameter represented by each feature is identified, thus forming a parameter location association.

[0157] Reference Figure 6 The parameters in the example are information about a patient's vital signs displayed on a monitor image. From the perspective of image information, the parameters themselves are also part of the image features, but different parameters have different meanings related to vital signs. Therefore, during the recognition process, it is necessary to assign specific meanings to these discrete feature information through feature association relationships. This application employs a feature association algorithm to associate textual information of specific meanings in the image with other information that has significant features, thereby assigning that information a specific meaning. For example, based on the association relationship, it can be known that... Figure 6 The value of the respiratory rate (RESP) parameter is located in the lower right corner of this "RESP" information.

[0158] Step S407: Based on the correlation between feature location and vital sign parameter type, obtain the parameter location association set;

[0159] Specifically, in the sample image feature set, based on the location information of the vital sign parameter layout model, the features at specific locations are matched to the corresponding vital sign parameter types, and the matching results are recorded in the parameter location association set to ensure that each value in the image features has a clear vital sign parameter meaning.

[0160] Step S408: Based on the parameter location association set and the sample image feature set, the pre-constructed neural network model is trained to obtain the trained vital sign recognition model.

[0161] Specifically, the labeled image feature set is input into the neural network model. The model learns step by step to identify and classify vital signs parameters such as heart rate and blood oxygenation for the device type, and locates their positions in the image. After training, this vital sign recognition model can accurately identify vital sign parameters of different devices, providing strong support for the automatic identification and parameter extraction of multiple device types.

[0162] In the above embodiments, by constructing a vital sign recognition model with multi-device adaptability, the model can automatically extract vital sign parameters according to the device type, thereby improving the recognition accuracy and versatility and providing reliable technical support for the diversified recognition needs of medical devices.

[0163] Reference Figure 7As one implementation of step S408, the step of training a pre-built neural network model based on the parameter location association set and the sample image feature set to obtain the trained vital sign recognition model includes:

[0164] Step S501: Label the feature set of the sample image based on the parameter location association set to construct the labeled dataset;

[0165] Before model training begins, a labeled dataset containing images displayed by medical devices is prepared. The text location information in each image is bound to the corresponding vital sign parameter labels (such as heart rate and blood pressure), ensuring that each image contains the location information and labels of vital sign parameters, thus generating a labeling file. For example, the labeling file includes the image ID of each image, the value of the vital sign parameter, the parameter type (such as heart rate HR, blood oxygen saturation SpO2), and the parameter's location coordinates in the image.

[0166] Step S502: Divide the labeled dataset into a training set, a validation set, and a test set;

[0167] The training set is used for actual training of the model, the validation set is used for tuning the model parameters, and the test set is used to finally evaluate the model's generalization performance.

[0168] Step S503: Input the training set into the pre-built convolutional neural network model for initial training, and perform validation and model parameter optimization based on the validation set and test set to obtain the trained vital sign recognition model.

[0169] Specifically, the training set is input into a pre-built convolutional neural network (CNN) model. Through multiple rounds of training iterations, the model learns the types and location information of vital signs in the image. After each training round, the model's performance is tested using a validation set, and the model's hyperparameters are dynamically adjusted based on the accuracy and loss changes on the validation set. After the model has completed training, the test set is input into the model, and metrics such as classification accuracy and location error are calculated to evaluate the model's generalization ability.

[0170] In one embodiment of this application, the model structure of the pre-built convolutional neural network model mainly includes the following parts:

[0171] 1. Input layer: Used to receive the preprocessed grayscale image; the input size must be consistent with the size of the preprocessed image.

[0172] 2. Convolutional and Pooling Layers: Each convolutional layer uses multiple convolutional kernels to extract image features at different levels. Pooling layers are used for downsampling to reduce the number of parameters and preserve the spatial location information of the features.

[0173] 3. Feature Pyramid Structure: A Feature Pyramid Network (FPN) is added after the convolutional layer to extract features from feature maps of different resolutions at multiple scales, thereby adapting to the diversity of parameter layouts in images from different devices.

[0174] 4. Fully connected layers: Multiple fully connected layers are connected after the convolutional layers to transform the convolutional features into higher-level feature representations for final classification and regression.

[0175] 5. Output Layer: Set up the output layer, including a classification branch and a regression branch. The classification branch is used to identify the parameter type (such as heart rate, blood oxygen, etc.), and the regression branch is used to predict the location coordinates of each parameter in the image.

[0176] In the above embodiments, after a complete model training process, the final generated vital sign recognition model can accurately identify the types and locations of vital sign parameters in medical device images, providing highly accurate and generalizable technical support for the automated extraction of vital sign parameters from multiple devices.

[0177] Reference Figure 8 As one implementation of step S503, the steps of inputting the training set into a pre-built convolutional neural network model for initial training, and performing validation and model parameter optimization based on the validation set and test set include:

[0178] Step S601: Input the training set into the pre-built convolutional neural network model for iterative training, optimize the model parameters and calculate the model's loss function until the loss function meets the preset conditions, and obtain the initially trained convolutional neural network model.

[0179] The convolutional neural network model iteratively trains on the training set multiple times to gradually optimize the model parameters, thereby achieving the ability to learn features and identify vital parameters of the input image. Through repeated iterations, the error between the model's predicted output and the true labeled data is gradually reduced until the loss function meets the preset convergence condition.

[0180] Specifically, images from the training set are batch-wise input into the convolutional neural network model, and convolution and pooling operations are performed layer by layer to extract image features. The predicted vital sign parameter types and locations are then output through fully connected layers. The difference between the model's predictions and the true labels is calculated using a loss function. For classification tasks (vital sign parameter type recognition), the cross-entropy loss function is used; for regression tasks (parameter location prediction), the smoothed L1 loss function is used. The gradient of the loss with respect to the model parameters is calculated using the backpropagation algorithm, and the model parameters are updated using an optimization algorithm (such as the Adam optimizer) to gradually reduce the loss value. These steps are repeated multiple times until the loss function value reaches the preset convergence condition or the maximum set number of training epochs is reached, resulting in a preliminarily trained convolutional neural network model.

[0181] Step S602: Validate the initially trained convolutional neural network model based on the validation set, evaluate the performance of the convolutional neural network model, and adjust the model parameters according to the validation results;

[0182] Specifically, the validation set is input into the initially trained convolutional neural network model, and the model's loss and accuracy on the validation set are calculated. The performance on the validation set is used to measure the model's predictive ability on unseen data. If the validation set loss is high or the accuracy is unsatisfactory, it indicates that the model may be overfitting or underfitting. In this case, the model's hyperparameters (such as learning rate, number of convolutional kernels, batch size, etc.) or regularization methods (such as Dropout layers) can be adjusted to further optimize the model's performance. After each adjustment, the model is retrained, and the performance on the validation set is evaluated again until the model achieves the expected accuracy and generalization ability on the validation set.

[0183] Step S603: Test the prediction ability of the adjusted convolutional neural network model based on the test set to obtain the trained vital sign recognition model.

[0184] Specifically, the final adjusted convolutional neural network model is applied to the test set, and the classification accuracy, location prediction error, and overall loss value on the test set are calculated. The overall performance of the model is evaluated based on the test set results to ensure that it meets the accuracy requirements for vital sign recognition and the error range standard for location prediction.

[0185] In the above implementation, the training process of the vital sign recognition model is gradually completed through iterative training on the training set, model verification and optimization on the validation set, and final performance evaluation on the test set. The final generated vital sign recognition model can accurately identify the types and locations of vital sign parameters in medical device images, and has good generalization ability and stability, providing reliable technical support for the automated identification and extraction of vital sign information from multiple devices.

[0186] This application also discloses an automatic extraction system for vital signs information based on medical devices.

[0187] An automatic vital sign information extraction system based on medical devices, the extraction system includes:

[0188] Image receiving module, used to receive images collected by the user and displayed on the medical device;

[0189] The preprocessing module is used to preprocess the images displayed by the medical device;

[0190] The image feature extraction module is used to extract image features from the preprocessed medical device display image to obtain an image feature set;

[0191] The device aggregation feature extraction module is used to extract device aggregation features from the image feature set based on the device identification feature library;

[0192] The device type identification module is used to identify the device type of medical devices based on device aggregation features;

[0193] The model matching module is used to match the vital sign recognition model corresponding to the device type;

[0194] The associated feature set generation module is used to extract the vital sign parameter types corresponding to the device type and the feature positions associated with the vital sign parameter types based on the vital sign recognition model, and obtain the associated feature set.

[0195] The parameter value recognition module is used to identify the parameter value at the feature location associated with each vital sign parameter type in the image feature set based on the associated feature set, so as to obtain the vital sign parameter set;

[0196] The structured processing module is used to process the set of vital sign parameters into a structured form and output structured vital sign information.

[0197] In the above embodiments, by efficiently and accurately extracting and structuring vital sign parameter information from images displayed by medical devices, not only are errors and workload of manual recording effectively reduced, and the efficiency and real-time performance of vital sign data collection improved, but the accuracy and standardization of medical data are also enhanced, making automated collection and application of data from various devices possible, and providing reliable data support and technical assurance for scenarios such as clinical monitoring and telemedicine.

[0198] As a further implementation of the extraction system, the extraction system also includes:

[0199] The sample image acquisition module is used to acquire sample image datasets; the sample image datasets include sample image data from various medical devices.

[0200] The preprocessing module is also used to preprocess the sample image dataset;

[0201] The image feature extraction module is also used to extract image features from the preprocessed sample image dataset to obtain a sample image feature set;

[0202] The clustering module is used to cluster equipment types based on equipment model labeling information and extract the equipment aggregate feature set from the sample image feature set;

[0203] The device identifier feature library construction module is used to construct a device identifier feature library based on the device aggregate feature set and the corresponding device type;

[0204] The layout model extraction module is used to extract the characteristic parameters layout model of the corresponding equipment type from the equipment identification feature library;

[0205] The location matching module is used to match each feature location in the feature set of the sample image with the location in the vital sign parameter layout model to determine the type of vital sign parameter corresponding to each feature location;

[0206] The association set generation module is used to obtain the parameter location association set based on the association relationship between feature location and vital sign parameter type;

[0207] The vital sign recognition model training module is used to train a pre-built neural network model based on the parameter location association set and the sample image feature set to obtain the trained vital sign recognition model.

[0208] In the above embodiments, by constructing a vital sign recognition model with multi-device adaptability, the model can automatically extract vital sign parameters according to the device type, thereby improving the recognition accuracy and versatility and providing reliable technical support for the diversified recognition needs of medical devices.

[0209] An automatic extraction system for vital signs information based on medical devices according to an embodiment of this application can implement any of the above extraction methods, and the specific working process of each module in the extraction system can refer to the corresponding process in the above method embodiments.

[0210] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for example, the division of a certain module is merely a logical functional division, and in actual implementation there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0211] This application also discloses a computer device.

[0212] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for automatically extracting vital signs information based on a medical device.

[0213] This application also discloses a computer-readable storage medium.

[0214] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above in any of the methods for automatic extraction of vital signs information based on medical devices.

[0215] The computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device; the program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0216] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0217] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A medical device-based vital sign information automatic extraction method, characterized by, The extraction method comprises: receiving a medical device display image collected by a user and pre-processing the image; extracting image features from the pre-processed medical device display image to obtain an image feature set; extracting device aggregation features from the image feature set based on a device identification feature library; based on the device aggregation features, identifying the device type of the medical device and matching the corresponding sign recognition model of the device type; based on the sign recognition model, extracting the vital sign parameter types corresponding to the device type and the feature positions associated with the vital sign parameter types to obtain an associated feature set; based on the associated feature set, identifying the parameter values at the feature positions associated with each of the vital sign parameter types in the image feature set to obtain a vital sign parameter set; performing structured processing on the vital sign parameter set to output structured vital sign information; further comprising a construction process of the sign recognition model, the specific steps of the construction process comprising: obtaining a sample image data set and pre-processing the data set; wherein the sample image data set comprises sample image data of multiple different medical devices; performing image feature extraction on the pre-processed sample image data set to obtain a sample image feature set; based on device model annotation information, clustering device types and extracting device aggregation features from the sample image feature set; based on the device aggregation feature set and the corresponding device type, constructing a device identification feature library; wherein the device identification feature library records the aggregation feature information of different device types and their corresponding relationship with the sign parameter layout model; extracting the sign parameter layout model of the corresponding device type from the device identification feature library; wherein the sign parameter layout model reflects the specific layout and position of each sign parameter in different device types; matching each feature position in the sample image feature set with the position in the sign parameter layout model to determine the vital sign parameter type corresponding to each feature position; based on the association between the feature position and the vital sign parameter type, obtaining a parameter position association set; based on the parameter position association set and the sample image feature set, training a pre-constructed neural network model to obtain a trained sign recognition model.

2. The method of claim 1, wherein the medical device-based vital sign information is automatically extracted by using a method comprising: The pre-processing step comprises: performing grayscale processing on the medical device display image to obtain a grayscale image; performing denoising processing on the grayscale image based on Gaussian filtering; performing edge detection on the denoised grayscale image; based on the edge detection result, determining and extracting the foreground boundary region of the grayscale image to obtain the corresponding foreground image.

3. The method according to claim 1, wherein the step of identifying the device type of the medical device based on the device aggregation features and matching the corresponding sign recognition model of the device type comprises: based on the aggregation feature matching library, determining the corresponding device type according to the device aggregation features; determining whether there is a sign recognition model corresponding to the device type; If yes, the next step is continued; if no, feedback prompt information indicating that the device type is not supported is sent to the user terminal.

4. The method of claim 3, wherein the medical device-based vital sign information is automatically extracted by using a method comprising: The step of training a pre-constructed neural network model based on the parameter position association set and the sample image feature set to obtain a trained vital sign recognition model comprises: annotating the sample image feature set based on the parameter position association set to construct an annotated data set; dividing the annotated data set into a training set, a validation set and a test set; inputting the training set into a pre-constructed convolutional neural network model for initial training, and verifying and optimizing model parameters based on the validation set and the test set to obtain the trained vital sign recognition model.

5. The method of claim 4, wherein the medical device-based vital sign information is automatically extracted by: The step of inputting the training set into a pre-constructed convolutional neural network model for initial training, and verifying and optimizing model parameters based on the validation set and the test set comprises: inputting the training set into a pre-constructed convolutional neural network model for iterative training, optimizing model parameters and calculating a loss function of the model until the loss function meets a preset condition to obtain the convolutional neural network model after initial training; verifying the convolutional neural network model after initial training based on the validation set, evaluating the performance of the convolutional neural network model and adjusting model parameters according to the verification result; testing the prediction ability of the adjusted convolutional neural network model based on the test set to obtain the trained vital sign recognition model.

6. A medical device-based vital sign information automatic extraction system characterized by comprising: The extraction system comprises: an image receiving module configured to receive a medical device display image collected by a user; a preprocessing module configured to preprocess the medical device display image; an image feature extraction module configured to extract image features from the preprocessed medical device display image to obtain an image feature set; a device aggregation feature extraction module configured to extract device aggregation features from the image feature set based on a device identification feature library; a device type identification module configured to identify the device type of the medical device based on the device aggregation features; a model matching module configured to match a vital sign recognition model corresponding to the device type; an associated feature set generation module configured to extract, based on the vital sign recognition model, a vital sign parameter type corresponding to the device type and a feature position associated with the vital sign parameter type to obtain an associated feature set; a parameter value identification module configured to identify, based on the associated feature set, a parameter value at the feature position associated with each vital sign parameter type in the image feature set to obtain a vital sign parameter set; a structured processing module configured to perform structured processing on the vital sign parameter set to output structured vital sign information; the extraction system further comprises: a sample image acquisition module configured to acquire a sample image data set; wherein the sample image data set comprises sample image data of multiple different medical devices; the preprocessing module is further configured to preprocess the sample image data set; the image feature extraction module is further configured to extract image features from the preprocessed sample image data set to obtain a sample image feature set; The clustering module is configured to cluster the device types based on the device model marking information, and extract a device aggregate feature set from the sample image feature set; The identification feature library construction module is configured to construct a device identification feature library according to the device aggregate feature set and the corresponding device types, wherein the device identification feature library records the aggregate feature information of different device types and the corresponding relationship between the aggregate feature information and the vital sign parameter layout model; The layout model extraction module is configured to extract the vital sign parameter layout model of the corresponding device type from the device identification feature library, wherein the vital sign parameter layout model reflects the specific layout and position of each vital sign parameter in different device types; The position matching module is configured to match each feature position in the sample image feature set with the position in the vital sign parameter layout model, and determine the vital sign parameter type corresponding to each feature position; The association set generation module is configured to obtain a parameter position association set according to the association relationship between the feature position and the vital sign parameter type; The vital sign recognition model training module is configured to train a pre-constructed neural network model based on the parameter position association set and the sample image feature set, and obtain a trained vital sign recognition model.

7. A computer device, characterized by: The computer program stored in the memory and executable on the processor, and the processor executes the program to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer program stored in the memory and executable on the processor, and the processor executes the program to implement the method of any one of claims 1 to 5.

Images