Vital sign information recognition method, system and electronic device

Abstract

The present application provides a method, system and electronic device for identifying vital sign information. A specific implementation of the method includes: obtaining an image to be identified; the image to be identified includes vital sign information; extracting image features representing physiological index areas in the image to be identified by an image feature extraction module of an information recognition model to obtain a target feature map; and identifying the vital sign information in the target feature map by the information recognition module of the information recognition model. The method can improve the accuracy and timeliness of obtaining vital sign information.

Description

Technical Field

[0001] The present application relates to the field of information processing, and in particular, to a method, system and electronic device for identifying vital sign information. Background Art

[0002] The monitor can monitor the patient's vital signs data to help medical staff fully and accurately understand the patient's condition, judge the development trend of the disease, etc., so that they can make corresponding diagnosis and treatment decisions in a timely manner.

[0003] In the process of analyzing the patient's condition, it is necessary to first capture the patient's respiratory rate, blood pressure, heart rate, body temperature and other vital signs from the data displayed by the monitor, and then analyze and process these vital signs to understand the patient's condition.

[0004] In the related art, medical staff need to manually obtain the patient's relevant vital signs information based on the display data of the monitor. The accuracy and timeliness of the patient's vital signs information obtained in this way are low. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method, system and electronic device for identifying vital sign information, so as to improve the accuracy and timeliness of obtaining vital sign information.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying vital sign information, the method comprising: acquiring an image to be identified; the image to be identified including vital sign information; extracting image features representing physiological indicator regions in the image to be identified using an image feature extraction module of an information recognition model to obtain a target feature map; and identifying the vital sign information in the target feature map using the information recognition module of the information recognition model. In this way, the vital sign information in the image to be processed can be automatically identified using the information recognition model, eliminating the need for medical personnel to manually acquire relevant vital sign information, improving the errors in vital sign information and slow acquisition speed caused by manual acquisition, and improving the accuracy and timeliness of acquired vital sign information.

[0007] Optionally, the image feature extraction module includes a convolutional neural network layer, a pooling operation layer, and a fusion layer; and the image feature extraction module using the information recognition model extracts image features representing the physiological indicator area in the image to be identified to obtain a target feature map, including: performing convolution calculation on the image to be identified through the convolutional neural network layer to obtain at least one convolution feature map; performing a pooling operation on the last convolution feature map output by the convolutional neural network layer through the pooling operation layer to obtain a pooling feature map; and fusing the convolution feature map and the pooling feature map through the fusion layer to obtain the target feature map. In this way, the image to be identified can be continuously processed through the convolutional neural network layer, the pooling operation layer, and the fusion layer, thereby extracting more accurate image features.

[0008] Optionally, the convolutional neural network layer includes multiple initial convolutional layers and multiple self-attention connection layers; wherein, at least one of the initial convolutional layers and at least one of the self-attention connection layers constitute a stacking unit of the convolutional neural network layer; and the convolutional neural network layer performs convolution calculation on the image to be identified to obtain at least one convolution feature map, including: for each of the stacking units, performing convolution calculation on the image to be identified through the initial convolutional layer to obtain a convolution result; and performing a feature extraction operation on the convolution result through the self-attention connection layer to obtain a convolution feature map; wherein the input of the current stacking unit is the output of the previous stacking unit. In this way, the above-mentioned initial convolutional layer and self-attention connection layer can be used to perform feature extraction operations on the image to be processed, thereby extracting more accurate image features.

[0009] Optionally, the self-attention connection layer includes an attention layer and a connection layer; and the performing of a feature extraction operation on the convolution result through the self-attention connection layer to obtain a convolution feature map includes: performing a feature extraction operation on the output result of the initial convolution layer through the CA attention mechanism provided by the attention layer to obtain an attention layer feature map; performing a feature extraction operation on the output result of the initial convolution layer through the connection layer to obtain a connection layer feature map; and fusing the attention layer feature map and the connection layer feature map to obtain the convolution feature map. In this way, the self-attention mechanism and the residual network can be combined to process the image to be recognized, which is conducive to obtaining more accurate image features.

[0010] Optionally, the CA attention mechanism provided by the attention layer performs a feature extraction operation on the output result of the initial convolution layer to obtain an attention layer feature map, including performing the following operations through the attention layer: performing a pooling operation on the output result of the initial convolution layer from the channel dimension to obtain a channel feature map; performing a pooling operation on the output result of the initial convolution layer from the spatial dimension to obtain a spatial feature map; fusing the channel feature map and the spatial feature map to obtain a channel-space fusion image; performing a convolution calculation on the channel-space fusion image through a convolution layer with different convolution kernels to obtain a heterogeneous kernel convolution result; fusing the heterogeneous kernel convolution result and the output result of the initial convolution layer to obtain the attention layer feature map. In this way, the CA attention mechanism can be used to focus on the channel information and position information of the image feature layer, thereby extracting more accurate image features.

[0011] Optionally, the connection layer includes multiple cascaded residual submodules; and performing a feature extraction operation on the output result of the initial convolutional layer through the connection layer to obtain a connection layer feature map includes performing the following operations through the connection layer: gradually performing feature extraction operations on the output result of the initial convolutional layer through multiple cascaded residual submodules to obtain a residual feature map; and fusing the residual feature map with the output result of the initial convolutional layer to obtain the connection layer feature map. In this way, the multiple cascaded residual submodules can better capture the details of image features and contextual information, thereby improving the expressive power of the image extraction module.

[0012] Optionally, the pooling operation layer includes at least two branch structures, wherein the first branch structure includes at least one initial convolution layer and multiple convolution layers in parallel; wherein the convolution kernels of the multiple convolution layers in parallel are different; the output of the at least one initial convolution layer is the input of the multiple convolution layers in parallel; the second branch structure includes at least one initial convolution layer; and the pooling operation is performed on the last convolution feature map output by the convolutional neural network layer through the pooling operation layer to obtain a pooling feature map, including: performing a convolution operation on the convolution feature map through the first branch structure to obtain a first pooling convolution result; performing multiple convolution operations on the convolution feature map through the second branch structure to obtain a second pooling convolution result; and fusing the first pooling convolution result and the second pooling convolution result to obtain the pooling feature map. In this way, the input convolution feature map can be processed by the first branch structure and the second branch structure to obtain a more accurate pooling feature map.

[0013] Optionally, before extracting image features representing physiological indicator regions in the image to be identified using the image feature extraction module of the information recognition model to obtain a target feature map, the method further includes: training the information recognition model; wherein the loss functions used in the training process include at least the following three: a classification loss function, a confidence loss function, and a localization loss function. In this way, the information recognition model can be trained using a loss function composed of the classification loss function, the confidence loss function, and the localization loss function, thereby improving the recognition capability of the information recognition model to a certain extent.

[0014] In a second aspect, an embodiment of the present application provides a vital sign information recognition system, which includes: an acquisition module for acquiring an image to be recognized; the image to be recognized includes vital sign information; a recognition module for extracting image features representing physiological indicator areas in the image to be recognized through an image feature extraction module of an information recognition model to obtain a target feature map; and identifying the vital sign information in the target feature map through the information recognition module of the information recognition model. In this way, the vital sign information in the image to be processed can be automatically identified through the information recognition model, eliminating the need for medical staff to manually obtain relevant vital sign information, improving the situation of vital sign information errors and slow acquisition speed caused by manual acquisition, and improving the accuracy and timeliness of obtaining vital sign information.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect above are executed.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the method provided in the first aspect are executed.

[0017] Other features and advantages of the present application will be described in the following description and, in part, will become apparent from the description or be understood by practicing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a method for identifying vital signs information provided in an embodiment of the present application;

[0020] Figure 2 A specific structural diagram of an image feature extraction module provided in an embodiment of the present application;

[0021] Figure 3 A specific structural diagram of a self-attention connection layer provided in an embodiment of the present application;

[0022] Figure 4 A specific structural diagram of a pooling operation layer provided in an embodiment of the present application;

[0023] Figure 5 A structural block diagram of a vital sign information recognition system provided in an embodiment of the present application;

[0024] Figure 6 A schematic structural diagram of an electronic device for executing a method for identifying vital sign information provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0027] It should be noted that, unless there is any conflict, the embodiments in this application or the technical features in the embodiments may be combined.

[0028] In the related art, there is a problem of low accuracy and timeliness in obtaining vital sign information; in order to solve this problem, the present application provides a vital sign information recognition method, system and electronic device; further, the patient's relevant vital sign information is automatically obtained from the display data of the monitor through an information recognition model without the need for manual acquisition, thereby improving the accuracy and timeliness of obtaining vital sign information, and achieving the purpose of assisting medical staff in treating patients.

[0029] In some application scenarios, the above-mentioned vital sign information recognition method can be applied to terminal devices, which may include, for example, desktop computers, laptop computers, etc. In other application scenarios, the above-mentioned vital sign information recognition method can also be applied to server-side devices, which may include, for example, servers, server clusters, cloud platforms, etc. The terminal device or the server-side device can obtain the video data from the monitor and then automatically obtain the patient's vital sign information, thereby improving the accuracy and timeliness of obtaining the vital sign information. By way of example, the following text of this application is applied to terminal devices.

[0030] The defects existing in the solutions in the above-mentioned related technologies are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above-mentioned problems and the solutions proposed in the embodiments of the present invention below for the above-mentioned problems should all be the contributions made by the inventors to the present invention during the process of the invention.

[0031] Please refer to Figure 1 , which shows a flow chart of a method for identifying vital signs information provided by an embodiment of the present application. Figure 1 As shown, the vital sign information recognition method includes the following steps 101 to 102.

[0032] Step 101: Acquire an image to be identified; the image to be identified includes vital sign information;

[0033] The above-mentioned vital sign information is also called vital sign information, which may include, for example, the patient's respiratory rate, blood pressure, heart rate, body temperature and other data information.

[0034] In some application scenarios, a patient monitor can collect real-time vital signs, which can be displayed on the monitor in video format. The terminal device can obtain video data from the monitor and then convert it into multiple frames, each of which can include the corresponding vital signs. Each frame can be considered as the image to be recognized.

[0035] It should be noted that the terminal device can obtain video data from the monitor in real time, can obtain video data from the monitor at a fixed time, or can obtain video data after receiving an acquisition instruction from medical staff. There is no limitation here.

[0036] Step 102: extract image features representing the physiological indicator area in the image to be identified through the image feature extraction module of the information recognition model to obtain a target feature map; and identify the vital sign information in the target feature map through the information recognition module of the information recognition model.

[0037] After the terminal device acquires the image to be identified, it can process the image to be identified through the information recognition model. Specifically, the terminal device can input the image to be identified into the information recognition model, and the information recognition model can identify the vital sign information. The above-mentioned information recognition model can, for example, use a target detection algorithm model including the YOLO (You Only Look Once, abbreviated as YOLO) series (such as YOLOv5, YOLOv7, etc.) as an image feature extraction module. The image feature extraction module can, for example, apply different convolution functions to identify image features representing the physiological indicator area from the image to be identified, thereby obtaining a target feature map. The above-mentioned physiological indicator area is also an image area including indicator data, which can, for example, be an image area recording respiratory rate data, an image area recording body temperature data, etc. The information recognition module can identify the corresponding vital sign information from the target feature map. In these application scenarios, the information recognition module can, for example, apply an OCR (Optical Character Recognition) algorithm to identify the corresponding vital sign information.

[0038] In this implementation, the terminal device can automatically identify the vital sign information in the image to be processed through the information recognition model, thereby eliminating the need for medical staff to manually obtain relevant vital sign information, improving the situation of vital sign information errors and slow acquisition speed caused by manual acquisition, and improving the accuracy and timeliness of obtaining vital sign information.

[0039] In some optional implementations, the image feature extraction module includes a convolutional neural network layer, a pooling operation layer, and a fusion layer;

[0040] The above convolutional neural network layer can be regarded as a neural network layer including multiple convolutional stacking units. For each convolutional stacking unit, a convolution operation can be performed to obtain a corresponding convolutional feature map.

[0041] It should be noted that the number of stacking units included in the convolutional neural network layer can be pre-set, which can be, for example, 3, 4, etc., and is not limited here.

[0042] The pooling operation layer can be considered as a neural network layer that performs a pooling operation on the last feature map output by the convolutional neural network layer. In some application scenarios, the pooling operation can be, for example, a maximum pooling operation or an average pooling operation.

[0043] The above-mentioned fusion layer can be regarded as a neural network layer that fuses multiple convolutional feature maps with pooling feature maps. In some application scenarios, it can be fused, for example, by a pixel weighted averaging method or a method based on the maximum pixel value. The above-mentioned pixel weighted averaging method can, for example, include assigning different weight values ​​to pixels at the same position in different feature maps, so that the pixel value of the corresponding pixel point of the fused image can be the weighted sum of the pixel values ​​in each feature map. The above-mentioned method based on the maximum pixel value can, for example, determine the maximum pixel value as the pixel value of the corresponding pixel point of the fused image for pixels at the same position in different feature maps.

[0044] In this way, the image feature extraction module of the information recognition model in step 102 extracts the image features representing the physiological indicator area in the image to be recognized to obtain a target feature map, including:

[0045] Sub-step 1021, performing convolution calculation on the image to be recognized through the convolutional neural network layer to obtain at least one convolution feature map;

[0046] Sub-step 1022: performing a pooling operation on the last convolutional feature map output by the convolutional neural network layer through the pooling operation layer to obtain a pooled feature map;

[0047] Sub-step 1023: fusing the convolutional feature map and the pooling feature map through the fusion layer to obtain the target feature map.

[0048] In this implementation, the image to be recognized can be continuously processed through the convolutional neural network layer, the pooling operation layer and the fusion layer, so as to extract more accurate image features.

[0049] In some optional implementations, the convolutional neural network layer includes multiple initial convolutional layers and multiple self-attention connection layers; wherein at least one of the initial convolutional layers and at least one of the self-attention connection layers constitute a stacking unit of the convolutional neural network layer;

[0050] The above initial convolution layer can be regarded as a network layer that initially performs convolution operations, which can apply convolution functions, normalization functions, activation functions, etc. to process the image to be recognized.

[0051] The above-mentioned self-attention connection layer can be regarded as a network layer using the self-attention mechanism, which can, for example, apply a multi-scale channel attention mechanism, a spatial attention mechanism, etc. to process the output results of the initial convolutional layer.

[0052] In this way, the above sub-step 1021 described in which convolution calculation is performed on the image to be identified through the convolutional neural network layer to obtain at least one convolution feature map includes: for each of the stacking units, convolution calculation is performed on the image to be identified through the initial convolution layer to obtain a convolution result; and feature extraction operation is performed on the convolution result through the self-attention connection layer to obtain a convolution feature map; wherein the input of the current stacking unit is the output of the previous stacking unit.

[0053] That is, the terminal device can input the image to be recognized into the first stacking unit, obtain the output result of the first stacking unit, then input the output result into the second stacking unit to obtain the second output result, and so on, until all stacking units are executed. Within a single stacking unit, the initial convolutional layer performs a convolution operation on the output result of the previous layer to obtain a convolution result. The self-attention connection layer then performs feature extraction based on the correlation between different parts of the convolution result, thereby obtaining a convolution feature map.

[0054] In this implementation, the above-mentioned initial convolution layer and self-attention connection layer can be used to perform feature extraction operations on the image to be processed, thereby extracting more accurate image features.

[0055] In some application scenarios, the specific structure between the layers can be as follows: Figure 2 See Figure 2 The image feature extraction module included in the YOLOv5 model can include a pixel combination layer, a focus layer, which can extract each pixel in the feature map of the image to be recognized and then recombine it to obtain a feature map with a height and width reduced to half of the input and a number of channels increased to four times that of the input. The image feature extraction module can also include a convolutional neural network layer consisting of an initial convolution layer Conv2d_BN_LR and a self-attention connection layer CA_CSP, as well as a pooling operation layer SPPCSPC and a fusion layer.

[0056] In these application scenarios, the image feature extraction module may, for example, receive an image to be identified with a size of (640, 640, 3), where the parameters of the size represent the height, width, and number of channels of the image, respectively. The image feature extraction module first outputs a feature map of size (320, 320, 12) through the focus layer, and then outputs a feature map of size (320, 320, 64) and a feature map of size (160, 160, 128) through two Conv2d_BN_LR layers respectively; then outputs a feature map of size (160, 160, 128) through a CA_CSP layer; then, the image feature extraction module outputs feature maps of size (80, 80, 256) through a Conv2d_BN_LR layer and a CA_CSP layer as feature layer 1; similarly, the image feature extraction module outputs a feature map of size (40, 40, 512) through a Conv2d_BN_LR layer and a CA_CSP layer as feature layer 2; then, a feature map of size (20, 20, 1024) is output through a Conv2d_BN_LR layer, a CA_CSP layer and a SPPCSPC layer. Figure 3 .

[0057] For these three feature maps, the image feature extraction module can first extract the features Figure 3 After the convolution operation (i.e., through the Conv2D layer), upsampling is performed (i.e., through the UpSampling2D layer), and then the upsampling result is combined with the feature Figure 2 The fusion process is performed, and the fusion result is processed by self-attention through the CA_CSP layer (that is, through the Concat+CA_CSP layer); then, the self-attention processing result is convolved (that is, through the Conv2D layer) and then up-sampled (that is, through the UpSampling2D layer), and then the up-sampling processing result is combined with the feature Figure 1 The fusion process is performed, and the fusion result is processed by self-attention through the CA_CSP layer (that is, through the Concat+CA_CSP layer) to obtain the first output result. Then, the self-attention processing result is downsampled (that is, through the DownSample layer) and combined with the feature Figure 2 and features Figure 3 The convolution result of the feature vector is fused and self-attention processed to obtain the second output result. Then, the self-attention result is downsampled (i.e., passed through the DownSample layer) and combined with the feature vector. Figure 3 The convolution results are fused and self-attention processed to obtain the third output result.

[0058] In this way, the image feature extraction module forms an FPN network, which can fuse the information of multiple feature maps in the context, improve the feature extraction capability, and to a certain extent improve the recognition capability of the information recognition model.

[0059] In some optional implementations, the self-attention connection layer includes an attention layer and a connection layer;

[0060] The above-mentioned attention layer applies the CA attention (Coordinate attention, referred to as CA attention) mechanism, which can focus on the channel information and position information of the feature layer, thereby extracting more accurate image features.

[0061] The above connection layer is established based on the residual network, which can better capture the details of image features and contextual information, and improve the expressive ability of the model.

[0062] In this way, the self-attention connection layer performs a feature extraction operation on the convolution result to obtain a convolution feature map, including:

[0063] Sub-step 1: performing a feature extraction operation on the output result of the initial convolutional layer through the CA attention mechanism provided by the attention layer to obtain an attention layer feature map;

[0064] In some optional implementations, step 1 may include performing the following operations via the attention layer:

[0065] First, a pooling operation is performed on the output of the initial convolutional layer from the channel dimension to obtain a channel feature map;

[0066] Then, a pooling operation is performed on the output result of the initial convolutional layer from the spatial dimension to obtain a spatial feature map;

[0067] Secondly, the channel feature map and the spatial feature map are fused to obtain a channel-space fused image;

[0068] Secondly, convolution calculation is performed on the channel-space fusion image through convolution layers with different convolution kernels to obtain a heterogeneous kernel convolution result;

[0069] Finally, the heterogeneous kernel convolution result and the output result of the initial convolution layer are fused to obtain the attention layer feature map.

[0070] In this implementation, the CA attention mechanism can be used to focus on the channel information and position information of the image feature layer, thereby extracting more accurate image features.

[0071] Sub-step 2: performing a feature extraction operation on the output result of the initial convolutional layer through the connection layer to obtain a connection layer feature map;

[0072] In some optional implementations, the connection layer includes a plurality of cascaded residual submodules; and the performing of a feature extraction operation on the output result of the initial convolutional layer by the connection layer in the above sub-step 2 to obtain a connection layer feature map includes performing the following operations by the connection layer:

[0073] First, a feature extraction operation is gradually performed on the output result of the initial convolutional layer through multiple cascaded residual submodules to obtain a residual feature map;

[0074] The above residual submodule is also known as the Res_Block module. Multiple cascaded Res_Block modules are the basic modules that constitute the ResNet network structure. By stacking multiple cascaded Res_Block modules, we can better capture the details of image features and contextual information, and obtain a more accurate residual feature map.

[0075] Then, the residual feature map and the output result of the initial convolutional layer are fused to obtain the connection layer feature map.

[0076] In this implementation, multiple cascaded residual submodules can be used to better capture the details of image features and contextual information, thereby improving the expressive power of the image extraction module.

[0077] Sub-step 3: Fusing the attention layer feature map and the connection layer feature map to obtain the convolution feature map.

[0078] In this implementation, the self-attention mechanism and the residual network can be combined to process the image to be recognized, which is conducive to obtaining more accurate image features.

[0079] In some application scenarios, the specific structure of the self-attention connection layer can be as follows: Figure 3As shown. Among them, the image features from the previous layer are input into two Conv2d_BN_LR layers after passing through the Base_Layer layer (input layer). Among them, the output result of a Conv2d_BN_LR layer is divided into three branches. One is not processed in any way. The second is to perform a pooling operation on the output result of the Conv2d_BN_LR layer from the channel dimension through the X_Avg_Pool layer to obtain a channel feature map; the third is to perform a pooling operation on the output result of the Conv2d_BN_LR layer from the spatial dimension through the Y_Avg_Pool to obtain a spatial feature map, and then the channel feature map and the spatial feature map are fused through the Concat_Conv2d_BN layer to obtain a channel-spatial fusion image; then, convolution calculations are performed on two Conv2d_Sigmoid layers with different convolution kernels to obtain heterogeneous kernel convolution results; and then the Re_Weight layer is used to fuse the heterogeneous kernel convolution results and the above-mentioned image feature map without any processing using the pixel weighted average method to obtain the attention layer feature map.

[0080] The output of another Conv2d_BN_LR layer is then divided into two branches. One branch does no processing, while the other branch performs residual processing on the output of the Conv2d_BN_LR layer through the stacked Res_Block module to obtain a residual feature map. This residual feature map is then fused with the unprocessed image feature map through the Concat layer to obtain the concatenated layer feature map.

[0081] Finally, a Concat layer is used to fuse the connection layer feature map and the attention layer feature map to obtain a convolutional feature map with higher accuracy.

[0082] In some optional implementations, the pooling operation layer includes at least two branch structures, wherein the first branch structure includes at least one initial convolution layer and multiple convolution layers in parallel; wherein the convolution kernels of the multiple convolution layers in parallel are different; the output of the at least one initial convolution layer is the input of the multiple convolution layers in parallel; and the second branch structure includes at least one initial convolution layer;

[0083] The step 102 described above of performing a pooling operation on the last convolutional feature map output by the convolutional neural network layer through the pooling operation layer to obtain a pooled feature map includes:

[0084] First, a convolution operation is performed on the convolution feature map through the first branch structure to obtain a first pooled convolution result;

[0085] Then, performing multiple convolution operations on the convolution feature map through the second branch structure to obtain a second pooled convolution result;

[0086] Finally, the first pooling convolution result and the second pooling convolution result are fused to obtain the pooling feature map.

[0087] In some application scenarios, the specific structure of the pooling operation layer can be as follows: Figure 4 As shown in the figure, the image feature map from the previous layer is input into the pooling operation layer through the input layer (also known as the Inputs layer), and then the output of the input layer is processed through two branch structures.

[0088] Among them, the first branch structure passes through a Conv2d_BN_LR layer to perform a convolution operation on the feature map, and then the convolution result is divided into 4 paths for processing. The first path does not perform any processing, the second path passes through a network layer with a convolution kernel of 5*5 for convolution operation, the third path passes through a network layer with a convolution kernel of 9*9 for convolution operation, and the fourth path passes through a network layer with a convolution kernel of 13*13 for convolution operation. The convolution results of the 4 paths are then subjected to a maximum pooling operation with a step size of 1, thereby obtaining the first pooled convolution result. The first pooled convolution result can also be convolved through two layers of Conv2d_BN_LR layers to improve the accuracy of the image feature map;

[0089] The second branch includes multiple Conv2d_BN_LR layers to perform multiple convolution operations on the feature map to obtain the second pooled convolution result.

[0090] The first pooled convolution result and the second pooled convolution result are then fused through a Conv2d_BN_RL layer to obtain a pooled feature map.

[0091] In this implementation, the input convolution feature map can be processed by the first branch structure and the second branch structure to obtain a more accurate pooling feature map.

[0092] In some optional implementations, before extracting the image features representing the physiological indicator area in the image to be identified by the image feature extraction module of the information recognition model to obtain the target feature map, the method also includes: training the information recognition model; wherein the loss function used in the training process includes at least the following three: classification loss function, confidence loss function and positioning loss function.

[0093] Before the information recognition model is put into use, it needs to be trained.

[0094] In some application scenarios, the sample dataset can be divided into a training set, a validation set, and a test set. For example, the data size ratio of the training set, validation set, and test set can be 8:1:1. The training set is used to train the information recognition model, the validation set is used to evaluate the information recognition model, and the test set is used to test whether the model is qualified.

[0095] In these application scenarios, for the sample data in the training set, the image feature extraction module may process it in the same or similar manner as the processing of the recognition image described above, and will not be described in detail here.

[0096] In these application scenarios, after the image feature extraction module outputs the target feature map corresponding to each sample data, it is necessary to calculate the loss value between the target feature map and the original feature map of the image to be identified.

[0097] In this way, when the information recognition model is established based on the Yolov5 model, its loss value can be calculated by, for example, a loss function including a classification loss function, a confidence loss function, and a positioning loss function.

[0098] After calculating the loss value, the Yolov5 model can be evaluated on the validation set. In some application scenarios, for example, the classification performance evaluation indicator mAP (mean Average Precision) can be used for evaluation. That is, mAP represents the average precision value of all categories, where "precision" refers to the degree of match between the model's detection results and the real data.

[0099] In some application scenarios, the evaluation process may include the following steps:

[0100] Step a: For each category, sort all prediction results from high to low according to their confidence.

[0101] Step b: Calculate the Precision and Recall of each prediction result in turn. Precision represents the ratio of correctly detected targets to all detected targets, and Recall represents the ratio of correctly detected targets to all true targets.

[0102] In some application scenarios, the calculation formula for Precision can be: The calculation formula for Recall can be: Among them, TP represents the number of samples predicted to be positive and actually are positive, FP represents the number of samples predicted to be positive and actually are negative, and TN represents the number of samples predicted to be negative and actually are positive.

[0103] Step c: draw a Precision-Recall curve based on Precision and Recall.

[0104] Step d: Calculate the area under the PR curve (Average Precision, referred to as AP) as the accuracy value of the category.

[0105] In step e, the average AP value of all categories is calculated to obtain mAP. The mAP calculation formula is: mAP = (AP_1 + AP_2 + ... + AP_n) / n; where n represents the number of categories and AP_n represents the average precision value of the nth category.

[0106] In this implementation, the information recognition model can be trained by a loss function composed of a classification loss function, a confidence loss function, and a positioning loss function, thereby improving the recognition ability of the information recognition model to a certain extent.

[0107] Those skilled in the art will understand that, in the above-mentioned method of a specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0108] Please refer to Figure 5 , which shows a structural block diagram of a vital sign information recognition system provided by an embodiment of the present application. The vital sign information recognition system can be a module, program segment or code on an electronic device. It should be understood that the system is similar to the above-mentioned Figure 1 The method embodiment corresponds to the embodiment that can be executed Figure 1 The various steps involved in the method embodiment.

[0109] Optionally, the vital sign information recognition system includes an acquisition module 501 and a recognition module 502. The acquisition module 501 is configured to acquire an image to be recognized, wherein the image to be recognized includes vital sign information. The recognition module 502 is configured to extract image features representing physiological indicator regions in the image to be recognized using an image feature extraction module of an information recognition model to obtain a target feature map. The information recognition module of the information recognition model is then used to recognize the vital sign information in the target feature map.

[0110] Optionally, the image feature extraction module includes a convolutional neural network layer, a pooling operation layer and a fusion layer; and the recognition module 502 is further used to: perform convolution calculation on the image to be recognized through the convolutional neural network layer to obtain at least one convolution feature map; perform pooling operation on the last convolution feature map output by the convolutional neural network layer through the pooling operation layer to obtain a pooling feature map; and fuse the convolution feature map and the pooling feature map through the fusion layer to obtain the target feature map.

[0111] Optionally, the convolutional neural network layer includes multiple initial convolutional layers and multiple self-attention connection layers; wherein, at least one of the initial convolutional layers and at least one of the self-attention connection layers constitute a stacking unit of the convolutional neural network layer; and the recognition module 502 is further used to: for each of the stacking units, perform convolution calculation on the image to be recognized through the initial convolutional layer to obtain a convolution result; and perform feature extraction operation on the convolution result through the self-attention connection layer to obtain a convolution feature map; wherein, the input of the current stacking unit is the output of the previous stacking unit.

[0112] Optionally, the self-attention connection layer includes an attention layer and a connection layer; and the recognition module 502 is further used to: perform a feature extraction operation on the output result of the initial convolution layer through the CA attention mechanism provided by the attention layer to obtain an attention layer feature map; and perform a feature extraction operation on the output result of the initial convolution layer through the connection layer to obtain a connection layer feature map; and fuse the attention layer feature map and the connection layer feature map to obtain the convolution feature map.

[0113] Optionally, the recognition module 502 is further used to perform the following operations through the attention layer: perform a pooling operation on the output result of the initial convolution layer from the channel dimension to obtain a channel feature map; perform a pooling operation on the output result of the initial convolution layer from the spatial dimension to obtain a spatial feature map; fuse the channel feature map and the spatial feature map to obtain a channel-space fusion image; perform convolution calculation on the channel-space fusion image through a convolution layer with different convolution kernels to obtain a heterogeneous kernel convolution result; fuse the heterogeneous kernel convolution result and the output result of the initial convolution layer to obtain the attention layer feature map.

[0114] Optionally, the connection layer includes multiple cascaded residual sub-modules; and the identification module 502 is further used to perform the following operations through the connection layer: gradually perform feature extraction operations on the output results of the initial convolutional layer through multiple cascaded residual sub-modules to obtain a residual feature map; and fuse the residual feature map and the output results of the initial convolutional layer to obtain the connection layer feature map.

[0115] Optionally, the pooling operation layer includes at least two branch structures, wherein the first branch structure includes at least one initial convolution layer and multiple convolution layers in parallel; wherein the convolution kernels of the multiple convolution layers in parallel are different; the output of the at least one initial convolution layer is the input of the multiple convolution layers in parallel; the second branch structure includes at least one initial convolution layer; and the recognition module 502 is further used to: perform a convolution operation on the convolution feature map through the first branch structure to obtain a first pooling convolution result; perform multiple convolution operations on the convolution feature map through the second branch structure to obtain a second pooling convolution result; and fuse the first pooling convolution result and the second pooling convolution result to obtain the pooling feature map.

[0116] Optionally, the system also includes a training module, which is used to: train the information recognition model before the image feature extraction module of the information recognition model extracts the image features representing the physiological indicator area in the image to be identified and obtains the target feature map; wherein the loss functions used in the training process include at least the following three: classification loss function, confidence loss function and positioning loss function.

[0117] In some application scenarios, the vital sign information recognition system may also include a data storage module. This data storage module is used to store the processing result data generated by each processing step, so as to facilitate subsequent data analysis and mining operations based on this processing result data. For example, a MySQL relational database can be used, using batch storage to store data. In these application scenarios, database data can also be regularly backed up to cope with emergencies such as data loss and corruption.

[0118] In some application scenarios, the vital sign information recognition system may also include a data visualization module. The data visualization module is used to display data in a visual manner so that medical staff can intuitively monitor the patient's vital signs;

[0119] In some application scenarios, the vital sign information recognition system may also include a data analysis module, which can be used to analyze and compile statistics on the collected vital sign information in order to better understand the distribution trend of the data and to handle abnormal data in a timely manner when it is found.

[0120] In some application scenarios, the above system can be used in the ICU (Intensive Care Unit) to automatically collect the patient's vital signs information and help medical staff obtain vital signs information quickly and accurately in the ICU.

[0121] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0122] Please refer to Figure 6 , Figure 6 A structural diagram of an electronic device for executing a method for identifying vital sign information provided in an embodiment of the present application, the electronic device may include: at least one processor 601, such as a CPU, at least one communication interface 602, at least one memory 603 and at least one communication bus 604. Among them, the communication bus 604 is used to realize direct connection and communication between these components. Among them, the communication interface 602 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The memory 603 can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 603 can optionally also be at least one storage device located away from the aforementioned processor. Computer-readable instructions are stored in the memory 603. When the computer-readable instructions are executed by the processor 601, the electronic device can execute the above-mentioned Figure 1 The method process shown.

[0123] I understand. Figure 6 The structure shown is only for illustration, and the electronic device may also include Figure 6 More or fewer components than shown, or with Figure 6 Different configurations shown. Figure 6 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0124] The embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the following can be performed: Figure 1 The method process in the illustrated method embodiment is performed by the electronic device.

[0125] An embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments. For example, the method may include: obtaining an image to be identified; the image to be identified includes vital sign information; extracting image features representing physiological indicator areas in the image to be identified through an image feature extraction module of an information recognition model to obtain a target feature map; and identifying the vital sign information in the target feature map through the information recognition module of the information recognition model.

[0126] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0129] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0130] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for identifying vital sign information, characterized in that: include: Acquire an image to be recognized from video data from a monitor; The image to be identified includes vital sign information; The physical sign information is vital sign information; Extracting image features representing the physiological index area in the image to be identified by an image feature extraction module of the information recognition model to obtain a target feature map; and identifying the vital sign information in the target feature map by the information recognition module of the information recognition model; The image feature extraction module includes a convolutional neural network layer, a pooling operation layer and a fusion layer; the convolutional neural network layer is used to perform convolution calculation on the image to be identified to obtain at least one convolutional feature map, and the convolutional feature map is used to obtain the target feature map; The convolutional neural network layer includes multiple initial convolutional layers and multiple self-attention connection layers; The self-attention connection layer includes an attention layer and a connection layer; and, a feature extraction operation is performed on the output result of the initial convolution layer through the CA attention mechanism provided by the attention layer to obtain an attention layer feature map; And, performing a feature extraction operation on the output result of the initial convolutional layer through the connection layer to obtain a connection layer feature map; fusing the attention layer feature map and the connection layer feature map to obtain the convolution feature map; The connection layer includes a plurality of cascaded residual submodules; as well as The step of performing a feature extraction operation on the output result of the initial convolutional layer through the connection layer to obtain a connection layer feature map includes performing the following operations through the connection layer: Stepwise feature extraction operations are performed on the output results of the initial convolutional layer through multiple cascaded residual submodules to obtain a residual feature map; The residual feature map and the output result of the initial convolutional layer are fused to obtain the connection layer feature map.

2. The method according to claim 1, characterized in that The image feature extraction module of the information recognition model extracts the image features representing the physiological index area in the image to be recognized to obtain the target feature map, including: Performing convolution calculation on the image to be recognized through the convolutional neural network layer to obtain at least one convolution feature map; Performing a pooling operation on the last convolutional feature map output by the convolutional neural network layer through the pooling operation layer to obtain a pooled feature map; The convolution feature map and the pooling feature map are fused through the fusion layer to obtain the target feature map.

3. The method according to claim 2, characterized in that At least one of the initial convolutional layers and at least one of the self-attention connection layers constitute a stacking unit of the convolutional neural network layer; And the convolution calculation is performed on the image to be recognized through the convolutional neural network layer to obtain at least one convolution feature map, including: For each of the stacking units, performing convolution calculation on the image to be recognized through the initial convolution layer to obtain a convolution result; and Performing a feature extraction operation on the convolution result through the self-attention connection layer to obtain a convolution feature map; The input of the current stacking unit is the output of the previous stacking unit.

4. The method according to claim 1, characterized in that: The CA attention mechanism provided by the attention layer performs a feature extraction operation on the output result of the initial convolutional layer to obtain an attention layer feature map, including performing the following operations through the attention layer: Performing a pooling operation on the output result of the initial convolutional layer from the channel dimension to obtain a channel feature map; Performing a pooling operation on the output result of the initial convolutional layer from a spatial dimension to obtain a spatial feature map; Fusing the channel feature map and the spatial feature map to obtain a channel-space fused image; Performing convolution calculation on the channel-space fusion image through convolution layers with different convolution kernels to obtain a heterogeneous kernel convolution result; The heterogeneous kernel convolution result and the output result of the initial convolution layer are fused to obtain the attention layer feature map.

5. The method according to claim 2, characterized in that: The pooling operation layer includes at least two branch structures, wherein the first branch structure includes at least one initial convolution layer and a plurality of convolution layers in parallel; wherein the convolution kernels of the plurality of convolution layers in parallel are different; the output of the at least one initial convolution layer is the input of the plurality of convolution layers in parallel; the second branch structure includes at least one initial convolution layer; and The step of performing a pooling operation on the last convolutional feature map output by the convolutional neural network layer through the pooling operation layer to obtain a pooled feature map includes: Performing a convolution operation on the convolution feature map through the first branch structure to obtain a first pooling convolution result; Performing multiple convolution operations on the convolution feature map through the second branch structure to obtain a second pooling convolution result; The first pooling convolution result and the second pooling convolution result are fused to obtain the pooling feature map.

6. The method according to any one of claims 2 to 5, characterized in that: Before extracting the image features representing the physiological indicator area in the image to be identified by the image feature extraction module of the information recognition model to obtain the target feature map, the method further includes: Training the information recognition model; Among them, the loss functions used in the training process include at least the following three: classification loss function, confidence loss function and positioning loss function.

7. A vital sign information recognition system, characterized in that: include: An acquisition module, used for acquiring an image to be identified from video data from a monitor; The image to be identified includes vital sign information; The physical sign information is vital sign information; A recognition module, used to extract image features representing the physiological index area in the image to be recognized through the image feature extraction module of the information recognition model to obtain a target feature map; and to recognize the vital sign information in the target feature map through the information recognition module of the information recognition model; the image feature extraction module includes a convolutional neural network layer, a pooling operation layer and a fusion layer; the convolutional neural network layer is used to perform convolution calculation on the image to be recognized to obtain at least one convolutional feature map, and the convolutional feature map is used to obtain the target feature map; the convolutional neural network layer includes multiple initial convolutional layers and multiple self-attention connection layers; The self-attention connection layer includes an attention layer and a connection layer; and, a feature extraction operation is performed on the output result of the initial convolution layer through the CA attention mechanism provided by the attention layer to obtain an attention layer feature map; And, performing a feature extraction operation on the output result of the initial convolution layer through the connection layer to obtain a connection layer feature map; fusing the attention layer feature map and the connection layer feature map to obtain the convolution feature map; the connection layer includes a plurality of cascaded residual sub-modules; and, performing a feature extraction operation on the output result of the initial convolution layer through the connection layer to obtain a connection layer feature map, including performing the following operations through the connection layer: gradually performing feature extraction operations on the output result of the initial convolution layer through a plurality of cascaded residual sub-modules to obtain a residual feature map; fusing the residual feature map and the output result of the initial convolution layer to obtain the connection layer feature map.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is executed.

Images