Cross infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition

Abstract

The invention relates to a cross infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition, and belongs to the technical field of medical cross infection recognition. According to the scheme, monitoring videos in hospital departments are analyzed by using a deep learning semantic segmentation technology, image information of medical staff and patients is extracted, and the image information is classified and identified. The contact behavior between the medical staff and the patient is identified through the deep learning image classification technology, and the moving tracks of the medical staff and the patient in the hospital department are tracked and recorded in real time in combination with the track identification technology, so that the contact condition and the cross infection risk of the medical staff and the patient are analyzed more comprehensively. Compared with the prior art, only existing video information is collected, the cost is low, an existing system is slightly modified, and only a video stream forwarding function is added; patient contact behaviors and medical personnel action tracks can be accurately identified, and cross infection behaviors can be accurately identified; embedded deployment and batch processing technologies are adopted, and second-level response is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of hand hygiene early warning and identification in medical cross-infection prevention, and relates to a cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition. Background Art

[0002] In the daily management of hospital departments and the process of patient care, cross infection is always a serious problem that needs to be solved urgently. Cross infection refers to the spread of diseases due to contact between different patients or medical staff. This phenomenon not only greatly increases the treatment risk of patients, but may also lead to worsening of the disease, prolonged hospitalization and increased medical costs. In severe cases, it may even directly threaten the life safety of patients. Therefore, effective identification and prevention of cross infection is of vital importance to improving the overall management level of hospitals, protecting the legitimate rights and interests of patients, and maintaining the stability and safety of the medical environment.

[0003] The main route of cross-infection transmission is through hand contact between medical staff. Improving hand hygiene awareness is the simplest and most effective measure for hospitals to reduce cross-infection. At present, the identification and prevention of hand hygiene mainly rely on traditional manual real-time identification and review monitoring. Among them, using infrared recognition technology to determine whether medical staff have cross-contact with patients is a common method. However, this method has many limitations in practical applications. Although infrared recognition technology can capture the movement trajectory and activity range of the human body, it is difficult to accurately distinguish the specific contact behaviors between different individuals, especially in a hospital environment with dense personnel and frequent interactions, the recognition effect is greatly reduced.

[0004] At the same time, image recognition technology is also widely used in the identification of cross-infection. However, due to the stability of image quality and the accuracy of recognition algorithms, this method is often prone to misjudgment or missed judgment in practical applications. For example, when the image quality is poor or there is occlusion, the image recognition algorithm may not be able to accurately identify the contact between medical staff and patients, resulting in missed or false positives of cross-infection risks.

[0005] In view of the shortcomings and limitations of the existing technology in cross-infection identification, this paper proposes a new hand hygiene cross-infection identification solution based on deep learning semantic segmentation, image classification and trajectory recognition. This solution aims to achieve accurate identification and tracking of personnel behavior in hospital departments through the comprehensive use of deep learning technology, so as to more effectively judge and warn of cross-infection risks, enhance the hand hygiene awareness of medical staff, and provide strong support for hospital department management. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition. The system uses deep learning semantic segmentation technology to analyze the surveillance video in the hospital department frame by frame, accurately extract the image information of medical staff and patients, and classify and identify them. At the same time, through deep learning image classification technology, the contact behavior between medical staff and patient bed units is further identified, including contact, physical examination, handshake, hug, sharing of medical equipment and other behaviors that may cause cross-infection. In addition, this solution will also combine trajectory recognition technology to track and record the movement trajectory of medical staff and patients in hospital departments in real time, so as to more comprehensively analyze their contact situation and cross-infection risks.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition, the system is based on deep learning semantic segmentation, image classification and trajectory recognition for recognition, including the following steps:

[0009] S1, forward the monitoring video stream of each intensive care unit to the embedded device through the real-time protocol, that is, select a streaming media server software, use the selected software to connect to the RTSP stream of the camera, set the forwarding parameters (including the target IP address, port and encoding format, etc.), and start the forwarding service;

[0010] S2, extract key frame images according to the ward video, use the stream processing function, and extract key frames through specific commands;

[0011] S3, label the standing medical personnel, bedside disinfection bottles, and bed units of bedridden patients in the picture as A, B, and C respectively, as semantic segmentation training data, which involves collecting key frames and labeling them using labeling tools;

[0012] S4, using the semantic segmentation map of the bed and a single person in contact as training data for whether the bed or patient is in contact, which involves determining the overlap between the person and the bed in the picture, and collecting such photos for labeling and preprocessing;

[0013] S5, using the picture composed of the person and the disinfectant as training data for determining whether to disinfect, involving determining the overlap between the person in the picture and the bedside disinfectant bottle, and collecting such pictures for marking and preprocessing;

[0014] S6. Train a deep learning YOLOv8 model for semantic segmentation and deploy it on an embedded device for semantic segmentation, which involves training the model using labeled data, installing dependencies, and writing and running inference scripts.

[0015] S7. Train a CNN model to identify contact behaviors. Each sequence has the following meanings: ABCB means that the medical staff touches the bedside disinfection bottle before and after touching the bed unit, which is marked as a normal event; ABC means that the medical staff does not wash their hands after touching the bed unit of the bedridden patient; ACB means that the medical staff does not wash their hands before touching the bed unit of the bedridden patient; AB means that the medical staff does not wash their hands before and after touching the bed unit of the bedridden patient. All three are warning events. Deploy and provide a call API on the embedded device, train the model, and deploy the model on the Raspberry Pi;

[0016] S8, grouping the key frames of multiple video streams into a batch, and passing them into the YOLOv8 model through batch processing to obtain the object segmentation result graph and JSON information graph, which involves grouping the key frames of multiple video streams into a batch of data and processing them to obtain the boundary information of semantic segmentation;

[0017] S9. According to the JSON information, determine the image information of people contacting the bed or the disinfectant, convert the corresponding combined semantic segmentation map into a fixed aspect ratio and send it to the three-category model to identify whether there is contact with the patient or the use of disinfectant, which involves finding the contact part according to the boundary information, preprocessing it, and sending it to the three-category model for classification;

[0018] S10, using the ByteTrack algorithm to track the trajectory of the person, ensuring that the judgment logic is for the same person, involving identifying the identity of the person in different photos and recording their location information;

[0019] S11. Real-time identification of whether medical staff have contacted patients or used disinfectants, determine whether they are the same person's previous and subsequent behaviors based on the trajectory tracking algorithm, use rules to judge cross-infection behaviors and record them, set up a behavior list for each person, record contact information and disinfectant usage, and judge cross-infection behaviors based on the rules.

[0020] Further, in step S1, the streaming media server software specifically includes FFmpeg or VLC, which is used to implement the forwarding function of the video stream.

[0021] Furthermore, in step S2, the extraction of the key frame image specifically uses the stream processing function of GStreamer or FFmpeg, and the extraction of the key frame is achieved through specific commands.

[0022] Further, in step S3, the annotation tools specifically include but are not limited to LabelMe and COCOAnnotator, which are used to annotate the characters, bed units, and bedside disinfection bottles in the picture.

[0023] Furthermore, the training data of the YOLOv8 model specifically includes labeled images of people, beds, and disinfectant alcohol, which are used to train the model for semantic segmentation.

[0024] Furthermore, the CNN model is designed as a small network structure, which is used to classify the output results into three categories: normal, contact with patients, and use of disinfectant.

[0025] Further, in step S8, the batch processing specifically includes forming the key frame information of n video streams into a list, and the YOLOv8 model processes n photos each time to improve processing efficiency.

[0026] Furthermore, the JSON information is used to determine image information of contact between people and beds, and people and disinfectants, and convert the corresponding combined semantic segmentation map into a fixed aspect ratio for subsequent three-category model processing; the ByteTrack algorithm is used to identify which people in different photos are the same person, and record their location information to achieve cross-time behavior tracking; the judgment of cross-infection behavior specifically includes setting up a behavior list for each person, recording contact information and disinfectant usage, and judging cross-infection behavior according to rules to achieve accurate cross-infection monitoring.

[0027] The beneficial effects of the present invention are:

[0028] The cross-infection identification scheme based on deep learning technology proposed in this invention is expected to overcome the shortcomings and limitations of existing technologies, improve the accuracy and efficiency of cross-infection identification, and provide real-time warnings to enhance the hand hygiene awareness of medical staff. This will provide more scientific and effective means and methods for hospital department management, help further reduce the risk of cross-infection, reduce the occurrence of nosocomial infections, and improve medical quality and patient satisfaction. At the same time, the successful implementation of this scheme will also provide strong support and guarantee for promoting the intelligent and information construction of hospitals.

[0029] Compared with the prior art, the beneficial effects of this technical solution are: 1) The present invention only collects existing video information, which is low-cost and requires little modification to the existing system, and only adds a video stream forwarding function. 2) The present invention can accurately identify the behavior of contacting patients, the movement trajectory of medical staff, and real-time identify hand hygiene warning events through the sequence of medical staff contacting disinfectants, thereby accurately identifying cross-infection behavior. 3) The present invention adopts embedded deployment and batch processing technology to achieve a response in seconds, so as to achieve real-time warning and correction.

[0030] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0032] Figure 1 It is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0034] Figure 1 The system flow chart of the present invention is shown in the figure. The cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition provided by the present invention includes the following steps:

[0035] S1, forward the monitoring video stream of each intensive care unit to the embedded device through the real-time protocol, that is, select a streaming media server software, use the selected software to connect to the RTSP stream of the camera, set the forwarding parameters (including the target IP address, port and encoding format, etc.), and start the forwarding service;

[0036] S2, extract key frame images according to the ward video, use the stream processing function, and extract key frames through specific commands;

[0037] S3, label the standing medical personnel, bedside disinfection bottles, and bed units of bedridden patients in the picture as A, B, and C respectively, as semantic segmentation training data, which involves collecting key frames and labeling them using labeling tools;

[0038] S4, using the semantic segmentation map of the bed and a single person in contact as training data for whether the bed or patient is in contact, which involves determining the overlap between the person and the bed in the picture, and collecting such photos for labeling and preprocessing;

[0039] S5, using the picture composed of the person and the disinfectant as training data for determining whether to disinfect, involving determining the overlap between the person in the picture and the bedside disinfectant bottle, and collecting such pictures for marking and preprocessing;

[0040] S6. Train a deep learning YOLOv8 model for semantic segmentation and deploy it on an embedded device for semantic segmentation, which involves training the model using labeled data, installing dependencies, and writing and running inference scripts.

[0041] S7. Train a CNN model to identify the contact sequence of ABC. ABCB, which means that the medical staff touches the bedside disinfection bottle before and after touching the bed unit, is marked as a normal event. ABC, which means that the medical staff does not wash their hands after touching the bed unit of the bedridden patient, ACB, which means that the medical staff does not wash their hands before touching the bed unit of the bedridden patient, and AB, which means that the medical staff does not wash their hands before and after touching the bed unit of the bedridden patient, are warning events. Deploy the model on the embedded device to provide a call API, train the model, and deploy the model on the Raspberry Pi;

[0042] S8, grouping the key frames of multiple video streams into a batch, and passing them into the YOLOv8 model through batch processing to obtain the object segmentation result graph and JSON information graph, which involves grouping the key frames of multiple video streams into a batch of data and processing them to obtain the boundary information of semantic segmentation;

[0043] S9. According to the JSON information, determine the image information of people contacting the bed or the disinfectant, convert the corresponding combined semantic segmentation map into a fixed aspect ratio and send it to the three-category model to identify whether there is contact with the patient or the use of disinfectant, which involves finding the contact part according to the boundary information, preprocessing it, and sending it to the three-category model for classification;

[0044] S10, using the ByteTrack algorithm to track the trajectory of the person, ensuring that the judgment logic is for the same person, involving identifying the identity of the person in different photos and recording their location information;

[0045] S11. Real-time identification of whether medical staff have contacted patients or used disinfectants, determine whether they are the same person's previous and subsequent behaviors based on the trajectory tracking algorithm, use rules to judge cross-infection behaviors and record them, set up a behavior list for each person, record contact information and disinfectant usage, and judge cross-infection behaviors based on the rules.

[0046] In this embodiment, the specific process is as follows:

[0047] 1. Forward the monitoring video streams of each intensive care unit to embedded devices such as Raspberry Pi through real-time protocols such as RTSP.

[0048] 1) Choose a streaming server software, such as FFmpeg or VLC;

[0049] 2) Use the selected software to connect to the camera's RTSP stream;

[0050] 3) Set forwarding parameters, including target IP address, port, and encoding format;

[0051] 4) Start forwarding service

[0052] "ffmpeg -i rtsp: / / camera_ip:port / stream -c copy-frtsp rtsp: / / server_ip:port / stream_name"

[0053] 2. Extract key frame images based on the ward video.

[0054] 1) Use the streaming function of GStreamer or FFmpeg;

[0055] 2) Use the following command of FFmpeg to extract key frames;

[0056] "ffmpeg -i rtsp: / / your_stream_url -vf select='eq(pict_type,PICT_TYPE_I)'-vsync vfr keyframes_%03d.jpg"

[0057] 3. Label the people, bed, and bedside disinfection bottle in the picture as semantic segmentation training data.

[0058] 1) Collect about 5,000 key frames obtained in step 2;

[0059] 2) Use tools such as LabelMe and COCOAnnotator to mark the people, beds, and disinfectant alcohol in the picture;

[0060] 4. The semantic segmentation map of the contact between the bed and a single person, that is, the image composed of the person image and the bed image, is used as training data for training whether the bed or the patient is in contact.

[0061] 1) Use the program to determine whether there are any people photos in the picture that overlap with the bed area;

[0062] 2) Collect about 4,000 such photos;

[0063] 3) Manually label these photos to identify which ones are real contacts and which ones are just overlaps;

[0064] 4) Cut and fill these photos and scale them to 256*256 size;

[0065] 5) Use these 256*256 size photos with contact or no contact mark as training data;

[0066] 5. Use the pictures composed of people and disinfectant as training data to determine whether to disinfect.

[0067] 1) Use the program to determine whether there are any people photos in the image that overlap with the disinfectant area;

[0068] 2) Collect about 4,000 such photos;

[0069] 3) Manually label these photos to identify which ones are real contacts and which ones are just overlaps;

[0070] 4) Cut and fill these photos and scale them to 256*256 size;

[0071] 5) Use these 256*256 size photos with contact or no contact mark as training data;

[0072] 6. Train models such as yolov8 for semantic segmentation, and deploy yolov8 on embedded devices such as Raspberry Pi for semantic segmentation.

[0073] 1) Use the data labeled in step 3 to train the yolov8 model;

[0074] 2) Install opencv, ultralytics and other dependencies on the Raspberry Pi;

[0075] 3) Write the inference running script;

[0076] 4) Run the script on the Raspberry Pi;

[0077] 7. Train a CNN model to identify the 3-category model of [normal, contact with patients, use of disinfectant].

[0078] 1) Divide the photos in step 4 and step 5 into three categories: [normal, contact with patients, use of disinfectant];

[0079] 2) Design commonly used small CNN networks;

[0080] 3) The training model performs three classifications on the output results;

[0081] 4) Deploy the model on the Raspberry Pi and provide an API for external calls;

[0082] 8. Group the key frames of multiple video streams into a batch, pass them into the yolov8 model through batch processing, and obtain the object segmentation result map and json information map.

[0083] 1) Put the keyword information of n video streams into a list as a batch of data;

[0084] 2) The yolov8 semantic segmentation model processes n photos each time;

[0085] 3) Boundary information of semantic segmentation of model data;

[0086] 9. After judging the image information of people contacting beds and people contacting disinfectants based on the JSON information, the corresponding combined semantic segmentation map is converted into a fixed aspect ratio and sent to the 3-classification model to identify whether contact with patients or use of disinfectants occurs.

[0087] 1) Based on the boundary information output by the model, find out the part where the person touches the bed and the part where the person touches the disinfectant;

[0088] 2) Store the above two types of contact parts in a new square area;

[0089] 3) Fill the remaining area with white;

[0090] 4) Scale the photo to 256*256;

[0091] 5) Pass the photos into the 3-classification model for classification;

[0092] 6) Determine whether there is contact based on the classification results;

[0093] The three-class model can be expressed as follows:

[0094] z=f(x;θ)

[0095] Where: x is the input data, θ is the model parameter, and f is the function of the entire neural network

[0096] z = [z1, z2, z3] is the original output (logits) of the model

[0097] Loss function: L(θ) = -Σ i Σ k y ik log(p(y=k|x i ))

[0098] Where: i represents the sample index, k represents the category, y ik is the one-hot encoding of the true label

[0099] predict:

[0100] For a new input x_new, the predicted category is:

[0101] y_pred=argmax p(y=k|x_new)

[0102] 10. Use the ByteTrack algorithm to track the trajectory of the person and ensure that the judgment logic is for the same person.

[0103] 1) Use ByteTrack to identify which people are the same person in different photos;

[0104] 2) Record the location information of the same person in different photos;

[0105] 11. When a medical staff member enters the ward, the semantic segmentation model identifies in real time whether his / her outline overlaps with the patient and the disinfectant. If so, the 3-classification model is used to determine whether he / she has contacted the patient and used disinfectant. The trajectory tracking algorithm is used to determine whether the previous and subsequent behaviors are those of the same person, and the rules are used to determine if the medical staff member did not use disinfectant before and after contacting the patient. The time and key images are recorded and a prompt is sent.

[0106] 1) Set up a list of behaviors for each person ByteTrack records;

[0107] 2) When the three-classification model identifies that a person is in contact with disinfectant or a person is in contact with a bed, the contact information is written into the corresponding person behavior list;

[0108] 3) Each time writing, determine whether the medical staff used disinfectant before and after contact with the same patient;

[0109] 4) When a medical staff member disappears from the video for more than 2 minutes, determine whether his historical behavior has cross-infection, and then delete his historical information.

[0110] The usage scenarios and methods of the technical solution provided by this patent may include character trajectory recognition in intensive care units, character information and screen information segmentation, patient contact and disinfectant use behavior recognition, as well as accurate identification of cross-infection behavior based on the above-mentioned multiple recognition combination rules.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition, characterized by: The system is based on deep learning semantic segmentation, image classification and trajectory recognition, and includes the following steps: S1, forwarding the surveillance video stream of each intensive care unit to the embedded device through the real-time protocol, that is, selecting a streaming media server software, using the selected software to connect to the RTSP stream of the camera, setting the forwarding parameters, and starting the forwarding service; S2, extract key frame images according to the ward video, use the stream processing function, and extract key frames through specific commands; S3, label the standing medical personnel, bedside disinfection bottles, and bed units of bedridden patients in the picture as A, B, and C respectively, as semantic segmentation training data, which involves collecting key frames and labeling them using labeling tools; S4, using the semantic segmentation map of the bed and a single person in contact as training data for whether the bed or patient is in contact, which involves determining the overlap between the person and the bed in the picture, and collecting such photos for labeling and preprocessing; S5, using the picture composed of the person and the disinfectant as training data for judging whether to disinfect, involving judging the overlap between the person in the picture and the disinfectant alcohol bottle at the bedside, and collecting such pictures for marking and preprocessing; S6. Train a deep learning YOLOv8 model for semantic segmentation and deploy it on an embedded device for semantic segmentation, which involves training the model using labeled data, installing dependencies, and writing and running inference scripts. S7. Train a CNN model to identify three types of contact behaviors, and record the subsequent behaviors as a combination of ABC, where ABCB, i.e., the medical staff touches the bedside disinfection bottle before and after touching the bed unit, is marked as a normal event, ABC, i.e., the medical staff does not wash their hands after touching the bed unit of the bedridden patient, ACB, i.e., the medical staff does not wash their hands before and after touching the bed unit of the bedridden patient, and AB, i.e., the medical staff does not wash their hands before and after touching the bed unit of the bedridden patient, and all three are warning events. Deploy and provide a call API on an embedded device, train the model, and deploy the model on a Raspberry Pi; S8, grouping the key frames of multiple video streams into a batch, and passing them into the YOLOv8 model through batch processing to obtain the object segmentation result graph and JSON information graph, which involves grouping the key frames of multiple video streams into a batch of data and processing them to obtain the boundary information of semantic segmentation; S9. According to the JSON information, determine the image information of people contacting the bed or the disinfectant, convert the corresponding combined semantic segmentation map into a fixed aspect ratio and send it to the three-category model to identify whether there is contact with the patient or the use of disinfectant, which involves finding the contact part according to the boundary information, preprocessing it, and sending it to the three-category model for classification; S10, using the ByteTrack algorithm to track the trajectory of the person, ensuring that the judgment logic is for the same person, involving identifying the identity of the person in different photos and recording their location information; S11. Real-time identification of whether medical staff have contacted patients or used disinfectants, determine whether they are the same person's previous and subsequent behaviors based on the trajectory tracking algorithm, use rules to judge cross-infection behaviors and record them, set up a behavior list for each person, record contact information and disinfectant usage, and judge cross-infection behaviors based on the rules.

2. The cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition according to claim 1, characterized in that: In step S1, the streaming media server software specifically includes FFmpeg or VLC, which is used to implement the forwarding function of the video stream.

3. The cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition according to claim 2, characterized in that: In step S2, the extraction of the key frame image specifically uses the stream processing function of GStreamer or FFmpeg, and the extraction of the key frame is achieved through specific commands.

4. The cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition according to claim 3, characterized in that: In step S3, the annotation tools specifically include but are not limited to LabelMe and COCOAnnotator, which are used to annotate the characters, bed, and bedside disinfection bottle in the picture.

5. The cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition according to claim 4, characterized in that: The training data of the YOLOv8 model specifically includes labeled images of people, beds, and bedside disinfection bottles, which are used to train the model for semantic segmentation.

6. The cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition according to claim 5, characterized in that: The CNN model is designed as a small network structure, which is used to classify the output results into three categories: normal, contact with patients, and use of disinfectant.

7. The cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition according to claim 6, characterized in that: In step S8, the batch processing specifically includes forming a list of key frame information of n video streams, and the YOLOv8 model processes n photos each time to improve processing efficiency.

8. The cross-infection image early warning system based on deep learning semantic segmentation, image classification and trajectory recognition according to claim 7, characterized in that: The JSON information is used to determine the image information of people in contact with beds and people with disinfectants, and convert the corresponding combined semantic segmentation map into a fixed aspect ratio for subsequent three-category model processing; the ByteTrack algorithm is used to identify which people in different photos are the same person and record their location information to achieve cross-time behavior tracking; the judgment of cross-infection behavior specifically includes setting a behavior list for each person, recording contact information and disinfectant usage, and judging cross-infection behavior according to rules to achieve accurate cross-infection monitoring.

Images