Patient anomaly detection method based on multi-video monitoring in medical environment

Abstract

The invention discloses a patient anomaly detection method based on multi-video monitoring in a medical environment. The method comprises the following steps: constructing a sample data set containing pictures and videos of abnormal behaviors; constructing an abnormal behavior detection model, enabling the abnormal behavior detection model to automatically recognize the marked key points of the human body, and recognizing whether an abnormal behavior occurs or not according to the dispersion change of the key points; the method comprises the following steps: constructing a personnel identification model, carrying out face identification on a patient with an abnormal behavior, finally carrying out real-time monitoring on multi-angle video monitoring data in a medical environment according to the established model, timely finding the abnormal behavior of the patient, and sending out an abnormal prompt. According to the method, by combining an advanced technology for improving human behavior recognition and a multi-modal data processing flow, anomaly detection of multi-video monitoring in a medical environment is realized, manual participation is reduced or even not needed, and effective technical support is provided for improving medical service quality and guaranteeing patient safety.

Description

Technical Field

[0001] The present invention relates to the technical field of video image detection, and in particular to a patient abnormality detection method based on multi-video monitoring in a medical environment. Background Art

[0002] Safety hazards and emergencies often occur in the medical environment, requiring the medical system to have efficient and accurate monitoring and management capabilities. Therefore, in the medical environment, how to ensure the safety of patients and respond to their needs in a timely manner has become the key to improving the quality of medical services.

[0003] Traditional medical monitoring mainly relies on direct observation and inspection by medical staff. This method has certain limitations: 1. Limited monitoring range: Medical staff cannot monitor multiple areas at the same time, resulting in monitoring blind spots. 2. Delayed response time: Manual inspections cannot achieve real-time monitoring, resulting in time delays in responding to emergencies. 3. Tight human resources: As the number of patients increases, the workload of medical staff increases, making it difficult to maintain efficient monitoring. 4. False alarms and missed alarms: Manual monitoring is susceptible to fatigue and distraction, resulting in false alarms and missed alarms. 5. Privacy protection issues: Manual monitoring may involve the risk of patient privacy leakage.

[0004] In recent years, the development of artificial intelligence technology, especially computer vision technology, has enabled intelligent video surveillance systems to automatically identify and analyze video content, improving the automation and intelligence level of monitoring. The resulting multi-video surveillance anomaly detection technology is increasingly used in medical environments. Multi-video surveillance technology integrates multiple cameras to achieve full coverage and continuous monitoring of medical areas, and can detect and handle abnormal events in a timely manner. The anomaly detection technology that matches this uses computer vision and artificial intelligence technology to conduct in-depth analysis of video data, automatically identify and alarm potential abnormal behaviors or conditions. The application of this technology not only improves the safety of the medical environment, but also reduces the workload of medical staff and improves the quality and efficiency of medical services.

[0005] However, the medical environment is a highly complex and dynamically changing place, involving many patients, medical staff and sophisticated medical equipment, and is full of variables in the diagnosis and treatment process. This complexity is not only reflected in the diversity of diseases and treatment methods, but also in the diversity of patient needs and the tight allocation of medical resources. The current multi-video surveillance technology is still superficial, and most of it can only achieve the fusion of multiple videos. For example, when an emergency occurs, the surveillance video of the location is retrieved and displayed. The specific video analysis still relies on manual completion. This method is inefficient and difficult to meet the needs of real-time monitoring. As a result, the problems existing in the above traditional manual monitoring methods can only be alleviated, but not truly solved.

[0006] In summary, although the application prospects of intelligent video surveillance systems in the medical field are broad and can significantly improve the safety and efficiency of medical services, they still face the following challenges when applied in the medical field and medical environment:

[0007] 1. Adaptability to complex environments: The medical environment is complex and changeable, including different lighting conditions, personnel flow, etc., which puts higher requirements on the adaptability of the monitoring system.

[0008] 2. Behavior recognition accuracy: It is necessary to accurately identify various abnormal behaviors, such as falls, sudden illnesses, etc., to reduce false positives and missed positives.

[0009] 3. Real-time processing capability: The system needs to be able to process video streams in real time in order to respond quickly to emergencies.

[0010] 4. Data privacy protection: When collecting and analyzing video data, it is necessary to ensure that patient privacy is properly protected.

[0011] In order to achieve deeper promotion and application of intelligent video surveillance systems in the medical field, further technological breakthroughs and innovations are still needed in multi-video surveillance anomaly detection technology. Summary of the invention

[0012] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0013] One object of the present invention is to provide a patient abnormality detection method based on multi-video surveillance in a medical environment, which improves the accuracy and real-time performance of identifying patient abnormalities when facing complex scenarios, while reducing or even eliminating manual intervention to alleviate human resource pressure and protect patient privacy.

[0014] In order to achieve the above-mentioned object, the present invention provides a method for detecting abnormalities of patients based on multi-video monitoring in a medical environment, comprising:

[0015] S1. Based on multi-angle video surveillance data in a medical environment, collect pictures and videos containing abnormal behaviors, build an abnormal sample dataset, and divide the abnormal sample dataset into a training dataset and a test dataset;

[0016] S2. Label the human body in the training data set and mark the key points that can reflect the posture and motion state of the human body;

[0017] S3, construct an abnormal behavior detection model, so that the abnormal behavior detection model automatically identifies the key points, and designs a discrete metric classifier to process the training set data, identify and track the key points in the picture, record the discreteness change of each key point in the x, y, z coordinates, generate an abnormal label, and complete the training optimization of the abnormal behavior detection model;

[0018] S4. Use the trained abnormal behavior detection model to detect the test data set, identify human behavior, and automatically determine whether it is abnormal;

[0019] S5. After determining the abnormal label, perform face image detection and collection, and process the face image to improve the image quality;

[0020] S6. Build a personnel recognition model, detect the processed face image data, obtain the face feature vector, and complete the training optimization of the personnel recognition model;

[0021] S7, establishing a facial feature database of patients;

[0022] S8. Load the trained abnormal behavior detection model and personnel identification model, process the real-time video stream, obtain the facial feature vector of the person who has abnormal behavior, compare it with the facial features in the database to identify the person, obtain the person information, and issue an abnormal prompt message.

[0023] A further preferred technical solution of the present invention is that in step S1, after collecting pictures and videos containing abnormal behaviors, the images are scaled, cropped and normalized to perform data enhancement.

[0024] Preferably, in step S2, after the human body in the training data set is labeled, conversion and enhancement operations of key point annotations are performed.

[0025] Preferably, in step S3, an abnormal behavior detection model is constructed so that the abnormal behavior detection model automatically identifies the key points, and the specific method is:

[0026] Use convolutional neural networks for deep learning. Use training data sets to train convolutional neural networks. Convolutional neural networks use mean filtering algorithms, edge processing, texture processing, and shape processing to process images and videos in the training data sets and identify key points.

[0027] Among them, the mean filtering algorithm replaces the value of the pixel by calculating the average value of the neighborhood around the pixel, which is expressed as:

[0028]

[0029] Where N(x,y) is the neighborhood of pixel (x,y), N is the number of pixels in the neighborhood, and I(i,j) is the original value of the pixel in the neighborhood;

[0030] Edge processing detects changes by calculating the first or second order derivatives of the image grayscale values;

[0031] Texture processing extracts texture features by analyzing the grayscale value, spatial distribution and arrangement of pixels in an image;

[0032] Shape processing extracts shape features by analyzing the outline, boundary or internal structure of an object.

[0033] Preferably, in step S3, a discrete metric classifier is designed to process the training set data, identify and track the key points in the picture, record the discreteness change of each key point in the x, y, z coordinates, and generate an abnormal label; the specific method is:

[0034] Parse the image or video frame by frame, use convolutional neural network to identify and track key points on the human body, and transform the coordinates of these key points from the image coordinate system to the three-dimensional space coordinate system;

[0035] Calculate the change in key point coordinates between adjacent frames, i.e., the change in discreteness;

[0036] The discreteness change is used as a feature vector to train a discreteness classifier;

[0037] Store the feature vector and coordinate values ​​of abnormal behavior in a CSV file;

[0038] Define key point acceleration, moving direction angle and relative distance rules for marking and storing key point data;

[0039] By traversing the key point data, the relevant metrics of each key point are calculated. If the rules are met, an abnormal label is generated and stored in a CSV file together with the key point.

[0040] The discreteness classifier is defined as:

[0041] Assume d 1 ,d 2 ,…d k There are k pattern classes and pattern samples Y i (i=I,…,N) is an n-dimensional real vector, and the average vector of the i-th class sample is The mean vector of all samples is Use X $ , X ％ Respectively represent the inter-class scatter matrix and intra-class scatter matrix of pattern samples:

[0042]

[0043] The overall discreteness classification criterion U(w) is as follows:

[0044]

[0045] When U(w) is the largest, w is used as the projection direction for classification to transform and classify the samples.

[0046] Preferably, the specific method for defining the rule of the key point acceleration and the moving direction angle is:

[0047] (1) Acceleration calculation: Assume that the key point position data of three consecutive frames is (P 452 ,P 451 ,P 4 ), where P 4 =(x 4 ,y 4 ) represents the coordinates of the key point at the tth frame, and Δt is the time interval between two frames; then:

[0048]

[0049] (2) Calculation of the moving direction angle: Assume that the key point position data of two consecutive frames is (P 451 ,P 4 ), calculate the moving direction vector and find the angle between the two vectors;

[0050] The calculation formula for the moving direction vector is:

[0051]

[0052] The calculation formula of the moving direction angle is:

[0053]

[0054] As an advantage, it also includes the steps of:

[0055] S9. Detect and collect voice information of personnel to be monitored, and establish an abnormal voice information database containing key words; specifically:

[0056] Use the pyaudio library to record audio. During the audio recording process, read the audio frames in a loop and store them in a list. Calculate the short-time Fourier transform of the audio and obtain the spectrum. Accept the audio signal and sampling rate parameters and return the frequency, time and spectrum density arrays. The spectrum density array is obtained by calculating the square of the amplitude or energy of the short-time Fourier transform result at each time-frequency point. Store the mean of the spectrum density as a feature.

[0057] The formula for short-time Fourier transform is:

[0058]

[0059] Where X(t,f) is the short-time Fourier transform result at time t and frequency f, x(n) is the input audio signal, w(n) is the window function used to perform windowing on the signal, and e 5Q2RSN is the complex exponential term of the Fourier transform;

[0060] S10, obtaining sound features in real time and matching them with features in a database to identify abnormal speech; the specific method is:

[0061] When the key words are obtained, the system automatically obtains the voice of the person with abnormal behavior, calculates the short-time Fourier transform of the audio, and obtains the spectrum diagram; accepts the audio signal and sampling rate parameters, and returns the frequency, time, and spectrum density arrays; takes the mean of the obtained spectrum density as the sound feature, and converts it into a numpy array for easy comparison; uses the corresponding numpy function to calculate the Pearson correlation between the extracted audio features and the features stored in the database, and sets a threshold. If the similarity between the extracted features and the features stored in the database exceeds this threshold, it is considered that a matching individual has been found;

[0062] S11. After identifying an individual with an abnormal voice match, an abnormal prompt message is issued.

[0063] Another aspect of the present invention provides a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions enabling a computer to execute the above-mentioned patient abnormality detection method based on multi-video monitoring in a medical environment.

[0064] Yet another aspect of the present invention provides an electronic device comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, and the processor calls the logic instructions in the memory to execute the above-mentioned patient abnormality detection method based on multi-video monitoring in a medical environment.

[0065] On the other hand, the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer executes the above-mentioned patient abnormality detection method based on multi-video monitoring in a medical environment.

[0066] Beneficial effects: Compared with the prior art, the present invention uses key point acquisition to identify and track multiple key target points in the picture. These target points cover the main joints and parts of the human body, providing a rich data basis for the identification of abnormal behavior. The present invention can quickly respond to changes in the video and identify abnormal behavior in a timely manner. At the same time, it can continuously optimize and expand by adding new abnormal behavior samples and labels, and improve the ability to identify different types of abnormal behaviors.

[0067] The present invention uses facial image information collected by ordinary cameras for analysis, does not require additional equipment, is applicable to various monitoring scenarios, including indoors and outdoors, different lighting conditions, etc., and has low cost. The present invention adopts a multi-video monitoring method, which can integrate various monitoring on one page and detect abnormal behaviors occurring in each location in real time. The present invention can adopt the H.265 encoding standard, which greatly improves the compression efficiency, ensures data quality, reduces the bandwidth required for data transmission, alleviates network congestion, and reduces the phenomenon of jamming. In addition, when multiple videos are detected in real time, a multi-threaded method is adopted, ffmpeg is used to support multi-threaded decoding, multiple threads are used to solve the problem of resource competition, and synchronization mechanisms such as locks, semaphores, and conditional variables are used to ensure data consistency and collaboration between threads.

[0068] The solution of the present invention is applicable to various monitoring systems, including airports, railway stations, schools, rehabilitation training and other places, and has broad application prospects and market potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The present invention is a flowchart of a method for detecting abnormalities in patients based on multi-video surveillance in a medical environment. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0071] Combine the following Figure 1 The present invention describes a patient abnormality detection method based on multi-video monitoring in a medical environment.

[0072] Embodiment 1: This embodiment provides a method for detecting patient abnormalities based on multi-video monitoring in a medical environment.

[0073] This embodiment is a multi-video monitoring anomaly detection method in a medical environment. An artificial intelligence detection module is created to detect abnormal behavior in real time, aiming to be able to detect and accurately identify abnormal behavior in real time. This module can be easily inherited in the complex monitoring system of the medical environment. The strategy of multi-video dynamic monitoring acquisition is adopted, and by deploying a high-density, high-definition monitoring camera network, all-round and no-dead-angle coverage of the medical area is achieved. This layout not only ensures the clarity and detail of the monitoring screen, but also greatly improves the probability of abnormal behavior being captured and identified. In the actual detection process, this module can analyze the video streams from multiple cameras in real time and perform intelligent analysis on its content. By comparing the behavior of the characters in the video, abnormal behavior patterns, such as falling, can be quickly identified, and an alarm is issued at the first time through dual-modal recognition, and it is pointed out who this person is.

[0074] Specifically, the method of this embodiment includes the following steps:

[0075] (1) Based on multi-angle video surveillance in complex medical environments, abnormal behavior data is collected, abnormal sample data sets (abnormal behavior pictures and videos) are collected, and they are divided into training sets and test sets to provide data support for subsequent model training and testing. This step is crucial in machine learning. First, by dividing the data set into training sets and test sets, the training model can learn the coordinates of each target point on the training set to form features, and then evaluate it on an independent test set. This separation operation ensures that the model can show generalization capabilities when encountering unknown data, rather than just letting the machine remember the training data. It also avoids overfitting. By separating the data set, the changes in the model during the training process can be monitored, and timely adjustments can be made to prevent overfitting and improve the stability of the model.

[0076] Data collection and segmentation: Different people are required to perform normal and abnormal operations to continuously collect video data. You can set a scheduled task or use a trigger to automatically start recording. Extract valid clips from the original video and remove useless or repeated parts. The example uses a python script to crop and filter video data.

[0077] (2) Use key point annotation and discreteness classifier to process the training set data.

[0078] First, we annotate the training sample data and mark 33 key points on the human body (including key positions such as the head, neck, shoulders, elbows, wrists, fingers, hips, knees, ankles, etc., which can fully reflect the posture and movement state of the human body). Based on these images and videos with annotated key points, we train a deep convolutional neural network model so that the model can automatically identify the 33 key points. The key points can be annotated using the COCO human posture estimation dataset, or you can collect and annotate them manually. During the data collection process, each image is processed. Here, the mean filter algorithm, edge processing, texture processing, and shape processing are used to improve the generalization ability of the model.

[0079] The mean filter algorithm is used in the video, which is a simple image noise reduction technique that replaces the value of the pixel by calculating the average value of the neighborhood around the pixel. This method can reduce the random noise of the image, expressed as:

[0080]

[0081] Where N(x,y) is the neighborhood of pixel (x,y), N is the number of pixels in the neighborhood, and I(i,j) is the original value of the pixel in the neighborhood;

[0082] Edge processing: Detecting changes by calculating the first-order or second-order derivatives of the image grayscale value can help identify the contours of the human body and the boundaries of various parts.

[0083] Texture processing: extract texture features by analyzing the grayscale value, spatial distribution and arrangement rules of pixels in the image.

[0084] Shape processing: extracting shape features by analyzing the outline, boundary or internal structure of an object.

[0085] Then, a new discrete metric classifier is designed based on the sample key point feature data to process the training set data, analyze the video frames one by one, identify and track the 33 key target points in the picture, record the discrete changes of each target point in the x, y, and z coordinates, and generate abnormal labels based on these changes and store them in a CSV file.

[0086] The discreteness classifier is particularly effective in processing data sets with multiple features, and can identify and distinguish data points of different categories. First, you need to read the video data frame by frame, and for each frame, identify the key points (33 key points are used here, which are the joints, head, hands, etc. of the human body). Because it is a 3D model, the (x, y, z) coordinates of each key point will be obtained. Because there are 33 key points, one key point is a list, and 33 key points will form a list of lists (including a list of 33 key points). These key points are converted from the image coordinate system to the three-dimensional space coordinate system (x, y, z). The discreteness change of the three-dimensional coordinates of each frame of the image can be achieved by calculating the position difference of the key points between adjacent frames. This discreteness change is used as a feature vector to train the maximum discreteness classifier. The feature vector and coordinate values ​​of the abnormal behavior are stored in a CSV file to serve as the basis for the following recognition.

[0087] The discreteness classifier is defined as:

[0088] Assume d 1 ,d 2 ,…d k There are k pattern classes and pattern samples Y i (i=I,…,N) is an n-dimensional real vector, and the average vector of the i-th class sample is The mean vector of all samples is Use X $ , X ％ Respectively represent the inter-class scatter matrix and intra-class scatter matrix of pattern samples:

[0089]

[0090] The overall discreteness classification criterion U(w) is as follows:

[0091]

[0092] When U(w) is the largest, w is used as the projection direction for classification to transform and classify the samples.

[0093] Define corresponding rules, mark and store key point data. For example: if the acceleration of a key point exceeds the normal range, it may indicate sudden violent movement or abnormal behavior (use the speed data of multiple consecutive frames to calculate the acceleration and compare it with the preset acceleration threshold); if the moving direction of the key point changes significantly in a short period of time, it may indicate unnatural movement or abnormal behavior (calculate the angle of the moving direction of the key point in consecutive frames, and compare it with the preset angle change threshold and retrograde); if the relative distance between two key points exceeds the normal range, it indicates an abnormal body posture (calculate the distance between the two key points in each frame and compare it with the preset distance range based on the normal posture). These rules are used as encoding as program logic. By traversing the key point data, the relevant metrics of each key point are calculated (these rules). If the rules are met, an abnormal label is generated and stored in a CSV file together with the key point. This step is for the comparison of abnormal behavior monitoring later.

[0094] The specific method for defining the rules of key point acceleration and moving direction angle is:

[0095] Record the changes of the key target points in x, y, and z coordinates, and record the coordinate position TX of a key point i,4 ,Y i,4 ,Z i, 4 V, t is the time point, i is the key point; then record the coordinate change ΔX i,4 =X i,4W1 -X i,4 , calculate the change of coordinates between adjacent time points;

[0096] (a) Acceleration calculation: Assume that the key point position data of three consecutive frames is (P 452 ,P 451 ,P 4 ), where P 4 =(x 4 ,y 4 ) represents the coordinates of the key point at the tth frame, and Δt is the time interval between two frames; then:

[0097]

[0098] (b) Calculation of the moving direction angle: Assume that the key point position data of two consecutive frames is (P 451 ,P 4 ), calculate the moving direction vector and find the angle between the two vectors;

[0099] The calculation formula for the moving direction vector is:

[0100]

[0101] The calculation formula of the moving direction angle is:

[0102]

[0103] (3) Load the trained model and analyze the surveillance video in real time. By comparing the real-time target point coordinate changes with the abnormal labels in the CSV, the system can automatically identify and judge abnormal behavior in the video.

[0104] Initialize the posture estimation module, use pandas to read the CSV file, read the known abnormal behaviors and their corresponding features, which are the key point data. Read the video data from the monitoring in real time, process it frame by frame, read the coordinates of the key points, calculate the features (the rules written above), compare them with the data in the CSV file, and automatically predict the label.

[0105] (4) The same key point detection and classification method as mentioned above is used to identify human behavior on the test samples, and automatically determine whether it is abnormal. After the abnormal label is determined, face image detection and collection are performed.

[0106] (5) Obtain facial image data and perform bit plane segmentation. Improve the contrast and brightness of the facial image data and remove noise through histogram equalization, gamma correction, contrast stretching, and Gaussian filtering. Save the processed facial image data as a numpy array.

[0107] (6) Use a convolutional neural network to detect the numpy array, obtain the four coordinate values ​​and feature vectors of each face area, calculate the total feature vector of the image and store all the coordinate values ​​and feature vectors.

[0108] (7) Compare the feature vector of this face with the face features stored in the database. Use the discrete classifier method designed in step (4) to first perform face identification feature representation. For each newly extracted feature vector, it will be compared with the feature vector stored in the database. The Euclidean distance between the new feature vector and each feature vector in the database is calculated. A threshold is set to determine whether the two feature vectors are close enough. Find the feature vector with the smallest distance. If this distance is less than the tolerance, a matching face is found.

[0109] (8) At the same time, the voice information of the personnel who need to be observed is detected and collected, and key words such as "help" and an abnormal voice information database are established. The process of abnormal voice collection is as follows: capture the audio of the personnel's voice, and in the process of recording the audio, loop and read the audio frames (1024 samples per frame) and store them in a list. Calculate the short-time Fourier transform (STFT) of the audio and obtain the spectrum diagram. It accepts parameters such as audio signal and sampling rate, and returns frequency, time and spectrum density arrays. The spectrum density array is obtained by calculating the square of the amplitude (or energy) of the STFT result at each time-frequency point. Here, the mean of the spectrum density is stored as a feature. The formula for short-time Fourier transform is:

[0110]

[0111] Where X(t,f) is the short-time Fourier transform result at time t and frequency f, x(n) is the input audio signal, w(n) is the window function used to perform windowing on the signal, and e 5Q2RSN is the complex exponential term of the Fourier transform.

[0112] (9) After the acquisition is completed, the sound recognition can be performed. First, when the key words such as "help" are obtained, the system automatically obtains the voice of the abnormal behavior person, calculates the short-time Fourier transform (STFT) of the audio, and obtains the spectrum diagram. It accepts parameters such as audio signal and sampling rate, and returns the frequency, time and spectrum density array. The mean of the spectrum density is obtained as the sound feature (this feature is a one-dimensional array, which represents the average energy distribution of the audio at different frequencies), and it is converted into a numpy array for easy comparison. The corresponding function of numpy is used to calculate the Pearson correlation book between the extracted audio features and the features stored in the database. This is a coefficient between [-1,1] (indicating the degree of linear correlation between the two features). A threshold is set. If the similarity between the extracted features and the features stored in the database exceeds this threshold, it is considered that a matching individual has been found.

[0113] Through the detection and identification of abnormal information in the image and sound bimodal in the multi-angle video surveillance in the above complex medical environment, the corresponding personnel information can be determined more accurately, and corresponding prompt information can be issued to provide auxiliary management for medical staff. This method combines the advanced technology of improving human behavior recognition and the multimodal data processing process to achieve abnormal detection of multiple video surveillance in the medical environment, providing effective technical support for improving the quality of medical services and ensuring patient safety.

[0114] Embodiment 2: This embodiment provides the above-mentioned abnormality detection method of the embodiment, and the hardware system that needs to be built includes cameras, monitoring equipment, network cables, switches, etc. The main parts are described as follows:

[0115] The monitoring equipment used for the test is Hikvision surveillance camera, which has 300-degree infrared recording and night vision functions and can be used for remote monitoring.

[0116] Initialization of surveillance cameras: For newly purchased cameras, initialization is required. According to the official website of the camera, follow the corresponding steps to reset the IP and password, and set up the corresponding RTSP streaming service. Use the cv2 library of Python to realize the real-time video playback function. If the corresponding video can be displayed, you can proceed to the next step.

[0117] Preprocessing of surveillance cameras: Use FFmpeg to convert RTSP streams, and explicitly specify TCP as the transport layer protocol through the RTSP protocol to ensure the reliability and stability of the video stream. Use the libx264 encoder to perform H.264 advanced video encoding on the video stream, and add the ACC encoder to encode the audio, output the video in FLV format, and finally send the video stream to the specified RTMP server.

[0118] Multi-camera processing: Use a multi-threaded module to create a queue. Thread A reads each frame from the video stream through the rtsp protocol and puts it into the queue. Thread B takes the photo out of the queue and displays it after processing. If thread A finds that there are two photos in the queue, it proves that thread B's reading speed cannot keep up with thread A. Then thread A will actively delete the old pictures in the queue and replace them with new ones. Therefore, through the multi-threaded method, the reading speed of thread A is always unaffected by thread B, preventing the cache area of ​​the network camera from being too full; thread A updates the pictures in the queue, so that thread B always reads the latest pictures, reducing latency.

[0119] In terms of hardware, a load balancer (nginx) can be deployed to distribute user requests to multiple processing nodes, thereby reducing the latency of the camera. There are two main ways to deploy a load balancer: direct mode and bypass mode. The direct mode connects the load balancer directly to the server, while the bypass mode places the load balancer next to the core switch of the network.

[0120] On the backend, use distributed cache technology (redis) (refers to a cache mechanism that separates the application system and cache components, so that multiple application systems can share a set of cache data. Its characteristics are shared cache services and cluster deployment, providing a highly available operating environment for the cache system, as well as a cache-sharing program operating mechanism)

[0121] Surveillance video communication technology: Surveillance video communication technology plays a vital role. This technology ensures real-time transmission, efficient processing and stable storage of video data, thereby supporting high-definition face capture without blind spots. When transmitting video data, efficient encoding and compression are performed to reduce bandwidth usage and latency.

[0122] In response to the complex and ever-changing monitoring needs in the medical environment, we have fully considered the differences in different scenarios and are using cameras with various resolutions and shooting angles. These cameras are deployed in key locations within the medical area to ensure high-definition monitoring coverage within a wide field of view and maintain the ability to capture changes in details within the video screen. Even at a long monitoring distance, it can maintain efficient and stable detection performance, providing a guarantee for abnormal behavior identification.

[0123] Embodiment 3: This embodiment provides a non-transitory computer-readable storage medium on which computer instructions are stored. The computer instructions enable a computer to execute the patient abnormality detection method based on multi-video monitoring in a medical environment of Embodiment 1.

[0124] Embodiment 4: This embodiment provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute the patient abnormality detection method based on multi-video monitoring in a medical environment of embodiment 1.

[0125] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0126] Embodiment 5: This embodiment provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the patient abnormality detection method based on multi-video monitoring in a medical environment of Embodiment 1.

[0127] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormalities in patients based on multi-video surveillance in a medical environment, characterized in that: include: S1. Based on multi-angle video surveillance data in a medical environment, collect pictures and videos containing abnormal behaviors, build an abnormal sample dataset, and divide the abnormal sample dataset into a training dataset and a test dataset; S2. Label the human body in the training data set and mark the key points that can reflect the posture and motion state of the human body; S3, construct an abnormal behavior detection model, so that the abnormal behavior detection model automatically identifies the key points, and designs a discrete metric classifier to process the training set data, identify and track the key points in the picture, record the discreteness change of each key point in the x, y, z coordinates, generate an abnormal label, and complete the training optimization of the abnormal behavior detection model; S4. Use the trained abnormal behavior detection model to detect the test data set, identify human behavior, and automatically determine whether it is abnormal; S5. After determining the abnormal label, perform face image detection and collection, and process the face image to improve the image quality; S6. Build a personnel recognition model, detect the processed face image data, obtain the face feature vector, and complete the training optimization of the personnel recognition model; S7, establishing a facial feature database of patients; S8. Load the trained abnormal behavior detection model and personnel identification model, process the real-time video stream, obtain the facial feature vector of the person who has abnormal behavior, compare it with the facial features in the database to identify the person, obtain the person information, and issue an abnormal prompt message.

2. The method for detecting abnormalities of patients based on multi-video surveillance in a medical environment according to claim 1, characterized in that: In step S1, after collecting pictures and videos containing abnormal behaviors, the images are scaled, cropped, and normalized to perform data enhancement.

3. The method for detecting abnormalities of patients based on multi-video surveillance in a medical environment according to claim 1, characterized in that: In step S2, after the human body in the training data set is labeled, the key point annotations are converted and enhanced.

4. The method for detecting abnormalities of patients based on multi-video surveillance in a medical environment according to claim 1, characterized in that: In step S3, an abnormal behavior detection model is constructed so that the abnormal behavior detection model automatically identifies the key points. The specific method is as follows: Use convolutional neural networks for deep learning. Use training data sets to train convolutional neural networks. Convolutional neural networks use mean filtering algorithms, edge processing, texture processing, and shape processing to process images and videos in the training data sets and identify key points. Among them, the mean filtering algorithm replaces the value of the pixel by calculating the average value of the neighborhood around the pixel, which is expressed as: Where N(x,y) is the neighborhood of pixel (x,y), N is the number of pixels in the neighborhood, and I(i,j) is the original value of the pixel in the neighborhood; Edge processing detects changes by calculating the first or second order derivatives of the image grayscale values; Texture processing extracts texture features by analyzing the grayscale value, spatial distribution and arrangement of pixels in an image; Shape processing extracts shape features by analyzing the outline, boundary or internal structure of an object.

5. The method for detecting abnormalities of patients based on multi-video surveillance in a medical environment according to claim 4, characterized in that: In step S3, a discrete metric classifier is designed to process the training set data, identify and track the key points in the picture, record the discrete changes of each key point in the x, y, and z coordinates, and generate anomaly labels; the specific method is: Parse the image or video frame by frame, use convolutional neural network to identify and track key points on the human body, and transform the coordinates of these key points from the image coordinate system to the three-dimensional space coordinate system; Calculate the change in key point coordinates between adjacent frames, i.e., the change in discreteness; The discreteness change is used as a feature vector to train a discreteness classifier; Store the feature vector and coordinate values ​​of abnormal behavior in a CSV file; Define key point acceleration, moving direction angle and relative distance rules for marking and storing key point data; By traversing the key point data, the relevant metrics of each key point are calculated. If the rules are met, an abnormal label is generated and stored in a CSV file together with the key point. The discreteness classifier is defined as: Let d1, d2, … d k There are k pattern classes and pattern samples Y i (i=I,…,N) is an n-dimensional real vector, and the average vector of the i-th class sample is The mean vector of all samples is Use X $ , X ％ Respectively represent the inter-class scatter matrix and intra-class scatter matrix of pattern samples: The overall discreteness classification criterion U(w) is as follows: When U(w) is the largest, w is used as the projection direction for classification to transform and classify the samples.

6. The method for detecting abnormalities of patients based on multi-video surveillance in a medical environment according to claim 5, characterized in that: The specific method of defining the rules of the key point acceleration and the moving direction angle is: (1) Acceleration calculation: Assume that the key point position data of three consecutive frames is (P 782 ,P 781 ,P7), where P7=(x7,y7) represents the coordinates of the key point at the tth frame, and Δt is the time interval between two frames; then: (2) Calculation of the moving direction angle: Assume that the key point position data of two consecutive frames is (P 781 ,P7), calculate the moving direction vector and find the angle between the two vectors; The calculation formula for the moving direction vector is: The calculation formula of the moving direction angle is:

7. The method for detecting abnormality of a patient based on multi-video surveillance in a medical environment according to claim 5, characterized in that: Also includes the steps: S9. Detect and collect voice information of personnel to be monitored, and establish an abnormal voice information database containing key words; specifically: Use the pyaudio library to record audio. During the audio recording process, read the audio frames in a loop and store them in a list. Calculate the short-time Fourier transform of the audio and obtain the spectrum. Accept the audio signal and sampling rate parameters and return the frequency, time and spectrum density arrays. The spectrum density array is obtained by calculating the square of the amplitude or energy of the short-time Fourier transform result at each time-frequency point. Store the mean of the spectrum density as a feature. The formula for short-time Fourier transform is: Where X(t,f) is the short-time Fourier transform result at time t and frequency f, x(n) is the input audio signal, w(n) is the window function used to perform windowing on the signal, and e 8T2UVQ is the complex exponential term of the Fourier transform; S10, obtaining sound features in real time and matching them with features in a database to identify abnormal speech; the specific method is: When the key words are obtained, the system automatically obtains the voice of the person with abnormal behavior, calculates the short-time Fourier transform of the audio, and obtains the spectrum diagram; accepts the audio signal and sampling rate parameters, and returns the frequency, time, and spectrum density arrays; takes the mean of the obtained spectrum density as the sound feature, and converts it into a numpy array for easy comparison; uses the corresponding numpy function to calculate the Pearson correlation between the extracted audio features and the features stored in the database, and sets a threshold. If the similarity between the extracted features and the features stored in the database exceeds this threshold, it is considered that a matching individual has been found; S11. After identifying an individual with an abnormal voice match, an abnormal prompt message is issued.

8. A non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions enabling a computer to execute the patient abnormality detection method based on multi-video monitoring in a medical environment as described in any one of claims 1-7.

9. An electronic device, comprising: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, and the processor calls the logic instructions in the memory to execute the patient abnormality detection method based on multi-video monitoring in a medical environment as described in any one of claims 1-7.

10. A computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer executes the patient abnormality detection method based on multi-video monitoring in a medical environment as described in any one of claims 1 to 7.

Images