Subway tumble detection method and system based on cloud edge collaboration and X-CLIP model

Abstract

The invention provides a subway tumble detection method and system based on cloud edge collaboration and an X-CLIP model, and relates to the technical field of subway tumble detection, and the method comprises the steps: obtaining a real-time video stream in a subway, and analyzing the video stream data into frame image data and video stream data; inputting the frame image data into the YOLOv8n-c3xghost neural network model to obtain all human body target area detection frames; performing identification detection on the human body target area detection frame through a BoT-SORT target tracking algorithm to obtain candidate images of abnormal behaviors; inputting the candidate image of the abnormal behavior into an X-CLIP model, outputting an abnormal region fragment classification result of the candidate image, identifying an abnormal region fragment again, determining the falling behavior according to the situation that the falling probability is greater than a set threshold value, outputting a position coordinate where the falling behavior occurs and a corresponding original image, and giving an alarm; wherein the model is deployed at the cloud end and trained, the model sinks to the edge side after being compressed and packaged, and real-time video frame detection is realized at the edge side.

Description

Technical Field

[0001] The present disclosure relates to the technical field of subway fall detection, and specifically to a subway fall detection method and system based on cloud-edge collaboration and an X-CLIP model. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In the subway operation scenario, timely detection of falls in the subway is conducive to the rapid handling of emergencies, timely arrival at the scene to provide necessary services, prevention of secondary disasters, ensuring the safety of passengers, reducing the impact on passengers' travel, and providing a safer and more orderly riding environment and improving service quality. Therefore, it is necessary to build an automatic fall monitoring system for the subway, use machine vision to monitor the trial code stream, issue warnings when abnormalities occur, and push relevant information to operators.

[0004] The application of machine vision technology in the field of fall monitoring is developing rapidly. Its core methods include real-time video analysis based on the YOLO algorithm, wearable device data analysis, etc. By analyzing video streams, sensor data or combining multiple data sources, it can detect and warn of fall events in real time, thereby improving safety and response speed. However, existing technologies still face some challenges, such as complex subway scenes, demand for computing resources, high equipment costs and maintenance complexity. How to improve detection accuracy, reduce operating costs and improve robustness is the current focus. Adjust the complexity of the network structure, the challenge of parameter tuning, reduce the demand for computing resources while maintaining high performance, avoid overfitting, and improve the real-time and deployment efficiency of the model on limited resource devices. In addition, it is necessary to continuously explore new technologies and methods to cope with emerging new challenges and needs. The traditional yolov8 is prone to missed reports and false positives, and the single xclip cannot accurately lock the target coordinates when abnormal behavior occurs. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a subway fall detection method and system based on cloud-edge collaboration and X-CLIP model, proposes a fall detection method of X-CLIP-YOLO-BoT-SORT architecture, applies X-CLIP model to realize video classification, proposes an optimization method based on YOLO improved YOLOv8n-ghostlscd to realize pedestrian detection, uses BoT-SORT algorithm to realize pedestrian tracking, and applies pruning-distillation-quantization method for model compression, proposes cloud-edge collaborative working architecture, designs "problem feedback-model correction" method, and solves the problems of excessively high underreporting rate and false alarm rate in engineering applications.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions:

[0007] The subway fall detection method based on cloud-edge collaboration and X-CLIP model includes:

[0008] Obtain real-time video stream in the subway and parse the video stream data into frame image data;

[0009] Input the frame image data into the YOLOv8n-c3xghost neural network model to obtain all human target area detection frames; identify and detect the human target area detection frames through the BoT-SORT target tracking algorithm to obtain candidate images of abnormal behavior;

[0010] Input the candidate image of the abnormal behavior into the X-CLIP model, output the classification result of the abnormal region fragment of the candidate image, identify the abnormal region fragment again, determine the fall behavior according to the probability of fall being greater than the set threshold, output the location coordinates of the fall behavior and the corresponding original image, and issue an alarm;

[0011] Among them, the c3xghost module of the backbone network of the YOLOv8n-c3xghost neural network model is divided into two channels, the lower channel adopts the bottleneckx module, including candy processing and dwconv convolution, so that it can fuse edge information and local feature information. The YOLOv8n-c3xghost neural network model is deployed and trained in the cloud, and then sunk to the edge side after compression and packaging, realizing real-time video frame detection on the edge side.

[0012] According to some embodiments, the present disclosure adopts the following technical solutions:

[0013] The subway fall detection system based on cloud-edge collaboration and X-CLIP model includes:

[0014] The data acquisition module is used to acquire the real-time video stream in the subway and parse the video stream data into frame image data;

[0015] The target detection module is used to input the frame image data into the YOLOv8n-c3xghost neural network model to obtain all human target area detection frames; the human target area detection frames are identified and detected by the BoT-SORT target tracking algorithm to obtain candidate images of abnormal behavior;

[0016] A behavior judgment module is used to input the candidate image of the abnormal behavior into the X-CLIP model, output the classification result of the abnormal region segment of the candidate image, further identify the abnormal region segment, determine the fall behavior according to the probability of fall being greater than the set threshold, output the location coordinates of the fall behavior and the corresponding original image, and issue an alarm;

[0017] Among them, the c3xghost module of the backbone network of the YOLOv8n-c3xghost neural network model is divided into two channels, the lower channel adopts the bottleneckx module, including candy processing and dwconv convolution, so that it can fuse edge information and local feature information. The YOLOv8n-c3xghost neural network model is deployed and trained in the cloud, and then sunk to the edge side after compression and packaging, realizing real-time video frame detection on the edge side.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] A non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the subway fall detection method based on cloud-edge collaboration and X-CLIP model is implemented.

[0020] According to some embodiments, the present disclosure adopts the following technical solutions:

[0021] An electronic device, characterized in that it includes: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the subway fall detection method based on cloud-edge collaboration and X-CLIP model.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] In the subway fall detection method based on cloud-edge collaboration and X-CLIP model disclosed in the present invention, the target detector and video classifier are trained separately on the cloud, and then they are sent to the edge after compression and packaging operations to detect the real-time bit stream. After a fall is detected, the information is pushed to the video processing platform, and the "problem feedback-model correction" method is applied to update the model and improve the generalization ability. At the same time, after the judgment of the secondary discriminator, it is decided whether to push the information to the terminal, thereby reducing false alarms.

[0024] The disclosed subway fall detection method based on cloud-edge collaboration and X-CLIP model, Ghosntnet and deep convolution (dwconv) in the YOLOv8n-c3xghost neural network model both have the function of reducing network parameters and are suitable for deployment on edge devices; the detection head is replaced, and the main function of the Detect_LSCD detection head is to serve as the target detection head of the YOLOv8 model, responsible for predicting the target category and bounding box in the image. It is implemented through multiple detection layers, each of which can detect targets of different scales. This class also supports dynamic size input and model export, so that it can adapt to different application scenarios and deployment requirements. The Detect_LSCD class is a highly modular and configurable target detection component that uses shared convolutional layers, deformable convolutions, and specific initialization strategies to improve detection performance and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0026] Figure 1 It is an overall process architecture diagram of an embodiment of the present disclosure;

[0027] Figure 2 is a system flow chart of an embodiment of the present disclosure;

[0028] Figure 3 It is a flow chart of the working mechanism based on cloud-edge collaboration in an embodiment of the present disclosure;

[0029] Figure 4 It is a YOLOv8n-c3xghost neural network model architecture diagram of an embodiment of the present disclosure;

[0030] Figure 5 This is a c3xghost module architecture diagram of an embodiment of the present disclosure;

[0031] Figure 6 This is a phenomenon diagram of a false alarm in an embodiment of the present disclosure;

[0032] Figure 7 It is a diagram of the enclosure equipment of an embodiment of the present disclosure;

[0033] Figure 8 This is a diagram showing the AlphaPose detection results of an embodiment of the present disclosure;

[0034] Fig. 9 This is a comparison diagram of underreporting in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] Example 1

[0039] In one embodiment of the present disclosure, a subway fall detection method based on cloud-edge collaboration and an X-CLIP model is provided, including:

[0040] Step 1: Obtain the real-time video stream in the subway and parse the video stream data into frame image data;

[0041] Step 2: Input the frame image data into the YOLOv8n-c3xghost neural network model to obtain all human target area detection frames; identify and detect the human target area detection frames through the BoT-SORT target tracking algorithm to obtain candidate images of abnormal behavior;

[0042] Step 3: Input the candidate image of the abnormal behavior into the X-CLIP model, output the classification result of the abnormal region fragment of the candidate image, identify the abnormal region fragment again, determine the fall behavior according to the probability of fall being greater than the set threshold, output the location coordinates of the fall behavior and the corresponding original image, and issue an alarm;

[0043] Among them, the c3xghost module of the backbone network of the YOLOv8n-c3xghost neural network model is divided into two channels, the lower channel adopts the bottleneckx module, including candy processing and dwconv convolution, so that it can fuse edge information and local feature information. The YOLOv8n-c3xghost neural network model is deployed and trained in the cloud, and then sunk to the edge side after compression and packaging, realizing real-time video frame detection on the edge side.

[0044] Specifically, the subway fall detection method based on cloud-edge collaboration and X-CLIP model provided by the present invention has three major functions: fall detection, cloud-edge collaboration, and storage analysis. The specific method is as follows.

[0045] 1: A method for fall detection, obtaining a real-time video stream in the subway, parsing the video stream data into frame image data and video data; inputting the frame image data into the YOLOv8n-c3xghost neural network model to obtain all human target area detection frames; identifying and detecting the video clips generated by the human target area detection frames through the BoT-SORT target tracking algorithm to obtain candidate videos of abnormal behavior; inputting the candidate videos of abnormal behavior into the X-CLIP model, outputting the classification results of the abnormal area clips of the candidate images, and further identifying the abnormal area clips, determining the fall behavior based on the probability of falling being greater than the set threshold, outputting the location coordinates of the fall behavior and the corresponding original image and issuing an alarm; wherein, the model is deployed and trained in the cloud, and is compressed and packaged and then sunk to the edge side, so as to realize real-time video frame detection on the edge side.

[0046] Whether the results of XCLIP diagnosis are correct or not, they are pushed to the video platform. The video platform performs secondary classification on the video results obtained by xclip, expands its own fall and non-fall samples, and supports secondary model training to improve the model's capabilities. It also has the function of storing images and videos.

[0047] 2: Cloud-edge collaborative processing logic, the cloud is mainly responsible for training the network, collecting data, and secondary judgment, the edge side diagnoses the real-time code stream (video stream), and when an alarm occurs, the alarm data is pushed to the cloud for storage, and a secondary judgment is performed to reduce missed reports. Among them, edge personnel filtering method, special personnel filtering method, and other cognitive mechanism methods are used in the secondary discriminator to reduce false alarms of personnel. When the secondary discriminator determines that it is a correct alarm, the alarm is pushed to the station edge measurement to reduce interference with station operators.

[0048] After the X-CLIP-YOLO-BoT-SORT architecture is deployed in the cloud, the weights are sent down to the edge test. The video stream data is detected in real time on the edge side, and the abnormal image information diagnosed by the system is transmitted to the cloud for manual analysis in the cloud. False alarms mainly include people at the edge of the image, suspected falls (squatting, bending over, wiping facilities, sitting on elevators and other equipment, maintenance personnel maintaining equipment), etc. Based on this situation, information interaction is achieved and the automatic diagnosis process is completed.

[0049] 1) When the system detects that the false alarm rate of the edge does not exceed the threshold within a period of time, the method shown in the figure is set to directly expand the alarm data into the data set to achieve retraining and diagnosis of the network model. In the model training diagnosis, the data samples with low confidence in the network output are directly deleted to achieve data self-diagnosis analysis method and improve data quality.

[0050] 2) When the system detects that the false alarm rate of the edge exceeds the threshold within a period of time, for the fall alarm, if the edge detects a fall, but the cloud does not detect the edge's handling information of the fall behavior within a certain period of time, then it is considered that the system is falsely reporting the fall, and these misjudged images are input into the non-fall data set of the cloud. At the same time, when a fall is detected and the edge is detected for handling, then the system's fall is considered to be a correct fall behavior, and these images are expanded to fall samples. When the system detects that the false alarm rate of the edge exceeds the threshold within a period of time, the classifier and xclip model in the figure are retrained to complete the simultaneous retraining of the classifier and XCLIP model shown in the figure to improve the model's discrimination ability.

[0051] Three: The present disclosure also has the functions of storage and secondary analysis. Specifically. In addition, after detecting a fall, the image has a storage function, and the data is stored in a fixed location through compression methods such as h265 or h264, realizing data conservation processing. In particular, a secondary classifier is designed and classified for the stored images to realize the classification and model retraining of the compressed images, and statistical analysis is realized according to the design of the classifier, such as the frequency of statistical events, probability density curves, etc.

[0052] As an embodiment, the present invention discloses a subway fall detection method based on cloud-edge collaboration and X-CLIP model, proposes a fall detection method of X-CLIP-YOLO-BoT-SORT architecture, proposes an optimization method of YOLOv8n-ghostlscd based on YOLO improvement to realize pedestrian detection, and uses BoT-SORT algorithm to realize pedestrian tracking, applies X-CLIP model to realize video classification, proposes cloud-edge collaborative work architecture, applies X-CLIP model to realize video classification, designs "problem feedback-model correction" method, solves the problem of high missed alarm rate and false alarm rate in engineering applications, and the specific implementation process of the method is as follows:

[0053] Step 1: Obtain the real-time video stream in the subway and parse the video stream data into frame image data;

[0054] First, the video stream collected by the camera in real time is obtained, the video stream data is parsed into frame image data, and a frame queue is set, and the pre-processed frame image data is sent to the frame queue to wait for target detection.

[0055] After being parsed into frame image data, pre-processing operations such as cropping and size unification are performed on the frame image.

[0056] The preprocessed frame image data is then sent to the frame queue to wait for human target recognition by the YOLOv8n-c3xghost neural network model.

[0057] Step 2: Build the YOLOv8n-c3xghost neural network model to perform human target detection on the preprocessed frame image data;

[0058] Specifically, the preprocessed frame image data is input into the YOLOv8n-c3xghost to realize human target positioning and recognition.

[0059] Fine-tune the YOLOv8n-c3xghost weights to obtain the network weights suitable for the subway, and then implement the detection YOLOv8n-c3xghost neural network model, such as Figure 3 As shown, the YOLOv8n-c3xghost neural network model includes a backbone network, a neck network and a detection head, wherein the backbone network includes Ghostnet, C3Ghost and c3xghost modules, and the c3xghost module is improved, such as Figure 3 As shown in the figure, the c3xghost module is divided into two channels, the lower channel uses the bottleneckx module, including candy processing and dwconv convolution, so that it integrates edge information and local feature information, making the model's perception information richer; the upper channel is the Bottleneck module, including GhostConv and DwConv convolution, and the lower channel uses the bottleneckx module, including candy processing and dwconv convolution operations, where the candy transformation obtains edge information, and dwconv obtains local feature information, realizing the fusion of multiple types of feature information and enhancing the network feature extraction capability. Ghosntnet and deep convolution (dwconv) both have the function of reducing network parameters and are suitable for deployment on edge devices. The upper channel is a combination of Bottleneck and conv channels, and the lower channel is a combination of Bottleneckx and conv channels. Then, after the combination, the combination is processed again, which improves the feature extraction capability, and can also obtain the edge information after candy processing, and can maintain the ability to extract local features.

[0060] Furthermore, the YOLOv8n-c3xghost neural network model also includes a neck network and a detection head. The detection head is replaced with the Detect_LSCD detection head, whose main function is to serve as the target detection head of the YOLOv8n-c3xghost neural network model, responsible for predicting the target category and bounding box in the image. It is implemented through multiple detection layers, each of which can detect targets of different scales. The class also supports dynamic size input and model export, so that it can adapt to different application scenarios and deployment requirements. The Detect_LSCD class is a highly modular and configurable target detection component that uses shared convolutional layers, deformable convolutions, and specific initialization strategies to improve detection performance and accuracy. The detection head uses Detect_LSCD, and the GongGN module is used in the detection head. The GongGN module is composed of Group Normalization. If the feature map X input to the detection head has C channels, the Group Normalization divides it into G groups, and each group g has C / G channels. For each g, the normalization operation is as follows:

[0061] xg_1=(xg-μg) / (σg 2 +∈),

[0062] Among them, xg_1 is the output result, xg is the input of the g-th group, μg is the mean of the g-th group, σg 2 is the variance of the gth group and ∈ is a small constant for numerical stability.

[0063] The YOLOv8n-c3xghost neural network model is deployed and trained in the cloud, and then compressed and packaged and sent to the edge side, where real-time video frame detection is achieved. The frame image data is input into the YOLOv8n-c3xghost neural network model, and the output label is the human body region target detection box of the person.

[0064] Figure 3 This involves the convolutional neural network (CNN) in deep learning. This structure is a network layer that uses depthwise separable convolution (DwConv) and residual connection (shortcut). Among them:

[0065] 1. **Bottleneck**: This usually refers to a bottleneck layer in the network, which is used to reduce the dimension of the feature map and then expand it back to its original dimension. This helps to reduce the amount of calculation and the number of parameters.

[0066] 2. **G3xGhost**: This refers to a special convolution operation, and "Ghost" may refer to GhostConvolution, which is a technique used to reduce computational complexity.

[0067] 3. **Ghost**: This is the abbreviation of Ghost Convolution, which is a lightweight convolution operation that reduces the computational cost by reducing the number of convolution kernels.

[0068] 4. **Conv**: This is a regular convolution operation used to extract features.

[0069] 5. **ifshortcut**: This may represent a conditional statement that decides whether to use residual connections. Residual connections are a technique that helps train deep networks by directly connecting inputs and outputs by skipping some layers.

[0070] 6. **DwConv**: Depthwise Separable Convolution, as mentioned earlier, it decomposes the standard convolution into depthwise convolution and pointwise convolution to reduce parameters and computation, local feature extraction: Since the depthwise convolution processes each channel independently, it is particularly good at capturing local features such as edges, corners, and textures. These local features are crucial for image recognition and classification tasks.

[0071] 7. **add**: This means adding two feature maps together, usually in a residual connection, adding the input directly to the output of the convolutional layer.

[0072] 8. **Bottleneckx**: This bottleneck layer structure includes candy and dwconv operations.

[0073] 9.***candy**: An edge detection algorithm.

[0074] 10. *cat**: This is the abbreviation of "Concatenate", which means connecting multiple feature maps together.

[0075] Overall, this structure describes a complex network structure containing multiple convolutional layers, depth-separable convolutional layers, and residual connections. This structure is often used in tasks such as object detection to improve the performance and efficiency of the network.

[0076] The convolutional neural network structure in the deep learning model extracts and processes image features through a series of convolutional layers, depthwise separable convolution (DwConv), ghost convolution, and bottleneck layers (Bottleneck). The upper channel is a combination of Bottleneck and conv channels, and the lower channel is a combination of Bottleneckx and conv channels. The combined processing improves the feature extraction capability, and can also obtain edge information after candy processing, while maintaining the ability to extract local features.

[0077] After each bottleneck layer, it is conditionally determined whether to use a residual connection, that is, to add the input directly to the output to help solve the gradient vanishing problem in deep networks. The entire structure may contain multiple repeated layers, each of which extracts features through convolution operations and optimizes network performance through feature fusion and dimensionality adjustment. Ultimately, these processed feature maps are used for prediction of image classification or other related tasks.

[0078] Step 3: The BoT-SORT target tracking algorithm performs recognition and detection to obtain candidate images of abnormal behavior;

[0079] Specifically, the YOLOv8n-c3xghost neural network model filters out human region targets with the label person, assigns a unique ID through the BoT-SORT target tracking algorithm, performs polling detection on every 32 frames of images, creates a fall detection object model through the fall_detect function, and then calls the X-CLIP model.

[0080] Among them, the process of identifying and detecting the human target area detection frame through the BoT-SORT target tracking algorithm to obtain the candidate image of abnormal behavior is as follows:

[0081] Accurate multi-target tracking is achieved by improving Kalman filter, state variable selection, camera motion compensation, and fusion of IoU and Re-ID features.

[0082] 1) Improved Kalman filter: BoT-SORT improves the state vector of the Kalman filter from the previous center point, area, aspect ratio, speed, and area change rate to a vector containing elements such as center point, width, height, width speed, and height speed. The prediction step and update step follow the standard formula of the Kalman filter, that is, the prediction step is:

[0083]

[0084] The update steps are:

[0085]

[0086] P t∣t =P t∣t-1 -K t H t P t∣t-1

[0087] In the formula, represents the predicted state at time point t based on all previous information; F t is the state transfer matrix, which describes the evolution of the state over time; B t 、u t represents the control input matrix and control signal respectively; Q represents the process noise covariance. In the observation update phase, z t is the actual observed value, H t is the observation matrix, K t is the Kalman gain, which is used to fuse the prediction and observation information, P t∣t-1 , P t∣t are the predicted and updated covariance matrices, respectively, reflecting the uncertainty of state estimation. Through the prediction step and the update step, the Kalman filter can effectively estimate and track the system state.

[0088] 2) State variable selection: The state variables of BoT-SORT are

[0089]

[0090] Among them, x c ,y c is the coordinate of the target center point, w,h is the width and height of the target, is the corresponding speed.

[0091] 3) Camera motion compensation: The BoT-SORT algorithm estimates the camera motion using image registration technology, obtains the affine matrix through the RANSAC (Random Sample Consensus) algorithm, and then compensates the Kalman prediction results. The affine matrix is ​​defined as A = [MT], where M is the rotation matrix and T is the translation vector.

[0092] 4) Fusion of IoU, cosine distance, head Pearson loss function and Re-ID: BoT-SORT combines IoU and Re-ID features to establish the association between detection and trajectory by minimizing IoU distance, cosine distance, and head Pearson loss function. It uses exponential sliding average to update the appearance state of the trajectory, and combines motion and appearance features as the cost matrix in the matching process.

[0093] The Pearson correlation coefficient is a statistic that measures the linear correlation between two variables. Its value is between -1 and 1. It was proposed by Karl Pearson to measure the correlation between two variables. In multi-target tracking tasks, the Pearson correlation coefficient can be used to quantify the similarity between feature vectors and establish the association between detection and trajectory.

[0094] Minimizing the IoU distance, cosine distance, and head Pearson loss function can effectively solve the problem of occluding the human body but not the head, and solve the problem of missing label ids. We first build a human detection target detector and a head target detector. When the IOU and cosine distance of the human body are detected to be small, the IOU is less than 0.2, and the cosine distance is less than 0.5. It is considered that there may be occlusion, and then the head area in the human detection frame is first found, and then the Pearson loss function is solved for the head area and the head area of ​​the previous frame. If the Pearson correlation coefficient is greater than 0.8, the human target corresponding to the head target and the corresponding data of the previous frame are successfully matched.

[0095] To handle occlusion issues in human body and head detection and perform effective matching, follow these steps:

[0096] Step 1: Initialize the detector

[0097] -Build and train a human detector and a more accurate head detector.

[0098] Step 2: Detect human bodies and heads in the frame

[0099] -For each frame of video, use a human detector to detect the human body.

[0100] -Within the detected human body area, use the head detector to detect the human head.

[0101] Step 3: Calculate IOU and cosine distance

[0102] -For each pair of detected human body and head, calculate the intersection over union (IOU) between them.

[0103] -Extract the features of the head area and calculate the cosine distance between the head area of ​​the current frame and the previous frame.

[0104] Step 4: Determine the occlusion situation

[0105] If the IOU between the human body and the head is greater than 0.2 or the cosine distance is less than 0.5, the matching is performed according to the conventional matching method, that is, calculated according to the original botsort algorithm.

[0106] -If the IOU between the human body and the head is less than 0.2 and the cosine distance is less than 0.5, it is considered that there may be occlusion.

[0107] Step 5: Extract head area features

[0108] -Extract the features of the head area from the human detection frame.

[0109] Step 6: Calculate the Pearson correlation coefficient

[0110] -Use the extracted head region features to calculate the Pearson correlation coefficient between the current frame and the previous frame.

[0111] Step 7: Matching Decision

[0112] -If the Pearson correlation coefficient is greater than a certain threshold, such as 0.8), it is considered that the head target in the current frame matches the head target in the previous frame successfully.

[0113] Step 8: Update matching results

[0114] -Associate the human body target corresponding to the successfully matched head target with the data of the previous frame.

[0115] -Update the trajectory information to provide a reference for the next frame matching.

[0116] Step 9: Process the next frame

[0117] - Repeat steps 2 to 8 to process the next frame of the video.

[0118] Step 10: Output the results

[0119] - Output the matching results, including the ID, location and tracking status of each target.

[0120] Through these steps, human body and head targets can be effectively matched in the presence of occlusion, improving the accuracy and robustness of multi-target tracking.

[0121] in,

[0122] Cosine distance: Cosine Distance = 1-AB||A||B, where A and B are two eigenvectors.

[0123] Pearson correlation coefficient:

[0124]

[0125] Among them, x i and i are the components of the two eigenvectors, and is their mean.

[0126] 5) Trajectory compensation and operation.

[0127] This disclosure adds an additional method. If a certain ID is missing in the middle frame, linear interpolation compensation is performed on the middle frame, mainly compensating its position data, including the center point and length and width data. The method adopts a linear fitting method. The data of the previous frame is lost as x1 c ,y1 c ,w1,h1, find the data with id number x c ,y c ,w,h,Assume that n frames are missing, then linear fitting is used to obtain the data information of the middle n frames.

[0128] The method is: when the data of the previous frame is lost in multi-target tracking, and the data information of the middle n frames needs to be predicted, the linear fitting method can be used to estimate the data of these lost frames. The following are the specific steps for linear fitting:

[0129] 1. Identify known data points

[0130] First, at least two known data points need to be determined. These two points can be trajectory data before and after the lost frame. Assume that the data of the t_1th frame and the t_2th predicted frame are known, where t_2=t_1+n+1.

[0131] 2. Define the linear fitting model

[0132] The linear fitting model can be expressed as:

[0133]

[0134] Among them, {z}_t is the predicted data at time t, and zt1 and zt2 are the known start and end frame data respectively.

[0135] 3. Calculate the data for each lost frame

[0136] For each lost frame t, the prediction data is calculated using the above formula:

[0137] 4. Implementation Details

[0138] The center coordinate, length, width and other data of the bounding box need to be linearly interpolated for each coordinate and finally combined to get the final data value.

[0139] Step 4: The X-CLIP model outputs the classification results of the abnormal region fragments of the candidate image;

[0140] Specifically, the candidate image of abnormal behavior is input into the X-CLIP model, the classification result of the abnormal area fragment of the candidate image is output, the abnormal area fragment is further identified, the fall behavior is determined based on the fall probability being greater than the set threshold, the location coordinates of the fall behavior and the corresponding original image are output, and an alarm is issued.

[0141] Furthermore, the X-CLIP model is called to judge the fall behavior of all the segmented areas, and the area IDs with a fall probability greater than the set threshold are added to the fall_ids list. The target IDs and their corresponding boxes are traversed one by one, and the box coordinates corresponding to the target ID added to the fall_ids list in fallDetect are added to the fall_boxes list. In preparation for the subsequent marking of the fall behavior, the coordinates in the frame.fall_boxes list are traversed and visualized, and the target is framed with a red line. If it is a fall, data is pushed and the fallen target is marked and displayed.

[0142] Among them, the process of obtaining the fall probability, that is, the confidence, of the X-CLIP model is as follows.

[0143] When using the X-CLIP model for video recognition, combined with the inherent model method, the process of obtaining the confidence of the classification label can be summarized into the following key steps:

[0144] 1. First, xclip is used to train the subway fall video. Through the previously marked fall video data, multiple video clips are marked as falls or non-falls, and the labels are processed completely. Then the data is imported into the xclip model for training, the text sequence encoding is frozen, and the video data is re-pretrained to complete the relevant operations. **Video and text encoding**: The video is passed through the video encoder (including Patch Embedding, Cross-frame Communication Transformer and Multi-frame Integration Transformer) to obtain the video feature vector v, and the text (category label) is passed through the text encoder to obtain the text feature vector c.

[0145] 2. Video-specific prompting: The fall prompt information is input into the xclip model, and the video-specific prompting is used on the text side to enhance the text features. This is achieved by adding visual context information to each layer of the text encoder to incorporate the semantic information of the video content and obtain the enhanced text feature vector cc.

[0146] 3. Similarity calculation: Calculate the similarity between the video feature vector v and the enhanced text feature vector cc. This is usually done by calculating the cosine similarity between the two.

[0147] 4. Confidence score: For the fall and other two categories, the similarity between the video feature vector and the text feature vector of the category is calculated to obtain a score, which can be regarded as the model's confidence that the video belongs to the category.

[0148] 5. Apply the Softmax function: The above scores are converted into probability distributions through the Softmax function. The confidence of each category is the value of the category in the probability distribution.

[0149] 6. Output classification results: Finally, the model outputs the confidence score for each category, which indicates the probability that the model believes the video belongs to each category.

[0150] In X-CLIP, through the cross-frame attention mechanism and video-specific prompts, the model can effectively utilize the temporal information of the video content and the semantic information of the category label, thereby improving the accuracy and confidence of video recognition. Through the above methods, the effectiveness and efficiency of the computational process are achieved.

[0151] If a fall occurs, the inferred data will be organized into the corresponding JSON format according to the unified interface specification, and then sent to the previously passed alarmSink and dataSink addresses, and then displayed and summarized on the subway's video platform.

[0152] As an embodiment, the present disclosure proposes a fall detection method based on an X-CLIP-YOLO-BoT-SORT architecture, and the training process is as follows:

[0153] 1) Label the data containing falls, label the first frame, the last frame, and the middle frame of the fall, and then save the sequence numbers of these frames and the name of the fall type.

[0154] 2) The YOLOv8n-c3xghost neural network model and tracker detect the video in step 1), and then track it according to the sequence number and target frame of the first frame to obtain the trajectory of the target and the target frame corresponding to each frame of the trajectory, and capture each frame corresponding to the trajectory and the image corresponding to the target frame and save it as a small video sequence.

[0155] 3) For the video sequence obtained in step 2), the video sequence is respectively subjected to the form of extracting video segments every 2 frames, extracting video segments every 3 frames, and extracting video segments every 4 frames to obtain multiple video samples.

[0156] 4) For the video samples obtained in step 3), randomly perform color jitter on a frame in the video, add noise, add snowflakes, and add fog image processing operations to obtain video samples after data set enhancement, thereby further enriching the video data samples.

[0157] 5) For step 4), complete the collection of positive samples of falling

[0158] 6) For step 1), remove the video parts of the first frame and the last frame, randomly select data frames, and complete the collection of negative samples according to steps 2345.

[0159] 7) Further input the obtained positive and negative samples into the X-CLIP model, load the official weights, load the data set, fine-tune the model, save the fine-tuned weights, and when diagnosing the real-time bitstream, first call the YOLOv8n-c3xghost algorithm code, then perform target tracking to form a video sequence, input the weights to detect whether there is a fall, and if so, select the image.

[0160] Specifically, weights are adjusted according to different location scenarios. After fine-tuning the model, the network is further trained and new weights are saved, thereby obtaining different parameters in the network. The network learns the parameter values ​​of the network architecture during training and improves the network level in a targeted manner.

[0161] Model fine-tuning refers to deploying different weights for different scene locations. For different scene locations of the camera, the same model is built, and the algorithm model is constructed using the corresponding location data set to obtain the weights. Different weights are obtained at different locations. The current location category is then obtained through the classification algorithm, and then the corresponding location weight is used for detection to achieve the effect of automatically obtaining weights and diagnosing. Among them, the classification of scene locations can adopt existing mature algorithms for location classification. The strategy adopted by scene classification is to obtain 10 images of the current location, and then input them into the network for judgment, and finally vote to obtain the final scene location type.

[0162] Specifically, the method for fine-tuning the fall network model disclosed in the present invention is:

[0163] 1) Collect the falling videos and non-falling videos, mark them with marking tools, and record the position of the target area in each frame.

[0164] 2) Perform dataset enhancement on the video data in step 1 to increase its sample size by randomly selecting some frames of the video, adding noise, brightness change, color change, candy change processing, contrast enhancement and other processing to these frames, and then resynthesizing the enhanced data into a new video clip, whose label and original video data remain unchanged.

[0165] 3) As the falling action is fast or slow, we extract data at regular intervals from the data set obtained in step 2 to form a new video clip. In a certain frame in the video, we repeat this process multiple times to slow down the falling action and form a video clip. We expand the video clips formed in this step into the data set to further expand the number of data sets.

[0166] 4) For the enhanced dataset, separate pedestrian target sequence fragments are captured, and new sample data is generated and input into the XCLIP model for re-pre-training.

[0167] 4.1) Cross-frame Attention:

[0168] The XCLIP model introduces a cross-frame attention mechanism that allows the model to exchange information between video frames. This mechanism is implemented by adding Cross-frame Communication Transformer (CCT) and Multi-frame Integration Transformer (MIT) modules to the video encoder to capture the temporal information of the video.

[0169] 4.2) Video-specific Prompting:

[0170] In order to better utilize the semantic information in the category labels, XCLIP adopts a video-specific prompting strategy, directly using "{fall, not fall}" as a template and enhancing the discriminability of text features by adding visual context information at each layer of the text encoder.

[0171] 4.3) Fine-tuning strategy:

[0172] During fine-tuning, the XCLIP model can selectively fine-tune the image encoder and the text encoder. For example, only fine-tune the image encoder while keeping the pre-trained weights of the text encoder unchanged. This approach can reduce the amount of computation while still being able to effectively adapt to specific video recognition tasks.

[0173] In particular, an improved method and strategy in training are proposed, in which an additional graphics processing channel chanelB is added to the image encoder channel, wherein the channel is the original video sequence that has been processed by adding random noise, enhancing contrast, adding noise, etc. to form a new video clip, which is pre-trained together and put into the same graphics encoder.

[0174] 4.4) Loss Function:

[0175] During training, the XCLIP model may use a contrastive learning loss function, which optimizes model parameters by minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs. Specifically, the loss function = original loss function + chanelB loss, where chanelB loss is the contrastive learning loss function under this channel.

[0176] The XCLIP model can effectively utilize the pre-trained language-image model. Its simplicity, efficiency, and versatility make it a method for fall detection in the field of video recognition.

[0177] 4.5 Real-time testing and simulation. Based on the fall pattern detected by YOLO, if a pedestrian area is detected, the corresponding video is input into the network model obtained in step 4 to test the result. During the test, the +chanelB channel used in training is removed and the original data is directly tested.

[0178] As an embodiment, the subway fall detection method based on cloud-edge collaboration and X-CLIP model disclosed in the present invention proposes a problem feedback-model correction method, such as Figure 2 As shown, including:

[0179] After the edge sinking and falling recognition model architecture is established, the recognition results will show frequent false alarms and missed alarms, which greatly reduces the user experience in the real engineering environment. The present invention proposes the "problem feedback-model correction" technology to solve the problem of poor applicability. It mainly includes re-pre-training, secondary discrimination method (including marginal personnel filtering method, special personnel filtering method, etc.), other cognitive mechanism methods, keypoint analysis method, etc., to correct the re-corrected results, obtain the final model, and realize application deployment.

[0180] The cloud is mainly responsible for training the network, collecting data, and secondary judgment. The edge side diagnoses the real-time code stream (video stream). When an alarm occurs, the alarm data is pushed to the cloud for storage, and a secondary judgment is performed to reduce missed reports. Among them, edge personnel filtering, special personnel filtering, and other cognitive mechanism methods are used in the secondary discriminator to reduce false alarms of personnel. When the secondary discriminator determines that it is a correct alarm, the alarm is pushed to the station edge measurement to reduce interference with station operators.

[0181] After the X-CLIP-YOLO-BoT-SORT architecture is deployed on the cloud, the weights are sent to the edge. The edge detects the video stream data in real time and transmits the abnormal image information diagnosed by the system to the cloud for manual analysis. False alarms mainly include people on the edge of the image, suspected falls (squatting, bending, wiping facilities, sitting on elevators and other equipment, maintenance personnel maintaining equipment), etc. According to this situation, the main corrective measures include the following methods:

[0182] 1) Re-pre-training. Find the corresponding video clips for obvious standing, bending, squatting, sitting and other actions, and then put them in the negative samples to achieve re-pre-training, and send the pre-training results to the edge device for detection.

[0183] During the detection, it was found that some actions such as squatting, sitting, tying shoelaces, and bending over were misjudged, and dark backgrounds were judged as falls. Such data was collected and placed in the network. Such data was misjudged as negative samples for re-pre-training, and then the changed model was deployed on site.

[0184] 2) Secondary Discriminator

[0185] ① Filtering of people at the edge of the image. Because part of a person, such as the upper body, will appear in the edge measurement area, and as the action progresses, it will gradually become half of the body and appear in the video screen. The edge person filtering method solves the problem of false alarms of people at the corners of the image.

[0186] The phenomenon of false positives of people at the corners of the image greatly reduces the user-friendliness of the operation. Therefore, edge person filtering is adopted. Judgment logic is added to the reasoning program of edge measurement;

[0187] 1. Define the detection area and narrow the detection range to a certain range. When a fall event occurs in the detection area, an alarm will be triggered, otherwise no alarm will be triggered.

[0188] 2. Size acquisition method, solve the center point of the pedestrian. If the coordinates of the center point are too close to the boundary, it is considered to be a boundary point and no alarm is issued. Otherwise, it will be judged according to the existing rules.

[0189] ②Special personnel filtering method, false alarms from maintenance workers, cleaning staff, and standing personnel.

[0190] Furthermore, when performing maintenance or cleaning, one needs to bend over or lie down to operate, which causes the action to be identified as falling. Although the action is very similar to falling, too many such events will lead to frequent false alarms, so an alarm should not be issued. The method is to use the above-mentioned target recognition and tracking classifier to realize judgment. Through the constructed video platform, alarm pictures are collected, and then divided into three categories according to the alarm pictures: cleaning, maintenance, and standing. Images of maintenance workers and cleaning personnel are trained, and the detection results are sent to the secondary judgement. If the result is maintenance workers and cleaning personnel, no alarm will be issued; the existing classification model is used to collect relevant data sets. When the detection result output is maintenance workers and cleaning personnel, no alarm will be issued.

[0191] Among them, the method of collecting the classification sample library is as follows: the system pushes images containing alarm target frames, and based on the warning images and target frames, considering that the enclosure of equipment by maintenance personnel during maintenance will affect the category judgment, more pixels around are intercepted to expand the sample set. 1,000 images of each category of maintenance workers, cleaning personnel, and standing personnel are collected. After classification, the accuracy rate is 99.5%, laying the foundation for the subsequent secondary detection and screening.

[0192] 3) Other cognitive mechanism methods

[0193] In the real use environment, there will be false alarms and missed alarms. After analyzing such cases, the cognitive judgment mechanism is adopted for secondary review, which greatly improves the availability of the system. In real projects, only in non-operating events, if the porters and maintenance personnel of other external units are judged to have fallen, the time threshold selection method is adopted here. When it is not during the operating hours, no judgment is made, which improves the accuracy of the system and achieves a friendly use effect.

[0194] 4)Keypoint analysis method.

[0195] This method is used to reduce the false alarm rate. In real projects, it is found that there are certain false alarms, which poses a great safety risk, because users would rather have more false alarms than miss false alarms. Therefore, logical judgment is added. If the head is below the footsteps and the head y coordinate is greater than the footsteps y coordinate, it is considered a fall and an alarm is directly issued. In a real environment, the crowd is detected every 2 seconds to obtain the key bone coordinates of the human body, which can greatly reduce the probability of false alarms and improve the system.

[0196] In particular, in traditional video tagging, it is necessary to tag the data of each frame of the target, which is a huge workload. Figure 2 , design a video automatic annotation tool, collect several fall samples, make them into video clips, mark the start, end, and middle frames of the fall, and then mobilize the automatic annotation tool. The tool marks the target frame according to the start, and then combines the YOLO pedestrian target detector and the BOTSORT tracking algorithm to complete the tracking process, so as to save the entire target clip, and use the end and middle frames to correct the accuracy of the BOTSort algorithm detection. If the detection is wrong, the video clip is automatically deleted, which is a good treatment. At the same time, when marking the start, end, and ID of the target, the pedestrian target detector can detect other pedestrians. If the ID of other pedestrians is not the ID of the fallen pedestrian, then the pedestrian is tracked and the video clip of the pedestrian is obtained as a non-fall video. If the IOU threshold of the target in the video and the end / middle frame of the fallen pedestrian is greater than 0.75, then the target is considered to be misjudged and the video clip is directly deleted.

[0197] In the subway fall detection method based on cloud-edge collaboration and X-CLIP model disclosed in the present invention, data is stored in real time, and in terms of storage, functions such as intelligent compression, secondary compression, compression training and testing are adopted.

[0198] When abnormal behavior is detected, pictures and videos are stored, and the station name, time of occurrence, location of occurrence, target area and coordinates of occurrence, event handling process, event handling status, frequency of current event occurrence, etc. are recorded, and the data is saved in the database for later query and analysis and diagnosis.

[0199] The images and videos are stored in the disk using a compressed storage method, where the storage can be h254 or h265 or a storage method designed for deep learning.

[0200] When recording the current data storage, when the event is exceeded, the storage is automatically scaled and restored. Specifically: when the storage exceeds 30 days, the system automatically compresses the original image and original video to 1 / 8 of the original size, making the storage space smaller. When the original image needs to be restored, the super-resolution reconstruction method is adopted to achieve restoration. The method can adopt the algorithm of deep learning scene. The main methods are as follows: example-based method: use a large number of low-resolution and high-resolution image pairs for training and learning mapping relationships; deep learning-based method: especially convolutional neural network (CNN), by learning a large number of data pairs, end-to-end super-resolution reconstruction is achieved; frequency domain-based method, using Fourier transform: processing the image in the frequency domain can more effectively process the edges and details of the image; - Wavelet transform: through multi-scale analysis, effectively represent the local features and texture information of the image.

[0201] Intelligent analysis of compressed images and videos. Intelligent analysis of compressed images and videos is used to build a database, take necessary tags, such as target detection tags and semantic analysis tags, use intelligent algorithms for prediction and classification, and save network weights for diagnostic analysis. The image analysis method is consistent with the x-clip method described above.

[0202] Furthermore, after the system stores a large number of abnormal images and videos, it first has a query function. When a certain type of abnormal data is input, the image is retrieved through the clip algorithm and the video is retrieved through the x-clip algorithm. The retrieved data is displayed, and the number of occurrences, frequency, location, and position are counted. A statistical report is automatically generated, and operational suggestions are given based on the frequency of occurrence. For example, when the frequency of occurrence is high, it is possible to strengthen operational management and increase the number of operational personnel.

[0203] The obtained abnormal data is counted according to the station or time dimension, and a time series is formed in order. Among them, the random response GRR (GRR, generalized random response) algorithm is recorded as CVGRR, which means the cross validation algorithm and GRR algorithm. The specific functions are shown below.

[0204] Store abnormal events and perform statistical analysis. First, detect abnormalities at specific locations using existing methods, and send and store them in storage space. If fighting, traffic jams, crossing gates, climbing fences, fires, kites, plastic bags, newspapers, cables, garbage connections, etc. are found, and the statistical histogram frequency is recorded.

[0205] Randomly select two groups from all the scenarios, and then take GRR respectively, then merge the two processed data, and record the error between the synthetic data and the real data. Repeat the steps, and the minimum error can be determined as the final privacy-preserving data according to the processing method of each frequency, and the data stored in the storage space.

[0206] After the above method is implemented, for publishing and sharing, a pseudo database is generated based on the frequency of real statistics, and sites, times, specific locations, contact numbers, and contact metrics are randomly generated. Figure 8 As shown in Figure 1, the sender counts abnormal events, such as fights, falls, etc. Then calculates their frequencies and represents them as a histogram. Then, a privacy algorithm is used to smooth these data and the results are sent to the receiver.

[0207] Example 2

[0208] In one embodiment of the present disclosure, a subway fall detection system based on cloud-edge collaboration and an X-CLIP model is provided, including:

[0209] The data acquisition module is used to acquire the real-time video stream in the subway and parse the video stream data into frame image data;

[0210] The target detection module is used to input the frame image data into the YOLOv8n-c3xghost neural network model to obtain all human target area detection frames; the human target area detection frames are identified and detected by the BoT-SORT target tracking algorithm to obtain candidate images of abnormal behavior;

[0211] A behavior judgment module is used to input the candidate image of the abnormal behavior into the X-CLIP model, output the classification result of the abnormal region segment of the candidate image, further identify the abnormal region segment, determine the fall behavior according to the probability of fall being greater than the set threshold, output the location coordinates of the fall behavior and the corresponding original image, and issue an alarm;

[0212] Among them, the c3xghost module of the backbone network of the YOLOv8n-c3xghost neural network model is divided into two channels, the lower channel adopts the bottleneckx module, including candy processing and dwconv convolution, so that it can fuse edge information and local feature information. The YOLOv8n-c3xghost neural network model is deployed and trained in the cloud, and then sunk to the edge side after compression and packaging, realizing real-time video frame detection on the edge side.

[0213] Furthermore, the system has three modes: low false alarm rate mode, low missed alarm rate mode, and balanced mode, and sets two models, model a and model b, for detection. The two models can choose parallel detection mode or serial detection mode. The function implementation process is as follows:

[0214] 1. When the system selects the low false alarm rate mode: select model a and model b for simultaneous detection, and increase the alarm thresholds of both. When both algorithms detect an abnormality, an alarm is issued, otherwise no alarm is issued.

[0215] 2. When the system selects the low false alarm rate mode: select model a and model b for simultaneous detection, and lower the threshold between the two algorithms. If one of the two algorithms detects an abnormality, an alarm will be issued, otherwise no alarm will be issued.

[0216] 3. When the balanced mode is selected: lower the confidence threshold of model a so that model a can detect all possible anomalies. For the anomalies detected by model a, obtain the corresponding target area and then input it into model b for detection. When model b also considers it an anomaly and raises the threshold of model b, push the abnormal data. As an implementation case, model a can be seen as XCLIP, and model b can be seen as a secondary classifier.

[0217] like Figure 5 As shown, the system of the present disclosure combines edge computing and cloud computing for intelligent monitoring, including:

[0218] 1. Edge servers and cloud servers: The system may use both edge servers and cloud servers to process data. Edge servers are usually located near the data source and are used to process and analyze data to reduce latency and network load. Cloud servers provide more powerful computing power and storage resources.

[0219] 2. End-side devices: End-side devices may include cameras, sensors, etc., which are used to collect real-time data.

[0220] 3. Real-time monitor: The real-time monitor is responsible for monitoring the data collected by the end-side device, such as video streams or sensor data.

[0221] 4. Trainer: The trainer is used to train machine learning models, such as object detection and skeleton detection models.

[0222] 5. Exception collector: When an abnormal situation is detected, the exception collector will collect relevant data.

[0223] 6. Alarm: If an abnormality is detected, the alarm will be triggered and sound an alarm through different communication channels (such as handheld terminals, workstations, large screens, etc.).

[0224] 7. Secondary Discriminator: After the alarm sounds an alarm, a secondary discriminator may be used to further analyze and confirm the authenticity of the alarm.

[0225] 8. Update the network: The system may optimize and update itself based on the collected data and feedback to improve the accuracy of detection.

[0226] 9. Maintenance and cleaning staff: The system may be integrated with the workflow of maintenance and cleaning staff to improve operational efficiency.

[0227] 10. Station: This may be the location where the surveillance system is deployed, such as a subway station or other transportation hub.

[0228] 11.Handheld terminals and workstations: Workers can use handheld terminals and workstations to receive alarms and monitoring data.

[0229] The present disclosure reduces dependence on cloud servers, reduces latency, and improves response speed by performing preliminary data processing on edge servers; the system is able to monitor and identify abnormal situations in real time, such as the presence of non-operation and maintenance personnel, non-cleaning and maintenance personnel, and non-standing personnel. False alarms are reduced and the accuracy of alarms is improved through secondary discriminators. The system is able to self-learn and update based on the collected data to adapt to the ever-changing environment. The system is closely integrated with the maintenance and cleaning workflow of the station, improving operational efficiency. Alarms can be issued through multiple channels (handheld terminals, workstations, large screens) to ensure that information can be conveyed to relevant personnel in a timely manner. This system is designed to improve the efficiency and accuracy of monitoring, while reducing manual intervention and improving the level of automation.

[0230] Example 3

[0231] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the subway fall detection method based on cloud-edge collaboration and X-CLIP model is implemented.

[0232] Example 4

[0233] In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the subway fall detection method based on cloud-edge collaboration and X-CLIP model.

[0234] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0235] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0236] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A subway fall detection method based on cloud-edge collaboration and X-CLIP model, characterized in that: include: Obtain real-time video stream in the subway and parse the video stream data into frame image data; Input the frame image data into the YOLOv8n-c3xghost neural network model to obtain all human target area detection frames; perform multi-target tracking on the human target area detection frames through the BoT-SORT target tracking algorithm, complete recognition detection, and obtain candidate images of abnormal behavior; Input the candidate image of the abnormal behavior into the X-CLIP model, output the classification result of the abnormal region fragment of the candidate image, identify the abnormal region fragment again, determine the fall behavior according to the probability of fall being greater than the set threshold, output the location coordinates of the fall behavior and the corresponding original image, and issue an alarm; Among them, the c3xghost module of the backbone network of the YOLOv8n-c3xghost neural network model is divided into two channels, the lower channel adopts the bottleneckx module, including candy processing and dwconv convolution, so that it can fuse edge information and local feature information. The YOLOv8n-c3xghost neural network model is deployed and trained in the cloud, and then sunk to the edge side after compression and packaging, realizing real-time video frame detection on the edge side.

2. The subway fall detection method based on cloud-edge collaboration and X-CLIP model as claimed in claim 1, characterized in that: The real-time video stream in the subway is collected by the camera, the video stream data is parsed into frame image data, and a frame queue is set. The frame image data is sent to the frame queue to wait for target detection, and the frame image data is input into the YOLOv8n-c3xghost neural network model. The output label is the human body area target detection box of person.

3. The subway fall detection method based on cloud-edge collaboration and X-CLIP model as claimed in claim 1, characterized in that: The YOLOv8n-c3xghost neural network model includes a backbone network, a neck network and a detection head, wherein the backbone network includes Ghostnet, C3Ghost and c3xghost modules, the upper channel of the c3xghost module is a Bottleneck module, including GhostConv and DwConv convolutions, and the lower channel adopts a bottleneckx module, including candy processing and dwconv convolution operations.

4. The subway fall detection method based on cloud-edge collaboration and X-CLIP model as claimed in claim 1, characterized in that: The YOLOv8n-c3xghost neural network model also includes a neck network and a detection head. The detection head adopts Detect_LSCD. The detection head adopts a GongGN module. The GongGN module is composed of Group Normalization. If the feature map X input to the detection head has C channels, the Group Normalization divides it into G groups, and each group has C / G channels.

5. The subway fall detection method based on cloud-edge collaboration and X-CLIP model as claimed in claim 1, characterized in that: After the YOLOv8n-c3xghost neural network model outputs the human target area detection frame, it is identified and detected through the BoT-SORT target tracking algorithm, and the human target area detection frame is assigned a unique ID. A set number of images are polled for detection to obtain candidate images of abnormal behavior. A fall detection object model is created through the fall_detect function, and the X-CLIP model is called to perform fall behavior judgment on all segmented areas.

6. The subway fall detection method based on cloud-edge collaboration and X-CLIP model as claimed in claim 1, characterized in that: In the X-CLIP model, the area IDs with a fall probability greater than a set threshold are added to the fall_ids list, the detection target IDs and their corresponding boxes coordinates are traversed one by one, the boxes coordinates corresponding to the area IDs in the fall_ids list are added to the fall_boxes list, the coordinates in the frame.fall_boxes list are traversed, the coordinate positions are output and visualized.

7. The subway fall detection method based on cloud-edge collaboration and X-CLIP model as claimed in claim 1, characterized in that: After determining that a fall has occurred, the output location and image data are organized into the corresponding json format according to the unified interface specification, and then sent to the previously passed alarmSink and dataSink addresses, and then displayed and summarized on the subway's video platform.

8. The subway fall detection system based on cloud-edge collaboration and X-CLIP model is characterized by: include: The data acquisition module is used to acquire the real-time video stream in the subway and parse the video stream data into frame image data; The target detection module is used to input the frame image data into the YOLOv8n-c3xghost neural network model to obtain all human target area detection frames; perform multi-target tracking on the human target area detection frames through the BoT-SORT target tracking algorithm, complete recognition detection, and obtain candidate images of abnormal behavior; A behavior judgment module is used to input the candidate image of the abnormal behavior into the X-CLIP model, output the classification result of the abnormal region segment of the candidate image, further identify the abnormal region segment, determine the fall behavior according to the probability of fall being greater than the set threshold, output the location coordinates of the fall behavior and the corresponding original image, and issue an alarm; Among them, the c3xghost module of the backbone network of the YOLOv8n-c3xghost neural network model is divided into two channels, the lower channel adopts the bottleneckx module, including candy processing and dwconv convolution, so that it can fuse edge information and local feature information. The YOLOv8n-c3xghost neural network model is deployed and trained in the cloud, and then sunk to the edge side after compression and packaging, realizing real-time video frame detection on the edge side.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the subway fall detection method based on cloud-edge collaboration and X-CLIP model as described in any one of claims 1-7 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the subway fall detection method based on cloud-edge collaboration and X-CLIP model as described in any one of claims 1-7.

Images