Subway pedestrian anomaly detection method and system based on improved yolov8 network and clip model

Abstract

The invention provides a subway pedestrian anomaly detection method and system based on an improved yov8 network and a clip model, and relates to the technical field of subway pedestrian detection, and the method comprises the steps: obtaining a video frame in a subway scene, and carrying out the preprocessing of the video frame; inputting the preprocessed frame data into an improved yolov8 neural network model to obtain an abnormal behavior target detection frame, and performing target candidate tracking on the target detection frame by using a target tracking algorithm to obtain a candidate image of a target area of a specific behavior; the method comprises the following steps: inputting a candidate image containing a target area into a clip model, inputting a set detection text into the clip model, mapping the detection text and the candidate image of the target area, performing text-image similarity matching, and optimizing a similarity matching threshold by adopting a Q-Learning reinforcement learning immune algorithm to obtain a similarity matching result; and the abnormal behavior detection accuracy is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of subway pedestrian detection, and in particular to a subway pedestrian anomaly detection method and system based on an improved yolov8 network and a clip model. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] As an important part of transportation, the safety of subways has always been the focus of public attention. The subway automated monitoring system can prevent accidents, respond to emergencies quickly, and ensure the safety of passengers and pedestrians through real-time monitoring and data analysis. Abnormal behavior detection is one of the key technologies in subway safety monitoring. It can identify and analyze specific behaviors and timely warn of potential safety risks, which is crucial to ensure the safety of passengers.

[0004] At present, neural network and deep learning methods have been used to detect and track target behaviors in subway pedestrian anomaly detection. Improving target detection performance is of great significance in various industries. Improving model performance and practicality are necessary. However, the current deep learning methods have the problem of too high false positives and false negatives in target monitoring. The currently used yolo neural network model performs anomaly detection on pedestrians, but when the confidence of the yolo neural network model is adjusted too high, false positives will occur, and when it is adjusted too low, false negatives will occur. Therefore, how to balance false positives and false negatives is the current difficulty. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a subway pedestrian anomaly detection method and system based on improved YOLOv8 network and CLIP model, constructs a target detection model based on improved YOLOv8 and CLIP models, uses the improved YOLOv8 model to detect targets in video frames, and then tracks the detected targets through tracking configuration to obtain the target area, uses the CLIP model to analyze the matching degree between the target area image and the preset text to determine whether to trigger an alarm, improves the processing efficiency through multi-threading, and realizes real-time monitoring and response.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions:

[0007] The subway pedestrian anomaly detection method based on the improved yolov8 network and clip model includes:

[0008] Obtain video frames in subway scenes and preprocess them;

[0009] The preprocessed frame data is input into the improved yolov8 neural network model to obtain the abnormal behavior target detection frame, and the target tracking algorithm is used to track the target candidate of the target detection frame to obtain the candidate image of the target area of ​​the specific behavior;

[0010] Input the candidate image containing the target area into the clip model, input the set detection text into the clip model, map the detection text and the candidate image of the target area, perform text-image similarity matching, and if the probability of similarity matching exceeds the set threshold, issue an abnormal behavior alarm;

[0011] Among them, the Q-Learning reinforcement learning immune algorithm is used to optimize the set threshold of similarity matching. The search process is guided by the fitness evaluation of individuals in the immune algorithm, and the best action selection is determined by combining reinforcement learning. The final optimal fitness is output as the set threshold.

[0012] According to some embodiments, the present disclosure adopts the following technical solutions:

[0013] The subway pedestrian anomaly detection system based on the improved yolov8 network and clip model includes:

[0014] The data acquisition module is used to acquire video frames in subway scenes and pre-process them;

[0015] The target detection module is used to input the preprocessed frame data into the improved yolov8 neural network model, obtain the abnormal behavior target detection frame, use the target tracking algorithm to track the target candidate of the target detection frame, and obtain the candidate image of the target area of ​​the specific behavior;

[0016] A review matching warning module is used to input the candidate image containing the target area into the clip model, and input the set detection text into the clip model, map the detection text and the candidate image of the target area, and perform text-image similarity matching. If the probability of similarity matching exceeds a set threshold, an abnormal behavior alarm is issued;

[0017] Among them, the Q-Learning reinforcement learning immune algorithm is used to optimize the set threshold of similarity matching. The search process is guided by the fitness evaluation of individuals in the immune algorithm, and the best action selection is determined by combining reinforcement learning. The final optimal fitness is output as the set threshold.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the subway pedestrian anomaly detection method based on the improved yolov8 network and the clip model is implemented.

[0020] According to some embodiments, the present disclosure adopts the following technical solutions:

[0021] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device implements the subway pedestrian anomaly detection method based on the improved yolov8 network and clip model.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The present invention discloses a method for detecting abnormal pedestrians in subways based on improved YOLOv8 network and clip model, adopts improved YOLOv8 neural network and botsort target tracking algorithm to achieve target acquisition and tracking, and the behavior filter is implemented by CLIP model to achieve behavior-specific filtering, a target monitoring model based on YOLO and CLIP model, which is used to detect and track specific targets in videos, such as falling, climbing over gates, climbing over fences, fighting, etc. or other abnormal behaviors, and improves efficiency through multi-threaded processing. First, the YOLOv8 model is used to detect the target of the video frame, and then the detected target is tracked through the tracking configuration to further obtain the target area, and then the CLIP model is used to analyze the matching degree between the target area image and the preset text, and the improved clip model is adopted for secondary review, so that the clip considers the behavior of falling to alarm, so as to achieve the effect of reducing the false alarm rate and the missed alarm rate in the project, so as to judge whether to trigger the alarm; the processed video frame is written into the output file to achieve real-time monitoring and response.

[0024] The present invention discloses a subway pedestrian anomaly detection method based on an improved YOLOv8 network and a clip model, which improves the traditional YOLOv8 neural network model. The model has a low computational cost while maintaining high performance, is suitable for real-time application scenarios, and has a better model effect.

[0025] The present invention discloses a method for detecting abnormal pedestrians in subways based on an improved yolov8 network and a clip model, which updates the behavior strategy of individuals by simulating the cloning and mutation operations in the biological immune system and the Q-Learning strategy in reinforcement learning. The immune algorithm guides the search process through the fitness evaluation of individuals, while reinforcement learning enhances the search efficiency by learning the best action selection. The combination of the two realizes the efficient solution of complex optimization problems, and finally continuously approaches the minimum value of the objective function in the iterative process, optimizes the setting threshold of the similarity matching degree in the clip model, improves the accuracy of specific behavior detection, can realize effective retrieval of results, finds the best parameters, is suitable for the fusion application of yolo and clip, facilitates the improvement of the effectiveness and usability of the model, reduces the number of false positives and false negatives, and achieves a balance between precision and recall. From the perspective of use, it reduces the interference with the use of operators and improves the usability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0027] Figure 1 It is a structural diagram of the ghostnet module in the improved yolov8 network according to an embodiment of the present disclosure;

[0028] Figure 2 It is a structural diagram of the c2fhard module in the improved yolov8 network of the embodiment of the present disclosure;

[0029] Figure 3 It is a structural diagram of the SPPF module in the improved yolov8 network according to an embodiment of the present disclosure;

[0030] Figure 4 It is a structural diagram of the detection head part in the improved yolov8 network of the embodiment of the present disclosure;

[0031] Figure 5 is the number of iterations and the curve value of the fitness function of the immune reinforcement learning algorithm of the embodiment of the present disclosure;

[0032] Figure 6 This is a structural diagram of the feature layer of the embodiment of the present disclosure after being input into the improved ghostconv-gn module;

[0033] Figure 7 The structure processing flow chart of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] Example 1

[0038] In one embodiment of the present disclosure, a method for detecting abnormal pedestrians in a subway based on an improved yolov8 network and a clip model is provided, comprising:

[0039] Step 1: Obtain video frames in the subway scene and preprocess them;

[0040] Step 2: Input the preprocessed frame data into the improved yolov8 neural network model to obtain the abnormal behavior target detection frame, use the target tracking algorithm to track the target candidate of the target detection frame, and obtain the candidate image of the target area of ​​the specific behavior;

[0041] Step 3: Input the candidate image containing the target area into the clip model, and input the set detection text into the clip model, map the detection text and the candidate image of the target area, and perform text-image similarity matching. If the probability of similarity matching exceeds the set threshold, an abnormal behavior alarm is issued;

[0042] Among them, the Q-Learning reinforcement learning immune algorithm is used to optimize the set threshold of similarity matching. The search process is guided by the fitness evaluation of individuals in the immune algorithm, and the best action selection is determined by combining reinforcement learning. The final optimal fitness is output as the set threshold.

[0043] As an embodiment, the specific implementation process of the subway pedestrian anomaly detection method based on the improved yolov8 network and clip model disclosed in the present invention is as follows:

[0044] Step 1: Collect pedestrian video frames in the subway scene and preprocess them;

[0045] The camera is used to collect video data of pedestrians in subway scenes and divide it into video frame data. The pixels in the video frame data are filtered. After preprocessing, a subway data set is formed and divided into a training set and a test set.

[0046] Step 2: Build and train the improved yolov8 neural network model to achieve target detection;

[0047] Specifically, the improved yolov8 neural network model is first used to train squatting, standing, bending, falling, sitting and other actions.

[0048] The text label of the action is set to ['squat', 'stand', 'bend', 'fall', 'sit', 'others'] to save the weights of the improved yolov8 neural network, which can distinguish between actions such as falling and other actions. During training, the confidence of the improved yolov8 neural network is reduced so that the target detector can fully detect the falling behavior. Only the algorithm detects the falling among the six behaviors. After the target is detected and tracked, the target is cropped according to a certain area size, and the cropped image is used as the candidate image of the target area of ​​the specific behavior.

[0049] Furthermore, the present invention improves on the basis of the traditional yolov8 neural network model. The present invention improves the network model and architecture, improves and optimizes the backbone network, the SPPF module of the neck, and the detection head model, and realizes the compatibility of the model lightweight and the detection effect. To this end, the fusion of the model accuracy and efficiency can be improved. In the backbone network, the conv module of the original model is adapted to the ghostnet module. Ghostconv is a public module, which is replaced by the original convolution module. The parameters of the model are reduced.

[0050] The traditional yolov8 neural network model generally includes three parts: the backbone network, the neck network and the detection head. The present invention improves each part, improves and optimizes the backbone network, the SPPF module of the neck and the detection head model, and realizes the compatibility of the model lightweight and the detection effect, as follows:

[0051] First, the present invention improves the structure of the backbone network by replacing the conv module of the original neural network model with the ghostnet module, thereby reducing the parameters of the model, and further replacing the c2f module with a lightweight module, such as Figure 1 As shown, one part adds the ghostnet module to the c2f module to obtain the c2fghost module, and the other part combines the improved module to implement the changes of c2f to obtain c2fhard, as shown Figure 2As shown in the figure, in the c2fhard module, two channel branches, ghostconv module and MHSA module, are designed. The two channel branches are fused and connected to the ghostconv module, and then combined with the original c2f to obtain the c2fhard module. This improvement method allows the network model to remain lightweight while maintaining a small drop in accuracy.

[0052] Secondly, the SPPF module of the neck network is improved by adding LSKa convolution box to the original model and performing block processing (two branches), such as Figure 3 As shown. The improvement process is as follows: after ghostconv convolution, the first output is obtained, and then processed by maxpool, hwd (haar wavelet downsampling), and maxpool modules respectively, and then enters two different blocks for processing. The upper two channel data are combined and processed by LSKa, and the lower two channel data are combined and processed by LSKA to improve the feature extraction capability, and then the two branch channels after LSKA processing are fused, and the fusion is connected to the ghostconv module. The fused channel is processed by ghostconv, and the whole process is improved. The ghostconv, maxpool, LSKA and other modules are combined and designed. Under the premise of maintaining lightweight, the feature extraction capability of the module is enhanced, thereby achieving precision and lightweight design architecture.

[0053] Finally, the detection head is improved by adopting the architecture combined with the ghostnet network model and realizing the shared reuse of the convolutional layer. Figure 4 As shown in the figure, after the feature layers p3, p4, and p5 are respectively input into the improved ghostconv-gn module, the parameters are shared and reused, and the regression and classification results are obtained respectively after passing through the two ghostconv-gn modules. After the feature layers p3, p4, and p5 are respectively input into the improved GhostConv-GN module, the parameters are shared and reused. The core of this method is to use the included Ghost module to generate more feature maps, thereby reducing the amount of calculation of the model. The included Ghost module generates additional feature maps from the original feature maps through a series of linear transformations (such as deep convolution). These feature maps are similar to the original feature maps. Figure 1 The two GhostConv-GN modules are used together to increase the expressive power of the network while keeping the computational cost low. After passing through the two GhostConv-GN modules, the model will obtain regression and classification results respectively. The regression result here usually refers to the location and size information of the target, while the classification result refers to the category information of the target. These two results are crucial for target tracking tasks because they help the model determine the location and identity of the target in the video sequence. The specific steps are as follows:

[0054] 1. Feature extraction: First, the model extracts feature layers p3, p4, and p5 from the input image.

[0055] 2. Shared reuse parameters: These feature layers are then input into the GhostConv-GN module, and parameters are shared within the module to reduce the number of model parameters.

[0056] 3. Feature enhancement: Through linear transformation including Ghost module, additional feature maps are generated from the original feature maps to enhance the feature expression ability of the model.

[0057] Regression and classification: Finally, the features processed by the GhostConv-GN module will be used for regression and classification tasks, outputting the location and size information of the target and the category information respectively.

[0058] The detection head is changed to a lightweight detection head, which is a lightweight shared convolution detection head used to predict the bounding box and category probability of the object in the target detection model. The feature map processed by the convolution layer of each detection head will pass through a shared convolution layer, and then the output of the shared convolution layer is added to the original feature map (residual connection), and finally the convolution layer output of the bounding box and category prediction is spliced ​​in the channel dimension.

[0059] The process is as follows:

[0060] 1. The input feature map passes through the convolutional layer of each detection head.

[0061] 2. Feature maps pass through shared convolutional layers.

[0062] 3. Residual connection: add the output of the shared convolutional layer back to the output of the attention mechanism.

[0063] 4. The convolutional layer outputs for bounding box and category predictions are concatenated along the channel dimension.

[0064] The network of the above image adopts a deformable convolution module, a deformable convolution, and a convolution fast operation that integrates convolution normalization activation and attention mechanism.

[0065] 1. Convolution layer (self.conv): use ghostconv convolution

[0066] 2. Group normalization layer (self.gn): Use GroupNorm with a group size of 16.

[0067] 3. Activation function (self.act): nn.eiLU() (exponential linear unit) is used by default, but it can be customized through the constructor parameter act.

[0068] 4. Attention mechanism (self.xi): using SELayer.

[0069] It extracts features through a series of convolutional layers and a specific distance focal loss (DFL) layer, and outputs the prediction results of each detection layer. The lightweight is achieved mainly through the following aspects:

[0070] 1) Shared convolutional layer (share_conv): By sharing convolution operations between multiple detection layers, the number of parameters and computational complexity of the model are reduced.

[0071] 2) Distance Focal Loss (DFL): Using the DFL layer instead of the traditional bounding box regression method can reduce parameters and simplify the calculation process.

[0072] The DFL class implements a special loss function for object detection tasks to improve the model's ability to identify small and difficult-to-detect objects. It converts the four coordinates of each anchor point in the input feature map (the upper left and lower right coordinates of the bounding box) into a probability distribution through a 1x1 convolutional layer. The weights of this convolutional layer are initialized to a linear sequence corresponding to the index of the input channel. During the forward propagation process, the input feature map is reshaped and normalized by the softmax function so that each of its coordinates is represented as a probability distribution. This distribution is then processed by a convolutional layer to aggregate the coordinate information of each anchor point, and finally outputs a tensor of shape (b,4,a), where b is the batch size, 4 represents the four coordinates of the bounding box, and a is the number of anchor points. This process helps the model learn the representation of the target more effectively, thereby improving the detection performance.

[0073] The DFL (Distributed Focal Loss) loss function is a relatively mature loss function.

[0074] DFL(Si,Si+1)=-((yi+1-y)log(Si)+(y-yi)log(Si+1))

[0075] Among them, yi and yi+1 are the predicted bounding box values, y is the actual bounding box value, Si and Si+1 are the probability distributions corresponding to yi and yi+1. The specific process is as follows:

[0076] 1. Predicted bounding box values: The model predicts the positions of the bounding boxes, which are represented as probability distributions, namely yi and yi+1.

[0077] 2. Calculate the probability distribution: For each predicted bounding box value, calculate its corresponding probability distribution Si and Si+1.

[0078] 3. Apply the DFL formula: Calculate the loss using the above DFL formula, where log represents the natural logarithm

[0079] 4. Loss calculation: The DFL loss function is used in combination with the official CIou Loss to simultaneously optimize the position and size of the bounding box.

[0080] 3) Convolutional layer (conv): Use ghostconv convolution, group normalization and 1x1 convolution to reduce parameters and computation while maintaining the effectiveness of feature extraction.

[0081] 4) Anchors and strides: By optimizing the settings of anchors and strides, redundant calculations during prediction can be reduced.

[0082] In the target detection model, the following strategies are adopted to reduce redundant calculations during prediction by optimizing the settings of anchor points and step sizes:

[0083] 1. Dynamic anchor adjustment: If the model supports dynamic anchors (self.dynamic), the anchors can be dynamically adjusted according to the shape of the input feature map (shape). This means that the anchors and strides (self.anchors, self.strides) are recalculated only when the shape of the input feature map changes, thus avoiding unnecessary repeated calculations. This method can reduce the repeated calculation of anchors during model inference, which is especially useful when dealing with feature maps of different sizes.

[0084] 2. Multi-scale anchors: Use anchors of different scales in different feature layers. This ensures that objects can be effectively detected on feature maps of different scales while reducing the calculation of irrelevant scales.

[0085] 3. Step size optimization: The step size (self.stride) determines the density of anchor points generated on the feature map. Properly increasing the step size can reduce the number of anchor points on the feature map, thereby reducing the amount of calculation.

[0086] 4. Anchor selection: By statistically analyzing the size and shape of the objects in the training dataset, the most representative anchors are selected. This can reduce unnecessary anchor calculations because the model will focus on those anchors that are most likely to contain the object.

[0087] Through the above method, the setting of anchor points and step sizes can be effectively optimized, the redundant calculation of the target detection model during prediction can be reduced, and the efficiency and performance of the model can be improved.

[0088] 5) Bias initialization: Through a specific bias initialization method, the initial performance of the model is optimized and the adjustment time in the early stage of training is reduced.

[0089] Initializing the bounding box regression task to 1.0 helps the model learn the location of the bounding box faster because in some cases the width and height of the bounding box are often close to 1 in the initial stage.

[0090] 6) Dynamic and static shape processing: Adapt to images of different input sizes by dynamically adjusting or pre-calculating anchor points and strides while reducing the computational overhead at runtime.

[0091] Furthermore, the improved yolov8 neural network model can fully detect abnormal behaviors. When detecting abnormal behaviors (such as falling, climbing over gates, climbing over fences, fighting, etc. or other abnormal behaviors), it can improve efficiency through multi-threaded processing, and then track the detected targets through tracking algorithms to further obtain target areas, including:

[0092] 1. Use the tracking configuration file specified by self.track_config, which may contain parameters of the tracking algorithm, such as the maximum IoU change, to maintain the consistency of the target between consecutive frames.

[0093] 2. Filter detection results: The detected target box is filtered according to the confidence (conf) and IoU (iou) thresholds. The confidence threshold is used to determine the credibility of the detection result, while the IoU threshold is used to determine the degree of match between the detection box and the real target.

[0094] 3. Extract target box information: The .boxes attribute in the detection result contains detailed information about the target box, such as boundary coordinates, target ID, confidence, and category. This information is converted into a list and assigned to frame.boxes.

[0095] 4. Update target ID record: Using the id_update method, the program will update the self.id_record dictionary based on the ID and timestamp in the target frame. This dictionary records the last time each target ID appeared. If target frames with the same ID are detected and the time interval is less than the set threshold, it is considered that this is a continuous appearance of the same target and the alarm will not be triggered repeatedly.

[0096] Among them, the present invention adopts the botsort target tracker to realize the tracking of the target. After the improved yolov8 neural network model takes the abnormal behavior target detection frame, the target tracking algorithm is used to track the target candidate of the target detection frame, and the target area of ​​the specific behavior (fall) is obtained to realize the acquisition and tracking of the target.

[0097] Step 3: Use the improved clip model to filter behaviors and implement early warning for specific behaviors.

[0098] As an embodiment, the tracking algorithm may be botsort or bytetrack, etc. The purpose of these tracking algorithms is to track the current target and determine the ID number. When the same behavior occurs with the same ID number within a period of time, multiple warnings will not be issued, thus achieving the goal and effect of filtering the number of warnings.

[0099] 1. Training of the clip model

[0100] Specifically, 1) the detection results finally output by the improved yolov8 neural network model and botsort target tracking are framed, and the detected targets are expanded to 1, 1.1, 1.2, 1.7, 2 and 3 times of the original target with the center as the initial position, and the target area is further cropped out, and these cropped images are converted into RGB format and then converted into PIL image objects to obtain candidate images of the target area, and then the corresponding text labels are marked on these data, such as falling needs to be marked with falling, someone fell, fell on the floor, fell on the stairs, etc.

[0101] 2) Map the text labels and candidate images obtained in step 1), use the clip model to compare the text-image similarity, and for a single image, find the text with the greatest similarity to the image as the text label of the image.

[0102] 3) Traverse steps 1) and 2) to obtain all the image-text pairs of the data and form a dataset.

[0103] 4) Output the data set obtained in step 3) to clip for fine-tuning, and save the weight model after fine-tuning for actual testing.

[0104] Furthermore, the specific process of training is:

[0105] 1) Load the pre-trained clip model.

[0106] 2) Freeze early layers and freeze Transformer layers: In the clip model, the early layers usually learn low-level features of the image, such as edges and textures, which are common in most visual tasks. Because the dataset is small, it is necessary to freeze the Transformer layers in the model to avoid overfitting.

[0107] 3) Modify the final classification layer to adapt to the number of categories of detection behavior.

[0108] 4) Define the cross entropy loss function as the loss function. Here, contrast loss is selected as the loss function of the model.

[0109] 5) Set up the optimizer, choosing an appropriate learning rate and other parameters (such as Adam or SGD).

[0110] 6) Train the model while monitoring the performance on the validation set to avoid overfitting.

[0111] As an embodiment, a candidate image containing a target area is input into a clip model, and a set detection text is input into the clip model, the detection text and the candidate image of the target area are mapped, and text-image similarity matching is performed. If the probability of similarity matching exceeds a set threshold, an abnormal behavior alarm is performed, which is specifically divided into the following steps:

[0112] 1. Initialize the CLIP model: In the constructor of the ClimbingDetection class, a CLIP model is initialized through modelscope.pipelines for the multimodal embedding task. This model is configured to use a specific pre-trained model and version for processing images and text.

[0113] 2. Extract text embeddings: In the class constructor, use the forward method of the CLIP model to convert the predefined text list (self.input_texts) into text embeddings (self.text_embedding), that is, input the set detection text in the clip model. These text embeddings are then used to match with the image embeddings.

[0114] 3. Image processing and feature extraction: In the task method, for each detected target box, the target area is cropped from the original video frame using OpenCV and PIL libraries, and these cropped images are converted to RGB format and then converted to PIL image objects to obtain candidate images of the target area;

[0115] 4. Get image embedding: Pass the candidate image as input to the forward method of the CLIP model to get the image embedding (img_embedding). This step is the core of multimodal matching. The model converts the image content into a numerical vector that can be compared with the text embedding.

[0116] 5. Calculate the matching degree: Use PyTorch's matrix operations to calculate the inner product of the image embedding and the text embedding to obtain the matching score (logits) of each image and each preset text. Then convert these scores into probability distributions through the softmax function.

[0117] Specifically, inner product calculation (calculating matching score logits): Assume that I is the image embedding vector and Tc is the text embedding vector corresponding to category cc. Their inner product can be expressed as:

[0118] logitsc=I.Tc

[0119] Here, . represents the dot product operation of the vectors.

[0120] If there are NN categories, we can compute a score vector S of length N for each image:

[0121] S=[I·T1,I·T2,…,I·TN]S=[I·T1,I·T2,…,I·TN]

[0122] Softmax function (converts score to probability distribution): Use the softmax function to convert the score to probability distribution. For each element si in the score vector S, the converted i-th probability p is:

[0123]

[0124] Here, e si Indicates i Exponentiation, where the denominator is the sum of all the score exponents, ensures that the probability distribution sums to 1.

[0125] Applying this to the entire score vector S, we get the probability distribution vector P:

[0126] P = softmax(S)

[0127] Thus, every element p in P i It represents the probability that the image belongs to the corresponding category i. In this way, a category probability distribution can be assigned to each image, and then the most likely category can be selected as the final classification result.

[0128] 6. Decision logic: Based on the probability distribution, the code implements the logic to decide whether to regard the currently detected target as a behavior that requires an alarm by comparing the sum of the probabilities that match a specific target text (such as falling down, lying on the ground, etc.) with the sum of the probabilities of the remaining texts.

[0129] 7. Update results: According to the analysis results of the clip model, update the boxes attribute of the video frame, retain those target boxes that have a high degree of match with the preset text, and use the plot_boxes_with_text_for_yolotrack function to draw these boxes on the video frame, filter other behaviors, and recheck the fall behavior.

[0130] 8. Multimodal matching and alarm logic, suitable for scenarios that require real-time monitoring and response to specific behaviors (such as fall detection).

[0131] Furthermore, the judgment rule for the secondary review of the clip model is:

[0132] self.target_texts = 8, the target text is the first 8, of course, you can also choose other numbers as the judgment number.

[0133] The method adopted by the present invention is to calculate the output probability of the first n values ​​when judging, and then normalize them, and sum the first n values. If the probability after the sum is greater than the remaining probability, then it is judged that the behavior belongs to the behavior of yolo judgment, otherwise it is considered not to be the behavior of yolo judgment.

[0134] At the same time, other methods can be selected in the judgment rules. For example, if the probability after the first n judgment results is greater than the threshold, then the yolo judgment behavior is considered correct, otherwise the yolo judgment behavior is considered wrong.

[0135] As an embodiment, the detected target area is fine-tuned to achieve more accurate target detection, so that the IOU value between the detected target and the real target is maximized, and the target area can be selected more accurately. The main methods are:

[0136] 1) Get the performance index of the original data sample, count the aspect ratio of each category, cluster all the aspect ratios, and use the kmeans algorithm to cluster them into n categories.

[0137] 2) When the target data is input into the clip, its center point and original length and width data are obtained, and the clustering result of the above step 1) is called. Taking its center point as an example, more sample data is intercepted for subsequent model experimental analysis.

[0138] 2.1) Based on the original length of 1 times, 0.9 times, 0.8 times, 1.1 times, 1.2 times and other scaling multiples, configure n aspect ratio data to obtain a new target frame A. 2.2) Based on the original width of 1 times, 0.9 times, 0.8 times, 1.1 times, 1.2 times and other scaling multiples, configure n aspect ratio data to obtain a new target frame B.

[0139] In particular, the number of new target frames is set according to the text-image matching degree of step 3. When the matching degree is small, the number and scaling ratio are increased, and when the matching degree is large, the number and scaling ratio are reduced. Specifically, the number and scaling ratio of new target frames are dynamically adjusted according to the calculated matching degree. If the matching degree is small, it means that the similarity between the text and the image is low. At this time, the number and scaling ratio of new target frames can be increased to increase the search space and possibility. If the matching degree is large, it means that the similarity is high. The number and scaling ratio of new target frames can be appropriately reduced to improve efficiency. Set a threshold. When the matching degree is lower than the threshold, the above-mentioned increase strategy is adopted; when the matching degree is higher than the threshold, the reduction strategy is adopted.

[0140] The specific, improved examples are as follows:

[0141] To generate a new target frame A based on the length and width of the original target frame by different scaling factors, follow these steps:

[0142] 1. Determine the length and width of the original target box:

[0143] Assume that the width of the original target box is \(w\), the height is \(h\), and the aspect ratio is 1.

[0144] 2. Define the zoom factor:

[0145] Define a series of scaling factors, such as \(s_1,s_2,...,s_n\), where \(s_1=1\), \(s_2=0.9\), \(s_3=0.8\), \(s_4=1.1\), \(s_5=1.2\), etc.

[0146] 3. Calculate the new target box size:

[0147] For each zoom factor s_i, the width w_i and height h_i of the new target box A can be calculated by the following formula:

[0148] w_i=s_i*w

[0149] h_i=s_i*h

[0150] 4. Form a new target frame A:

[0151] The new target box A can be expressed as \((w_i,h_i)\).

[0152] 5. Repeat the above steps:

[0153] Repeat the above steps for each scaling factor \(s_i\) to obtain \(n\) new target boxes.

[0154] In this way, a series of target boxes of different sizes can be generated, which can be used to train the model to improve its robustness to changes in target size, or to simulate the size changes of target boxes in different scenarios.

[0155] For example, if there are 5 scaling factors s1=1, s2=0.9, s3=0.8, s4=1.1, s5=1.2, and the width of the original target box is w=100 and the height is h=200, then the new target box size is:

[0156] In this way, 5 new target boxes are obtained, and the size of each box is calculated based on the original size through different scaling factors.

[0157] 3) The images corresponding to the generated A and B target boxes are cut out and input into the CLIP model. The score of the CLIP model is recorded. The target box corresponding to the maximum score is recorded as the final fine-tuning result.

[0158] 4) The target box obtained in the above steps is finally selected, and the image is captured and sent to the terminal of the relevant user.

[0159] Preferably, the improved model structure of the method proposed in the present disclosure can be extended and copied to

[0160] In YOLOV9, YOLOV10, YOLOV11, YOLOV3-YOLOV7, the present invention adopts the CLIP model to realize the secondary confirmation of YOLO captured images, thereby enhancing the effectiveness and usability of the model.

[0161] Step 4: Immune reinforcement learning algorithm optimizes the prediction of threshold

[0162] There are many non-fixed parameters in this disclosure, such as the confidence in the target detector, the iou threshold, match_thresh: (the threshold for matching trajectories, which determines whether two detection boxes belong to the same trajectory), and the threshold for clip filtering. It can be considered that the selection of these parameters can be determined by optimization algorithms, such as genetic algorithms, particle swarm algorithms, immune algorithms, grid algorithms, etc. The determination of these algorithms can lay the foundation for finding the best parameter values. This disclosure uses a combination of immune algorithms and reinforcement learning to achieve threshold optimization, and the process is as follows:

[0163] The present invention integrates an immune algorithm with a Q-Learning reinforcement learning strategy to minimize the objective function. The algorithm initializes a population and evaluates the fitness of individuals in each generation. It uses a Q-Learning update strategy to select clone or mutation operations in order to gradually improve the quality of the solution during the iteration process. Finally, the algorithm converges to a set of individuals close to the optimal solution, where the individual with the minimum fitness value is considered to be the solution to the optimization problem, and its corresponding value confidence, iou threshold, and clip threshold when filtering are the minimum points of the function.

[0164] Initialize YOLO hyperparameter confidence threshold, YOLO hyperparameter IOU threshold, text information, CLIP text information number, CLIP judgment threshold and other parameters. Among them, the confidence threshold and IOU threshold are the configuration parameters required by YOLO to detect the target. The confidence threshold is used to determine the credibility of the detection result, and the IoU threshold is used to determine the degree of match between the detection box and the real target. For example, a confidence of 0.3 means that only when the confidence of the detected target is greater than or equal to 0.3, the result will be considered credible; the text information is the text embedding list of CLIP, such as "fall, stand up", etc. In this way, the optimal point of the objective function can be obtained, and the objective function that can be adopted is the precision rate, recall rate, F1 index, etc. of the data set.

[0165] The core concepts of immune algorithm and reinforcement learning algorithm are integrated, and the individual behavior strategy is updated by simulating the cloning and mutation operations in the biological immune system and the Q-Learning strategy in reinforcement learning. The immune algorithm guides the search process through the fitness evaluation of the individual, while reinforcement learning enhances the search efficiency by learning the best action selection. The combination of the two realizes the efficient solution of complex optimization problems, and finally continuously approaches the minimum value of the objective function in the iterative process, reflecting a hybrid optimization strategy that simulates the natural immune system and reinforcement learning decision-making.

[0166] The implementation process is as follows:

[0167] 1. Initialization: Generate an initial population, randomly distribute it in the solution space, and initialize the Q table required for Q-Learning.

[0168] 2. Fitness evaluation: Calculate the fitness of each individual, which is inversely proportional to the objective function value.

[0169] 3. Select operation: Based on the Q-Learning strategy, choose to perform cloning or mutation operations. Each individual selects the best action based on its current Q value.

[0170] 4. Perform operations: clone or mutate the selected individuals to generate new individuals or modify existing individuals.

[0171] 5. Reward calculation: Calculate rewards based on the results of the operation, usually an increase in fitness.

[0172] 6. Q-Learning Update: Update the values ​​in the Q table based on the actions performed and the rewards obtained.

[0173] 7. Iteration: Repeat the above process until the maximum number of iterations is reached or a satisfactory solution is found.

[0174] 8. Result output: Output the best individual finally found and its fitness.

[0175] Among them, the Q-Learning update rule is:

[0176] ((sa((sa

[0177] Among them, Q(s,a) is the Q value of the current state-action pair; α is the learning rate; r is the immediate reward obtained after performing the action; γ is the discount factor; maxa′Q(s′,a′) is the maximum Q value of all possible actions in the next state.

[0178] The optimal threshold obtained by optimization is used as the parameter of the model to improve the accuracy of target detection and filtering behavior.

[0179] The combination of reinforcement learning and immune algorithm in this disclosure has the following advantages:

[0180] 1) Adaptive search: Reinforcement learning can adaptively adjust the search strategy based on historical information, while the immune algorithm provides a search mechanism that simulates the biological immune system. The combination of the two can more effectively explore the solution space.

[0181] 2) Global and local search: Immune algorithms are good at global search, while reinforcement learning can focus on promising areas for local search, improving search efficiency.

[0182] 3) Dynamic adjustment: The Q-Learning strategy in reinforcement learning can dynamically adjust the behavior of individuals based on their performance. The cloning and mutation operations in the immune algorithm can also be adjusted based on environmental feedback.

[0183] 4) Diversity maintenance: The diversity maintenance mechanism of the immune algorithm can avoid the premature convergence problem in RL and ensure that the algorithm can continue to explore new solutions.

[0184] 5) Robustness: Combining the two algorithms can improve the robustness of the overall optimization process and maintain stable performance even when facing complex and dynamic optimization problems.

[0185] 6) The parameters of the reinforcement learning model and immune algorithm can be customized according to the characteristics of the specific problem to achieve more efficient optimization.

[0186] Example 2

[0187] In one embodiment of the present disclosure, a subway pedestrian anomaly detection system based on an improved yolov8 network and a clip model is provided, comprising:

[0188] The data acquisition module is used to acquire video frames in subway scenes and pre-process them;

[0189] The target detection module is used to input the preprocessed frame data into the improved yolov8 neural network model, obtain the abnormal behavior target detection frame, use the target tracking algorithm to track the target candidate of the target detection frame, and obtain the candidate image of the target area of ​​the specific behavior;

[0190] A review matching warning module is used to input the candidate image containing the target area into the clip model, and input the set detection text into the clip model, map the detection text and the candidate image of the target area, and perform text-image similarity matching. If the probability of similarity matching exceeds a set threshold, an abnormal behavior alarm is issued;

[0191] Among them, the Q-Learning reinforcement learning immune algorithm is used to optimize the set threshold of similarity matching. The search process is guided by the fitness evaluation of individuals in the immune algorithm, and the best action selection is determined by combining reinforcement learning. The final optimal fitness is output as the set threshold.

[0192] Example 3

[0193] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the subway pedestrian anomaly detection method based on the improved yolov8 network and clip model is implemented.

[0194] Example 4

[0195] In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the subway pedestrian anomaly detection method based on the improved yolov8 network and clip model.

[0196] Example 5

[0197] In one embodiment of the present disclosure, a method for detecting abnormal pedestrians in subways based on an improved yolov8 network and a clip model is provided, wherein the network architecture based on YOLov8 disclosed in the present disclosure can also embed its improved network architecture into the original inherent network of YOLOV9, YOLOV10, YOLOV11, YOLOV3-YOLOV7. In particular, the present disclosure adopts the CLIP model to realize the secondary confirmation of the YOLO intercepted image, which enhances the effectiveness and usability of the model. In particular, CLIP can be replaced with GLIPV1, glipV2, Yolo-world and other models, which can be considered as the same method. The specific method includes:

[0198] Example 1 Fusion detection method of Yolo and AlphaClip

[0199] First, use yolo for target detection, set a certain iou threshold and confidence threshold, detect the target for the first time, cut out the target area, get the corresponding image area a, segment the area through yolo-world or sam or yolo-seg, get a new dimension ss, combine a and ss dimensions, and send them to alphaclip for detection. If the target is obtained, then early warning treatment is carried out, otherwise no early warning treatment is required. Among them, yolo-world or sam or yolo-seg and other methods can set a threshold cc for segmentation to get the final ss data. Specifically, the entire network contains five parameters: iou threshold, confidence threshold, segmentation threshold cc, number of text matches n, and a. According to the existing algorithm, it is input into the reinforcement immune learning to get the best parameters. Of course, the fitness function that can be designed includes three types: precision, recall, and F1. Finally, the best parameter configuration is obtained, and then fusion detection is performed to get the best results.

[0200] Example 2 Fusion method of yolo and GLIp.

[0201] Among them, yolo can choose yolov3, v4, v5, v6, v7, v8, v9, v10, v11, yolo-detr (detection head replaced by rtdetr detection head), yolo-world, rtdetr, mamba-yolo (Mamba-YOLO is a YOLO target detection algorithm based on the state-space model (SSM). It aims to optimize the performance of target detection by introducing new modules and structures.) and other algorithms. After the area is initially obtained, the image is intercepted and input into GLIP. GLip makes a judgment. If the result is the set target and the target width and height are both greater than 0.95, then an early warning is issued. Otherwise, no early warning is taken. The optimal method for finding other parameters adopts the immune reinforcement learning algorithm, the method of taking screenshots of images, and the method of adjusting the target frame of the target to be finally obtained are designed and implemented according to the yoloclip algorithm.

[0202] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. The subway pedestrian anomaly detection method based on the improved yolov8 network and clip model is characterized by: include: Obtain video frames in subway scenes and preprocess them; The preprocessed frame data is input into the improved yolov8 neural network model to obtain the abnormal behavior target detection frame, and the target tracking algorithm is used to track the target candidate of the target detection frame to obtain the candidate image of the target area of ​​the specific behavior; Input the candidate image containing the target area into the clip model, input the set detection text into the clip model, map the detection text and the candidate image of the target area, perform text-image similarity matching, and if the probability of similarity matching exceeds the set threshold, issue an abnormal behavior alarm; Among them, the Q-Learning reinforcement learning immune algorithm is used to optimize the set threshold of similarity matching. The search process is guided by the fitness evaluation of individuals in the immune algorithm, and the best action selection is determined by combining reinforcement learning. The final optimal fitness is output as the set threshold.

2. The subway pedestrian anomaly detection method based on the improved yolov8 network and clip model as claimed in claim 1 is characterized in that: The improved yolov8 neural network model includes improving the backbone network, replacing the conv module of the original model with a ghostnet module; and replacing the c2f module with a lightweight module, a part of which is a c2fghost module obtained by adding the ghostnet module to the c2f module, and the other part is a c2fhard module; in the c2fhard module, two channel branches, a ghostconv module and an MHSA module, are designed, the two channel branches are fused and connected to the ghostconv module, and then combined with the original c2f to obtain the c2fhard module.

3. The subway pedestrian anomaly detection method based on the improved yolov8 network and clip model as claimed in claim 1 is characterized in that: The improved yolov8 neural network model includes improving the SPPF module, adding an LSKa convolution box to the original model structure, and performing two branch processing. After ghostconv convolution, the first output is obtained, and after maxpool, hwd and maxpool processing, the LSKA algorithm is performed on the two branches respectively, and then the two branch channels after LSKA processing are fused, and the fusion is connected to the ghostconv module.

4. The subway pedestrian anomaly detection method based on the improved yolov8 network and clip model as claimed in claim 1 is characterized in that: The improved yolov8 neural network model includes improvements to the detection head, adopts an architecture combined with the ghostnet network model, and realizes the shared reuse of the convolutional layer. After the feature layers p3, p4, and p5 are respectively input into the ghostconv-gn module, the parameters are shared and reused, and the bounding box and category probability are output after passing through two ghostconv-gn modules.

5. The subway pedestrian anomaly detection method based on the improved yolov8 network and clip model as claimed in claim 1 is characterized in that: A candidate image containing a target area is input into a clip model, and a set detection text is input into the clip model, the detection text and the candidate image of the target area are mapped, and text-image similarity matching is performed, including: extracting text embedding for the set detection text, for each detected target frame, using OpenCV and PIL libraries to crop the target area from the original video frame and convert it into a PIL candidate image, inputting the candidate image into the clip model, extracting the image embedding of the candidate image, using PyTorch's matrix operation, calculating the inner product of the image embedding and the text embedding, obtaining the matching score of each candidate image and each preset text, and converting these scores into probability distributions through a softmax function.

6. The subway pedestrian anomaly detection method based on the improved yolov8 network and clip model as claimed in claim 5 is characterized in that: According to the probability distribution, by comparing the sum of the probabilities of matching a specific target text with the sum of the probabilities of the remaining texts, it is decided whether to regard the currently detected target as a behavior that requires an alarm.

7. The subway pedestrian anomaly detection method based on the improved yolov8 network and clip model as claimed in claim 1 is characterized in that: The Q-Learning reinforcement learning immune algorithm is used to optimize the setting threshold of similarity matching, including: 1) Initialization: Generate an initial population, randomly distribute it in the solution space, and initialize the Q table required for Q-Learning; 2) Fitness evaluation: Calculate the fitness of each individual, which is inversely proportional to the value of the objective function; 3) Operation selection: Based on the Q-Learning strategy, the clone or mutation operation is selected, and each individual selects the best action according to its current Q value; 4) Execute operations: clone or mutate the selected individuals to generate new individuals or modify existing individuals; 5) Reward calculation: Calculate rewards based on the results of the operation, usually an increase in fitness; 6) Q-Learning update: Update the values ​​in the Q table based on the actions performed and the rewards obtained; 7) Iteration: Repeat the above process until the maximum number of iterations is reached or a satisfactory solution is found; 8) Result output: Output the best individual finally found and its fitness.

8. The subway pedestrian anomaly detection system based on the improved yolov8 network and clip model is characterized by: include: The data acquisition module is used to acquire video frames in subway scenes and pre-process them; The target detection module is used to input the preprocessed frame data into the improved yolov8 neural network model, obtain the abnormal behavior target detection frame, use the target tracking algorithm to track the target candidate of the target detection frame, and obtain the candidate image of the target area of ​​the specific behavior; A review matching warning module is used to input the candidate image containing the target area into the clip model, and input the set detection text into the clip model, map the detection text and the candidate image of the target area, and perform text-image similarity matching. If the probability of similarity matching exceeds a set threshold, an abnormal behavior alarm is issued; Among them, the Q-Learning reinforcement learning immune algorithm is used to optimize the set threshold of similarity matching. The search process is guided by the fitness evaluation of individuals in the immune algorithm, and the best action selection is determined by combining reinforcement learning. The final optimal fitness is output as the set threshold.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the subway pedestrian anomaly detection method based on the improved yolov8 network and clip model as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the subway pedestrian anomaly detection method based on the improved yolov8 network and clip model as described in any one of claims 1-7.

Images