Pedestrian tumble detection method based on NanoDet-Plus and fine-tuning CLIP

Abstract

The invention discloses a pedestrian tumble detection method based on NanoDet-Plus and fine tuning CLIP. The method comprises the following steps: S1, collecting data to make a data set; s2, a NanoDet-Plus target detection model is trained, and a trained algorithm model weight file is obtained; s3, performing pedestrian recognition on the image by using a target detection model NanoDet-Plus, and independently storing the recognized pedestrian image as an image file; s4, training the CLIP model and the improved Resnet50 model; s5, performing preliminary classification prediction on the pedestrian attitude image by using a CLIP model, performing further classification prediction by using an improved Resnet50 model, and connecting two classification prediction results so as to obtain a pedestrian attitude prediction result; and judging the current state of the pedestrian according to a posture prediction result, and taking feedback measures. And the recognition accuracy of the pedestrian falling event in the complex scene is improved.

Description

Technical Field

[0001] The present invention belongs to the field of target detection and multimodal technology, and in particular relates to a pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP. Background Art

[0002] At present, pedestrian fall detection mainly uses YoloMD, Yolo-v5 and Yolo-v3 models to detect pedestrian fall events. The core of these solutions lies in data collection and model training. Specifically, researchers will widely collect image or video data containing pedestrian falls and normal walking, and make them into detailed data sets. Then, these data sets are used to deeply train the above models to detect pedestrian falls.

[0003] First, traditional closed-set object detection and image classification algorithms lack effective semantic understanding capabilities when processing complex images, resulting in insufficient adaptability to the diversity of pedestrian postures and scene complexity, thus affecting recognition accuracy. Second, the model generalization ability of existing methods is poor, making it difficult to cope with pedestrian fall detection tasks in different environments, reducing their practicality and flexibility. Finally, traditional methods perform poorly in terms of posture changes and scene diversity, and often require a lot of debugging and optimization.

[0004] Chinese patent document CN 114724070A discloses a YoloMD pedestrian fall detection method suitable for edge devices, including collecting data and generating a data set, and also including the following steps: performing data standardization processing on the data in the data set; training the fall detection model using the data in the training set; testing and verifying the trained fall detection model using the data in the validation set and the test set; and outputting the prediction results. This technical solution is to use the optimized YOLO-v5 model to train with a specific data set and detect pedestrian falls. This solution still belongs to closed set target detection, and still faces the problems of low recognition accuracy, poor model generalization ability, and insufficient practicality and flexibility in a variety of scenarios in complex traffic environments.

[0005] Therefore, it is necessary to provide a pedestrian fall detection method based on NanoDet-Plus and fine-tuned CLIP. The pedestrian fall detection algorithm is more flexible and efficient to improve the recognition accuracy of pedestrian fall events in complex scenes. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP to improve the recognition accuracy of pedestrian fall events in complex scenes.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: the pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP specifically includes the following steps:

[0008] S1: collect data and create datasets;

[0009] S2: Train the NanoDet-Plus target detection model and obtain the trained algorithm model weight file;

[0010] S3: Collect the image to be predicted and input it into the target detection model; use the target detection model NanoDet-Plus to perform pedestrian recognition on the image, and save the recognized pedestrian image separately as an image file;

[0011] S4: Training CLIP model and improved Resnet50 model;

[0012] S5: Use the CLIP model to analyze the image files one by one, that is, perform preliminary classification prediction on the pedestrian posture image, and then use the improved Resnet50 model to perform further classification prediction, connect the classification prediction results obtained by the two models, and thus obtain the pedestrian posture prediction result; determine the current state of the pedestrian according to the posture prediction result, and take corresponding feedback measures.

[0013] Using the above technical solution, the method introduces the CLIP model, and uses its powerful zero-shot classification ability to infer the posture of pedestrians by providing appropriate prompt words (such as "a pedestrian in red clothes fell down"), which is a capability that traditional target detection models and classification models do not have. Since the CLIP model has been trained with a large number of image-text pairs and has excellent image semantic understanding capabilities, even without additional training of the CLIP model, the prompt words can be flexibly set according to specific scenarios, thereby significantly improving the practicality and flexibility of the model. In order to further enhance the generalization ability and recognition accuracy of the fall detection model, NanoDet-Plus is used to identify the information of each pedestrian in the predicted image, and the fine-tuned CLIP model is used to classify and predict this information. Specifically, an improved image classification model Resnet50 is introduced; by creating a dataset containing rich pedestrian postures and deeply training the classification model, the prediction results of the Resnet50 model are residually connected with the zero-shot classification prediction results of the CLIP model, so as to fine-tune the CLIP model to comprehensively predict whether the pedestrian falls down, thereby improving the generalization ability and recognition accuracy of the model.

[0014] Preferably, in step S1, Pycharm is used to crawl pedestrian posture image data of different categories, and at the same time, the collected pedestrian video is frame sampled by the ffmpeg tool to obtain the pedestrian posture image data; the specific steps are:

[0015] S11: Analyze the loading methods of web pages on various websites, check the robots.txt protocols of web pages on various websites; write scripts to collect pedestrian posture image data based on the analysis results;

[0016] S12: Use the ffmpeg tool to perform frame sampling on the collected pedestrian video, and select image data containing the pedestrian posture;

[0017] S13: Manually screen the collected pedestrian posture data to remove poor quality image data, such as image data that does not meet the definition of the detection task; obtain data to create a data set.

[0018] Preferably, the specific steps of step S2 are:

[0019] S21: Segment the collected pedestrian posture image data, use the X-AnyLabeling tool to intelligently label the collected images, and conduct manual review to ensure that the target objects in each image are accurately labeled;

[0020] S22: After the labeling is completed, the data set is divided into a training set and a validation set to facilitate subsequent model training and evaluation; the data set includes different categories of pedestrian posture images in different scenes that need to be detected; the labeled data set is divided into two types, namely, a pedestrian data set and a pedestrian posture data set including pedestrian falls;

[0021] S23: Use the training set to train the NanoDet-Plus model, set the number of training rounds, and obtain the algorithm model weight file best.pt;

[0022] S24: Use the validation set to evaluate the trained NanoDet-Plus model.

[0023] Preferably, the evaluation indicators in step S24 include accuracy, average precision (AP), recall, and F1 score;

[0024] The accuracy calculation formula is:

[0025]

[0026] Among them, Accuracy is expressed as accuracy, TP (True Positive) means true positive, which means the number of samples that are actually positive and correctly predicted as positive by the model; TN (True Negative): true negative, which means the number of samples that are actually negative and correctly predicted as negative by the model; FP (False Positive): false positive, which means the number of samples that are actually negative but are incorrectly predicted as positive by the model (false positive); FN (False Negative): false negative, which means the number of samples that are actually positive but are incorrectly predicted as negative by the model (false negative);

[0027] The formula for calculating the average precision is:

[0028]

[0029] Among them, Precision is expressed as the average precision, TP (True Positive) represents true positive examples, which means the number of samples that are actually positive and correctly predicted as positive by the model; FP (False Positive) represents false positive examples, which means the number of samples that are actually negative but are incorrectly predicted as positive by the model (false positives);

[0030] The recall calculation formula is:

[0031]

[0032] Among them, Recall represents the recall rate, TP (True Positive) represents the true positive example, which means the number of samples that are actually positive and correctly predicted as positive by the model; FN (False Negative) represents the false negative example, which means the number of samples that are actually positive but are incorrectly predicted as negative by the model (missed reports);

[0033] The F1 calculation formula is:

[0034]

[0035] Among them, Fmeasure is expressed as F1, Precision is the average precision, and Recall is the recall rate.

[0036] Preferably, the specific steps of step S3 are:

[0037] S31: According to the deployed cameras, the area to be detected is monitored in real time, and continuous image acquisition is performed to obtain the image to be predicted;

[0038] S32: Load the model weight file best.pt and the model configuration file default.yaml, use the NanoDet-Plus algorithm to analyze the image to be detected, and obtain the rectangular frame position area of ​​the pedestrian.

[0039] Preferably, the specific steps of analyzing the image P to be detected of 1000×750 pixels by using the NanoDet-Plus algorithm in step S32 are as follows:

[0040] S321: First, the size of the image P to be detected is adjusted to the input size required by the NanoDet-Plus model, which is 640×640 pixels, so that the image P to be detected meets the input requirements of the model;

[0041] S322: normalize the resized image and scale the pixel values ​​to the range of [0, 1] so that the model can process it effectively;

[0042] S323: The NanoDet-Plus model is used to recognize the image. During the detection process, for each pedestrian recognized, the predicted frame information of the pedestrian is extracted, and the predicted frame information of each pedestrian is saved as an independent picture for backup and subsequent processing; if no pedestrian is recognized, the detection is stopped.

[0043] Preferably, the specific steps of step S4 are:

[0044] S41: Training the CLIP model. The contrastive learning mechanism adopted by the CLIP model effectively aligns image and text features. The contrastive loss formula is:

[0045]

[0046] Among them, Loss is the value of the loss function, which is used to measure the performance of the model, and the goal is to minimize this value; N is the total number of sample pairs (that is, the number of pairs of images and texts); i is the index of the current sample pair, from 1 to N; j is used to traverse the index of all samples; u i Feature representation of the i-th image (image embedding vector); t i The feature representation of the i-th text (text embedding vector); sim(a,b) is a function that calculates the similarity between image a and text b (usually cosine similarity), which is used to adjust the distribution of similarity; For all texts j The sum of the exponential similarity values ​​is used as the normalized denominator to represent the similarity between the i-th image and all texts; For all images u jThe sum of the exponentially modified similarity values ​​is used as the normalized denominator to represent the similarity between the i-th text and all images; the temperature parameter τ is a hyperparameter used to adjust the smoothness of the similarity distribution; specifically, τ is introduced in the cross entropy calculation of the contrast loss function, and affects the training effect of the model by scaling the similarity of image-text pairs. When τ is small, the similarity distribution becomes sharper, and the model pays more attention to very similar sample pairs, making it easier to distinguish similar and dissimilar samples; when τ is large, the similarity distribution becomes smoother, and the model pays more attention to all sample pairs, but it may make it more difficult for the model to distinguish similar and dissimilar samples;

[0047] S42: Training the improved Resnet50 model, namely the IResNet model, includes three components: information flow of the network layer, residual block ResBlock and projection shortcut; the residual block is the core structure in ResNet, which solves the training difficulty problem in deep networks by introducing skip connections to add the input directly to the output of a certain network layer, allowing the network to learn the residual mapping between input and expected output; the projection shortcut is a special skip connection used to deal with the situation where the input and output dimensions are inconsistent. Generally, the dimensions are adjusted through 1x1 convolution to match the input and output. These two together promote the effective training of deep networks; specifically:

[0048] S421: The IResNet model optimizes the information transmission between network layers; the IResNet model optimizes the structure of the residual block ResBlock, that is, introduces more convolutional layers or different convolution operations; and introduces multi-scale feature fusion in the projection shortcut;

[0049] S422: Train the IResNet model, determine the hyperparameters of model training, set the number of training rounds, use Adam as the optimizer to update the model parameters, use the activation function as ReLU, and the loss function as the cross entropy loss function to obtain the IResNet model; and evaluate the accuracy and average precision of the IResNet model.

[0050] By adopting the above technical solution, the improved Resnet50 model is introduced, and its prediction results are integrated with the zero-shot prediction results of CLIP to build a classification model to achieve the purpose of fine-tuning CLIP. At the same time, the improved Resnet50 is trained using the pedestrian fall posture dataset to obtain the weight file person_best.pt of the fine-tuned CLIP model.

[0051] In addition, in order to reduce costs, no parameters of the CLIP model are modified, and only the IResNet model is trained to enable it to have the ability to recognize different pedestrian posture images. Unlike previous training, in the process of training IResNet, the loss function of its training must be adjusted according to the prediction results of the CLIP model. That is, the image encoder and text encoder of CLIP are used to extract the training images and the provided text to obtain the corresponding prediction scores, and the prediction scores of the training images are added in combination with the IResNet model, the loss function value is calculated, and the model is trained and optimized by continuously adjusting the loss function value.

[0052] The advantages of this method are, first, integrating the zero-shot capability of CLIP with the improved ResNet50 model can significantly improve the accuracy of pedestrian fall detection, because the wide scene adaptability of CLIP combined with the detailed feature extraction capability of the improved ResNet50 provides a more comprehensive basis for judgment; second, by utilizing the zero-shot performance of CLIP, there is no need for large-scale annotation for each scene, which greatly reduces the cost of data annotation and training, and only the improved ResNet50 needs to be trained to quickly adapt to specific task requirements; in addition, the use of residual connections enhances the model's anti-interference ability in complex environments or noisy data, enabling it to accurately capture the characteristics of pedestrian falls; finally, the multimodal understanding capability of CLIP combined with the visual feature extraction capability of the improved ResNet50 achieves comprehensive feature fusion, further improving the classification accuracy of the model in complex scenarios.

[0053] Preferably, the specific steps of step S5 are:

[0054] S51: First, load the pre-trained CLIP model weight file CLIP_best.pt. The CLIP model performs image classification through natural language supervision. After loading the CLIP model, switch the CLIP model to evaluation mode for subsequent classification prediction tasks. At the same time, perform pedestrian detection on the input image using the NanoDet-Plus model in step S3, and pass the pedestrian area extracted from the image to the fine-tuned CLIP model for further posture classification.

[0055] S52: Perform zero-shot classification prediction through the CLIP model; given the prompt text "a pedestrian with a falling posture", the CLIP model performs preliminary classification prediction on the extracted pedestrian image according to the natural language description, and obtains the classification prediction result A;

[0056] S53: The IResNet model is used to further classify and predict the pedestrian image to obtain a classification prediction result B;

[0057] S54: Fusing the classification prediction result A and the classification prediction result B to obtain a fused classification prediction result, that is, a pedestrian posture prediction result;

[0058] S55: Perform similarity evaluation on the comprehensive prediction results; if the similarity score exceeds the preset threshold value of 0.75, the current pedestrian is determined to be in a falling posture, and corresponding feedback measures are initiated, such as alarming or notifying relevant personnel; otherwise, the detection is stopped.

[0059] Preferably, in step S52, the CLIP model is used to encode text prompts and image features, and a preliminary classification prediction result A is obtained by calculating the similarity between the two; in step S54, the residual connection method is used to combine the classification prediction result A of the CLIP model and the prediction result B of the IResNet model to obtain a fused classification prediction result, that is, the pedestrian posture prediction result.

[0060] This paper adopts the "additional model training" strategy, combining the zero-shot reasoning of CLIP with the classification prediction results of the improved ResNet50, and synthesizing the prediction results of the two models on the image through residual connections, thereby improving the accuracy of image classification; that is, the residual connection can effectively fuse the features of two different models to obtain a more comprehensive prediction result. In order to further improve the accuracy of classification, we introduce the IResNet classification model, which is an improved deep residual network that performs well in feature extraction and classification tasks. Through the IResNet model, we further classify and predict pedestrian images and obtain another set of prediction results B.

[0061] Preferably, the formula of the activation function ReLu in the training IResNet model in step S422 is:

[0062]

[0063] Among them, x represents the input value, and max represents the maximum value function;

[0064] The formula for using cross entropy as the loss function is:

[0065]

[0066] Among them, Loss represents the loss function value, N is the number of images, and y n is the label of image n, p n is one of the element values ​​of the network output, and y n is corresponding;

[0067] The derived formula for updating model parameters using Adam as the optimizer is shown below.

[0068]

[0069] m t =β 1 m t-1 +(1-β 1 ) t (9);

[0070]

[0071]

[0072] Among them, the first gradient g t It is the loss function For θ t Find the partial derivative, θ t is the optimized parameter, the second term m t is the first-order moment estimate of the gradient in momentum form at time t, and the third term u t is the second-order moment estimate of the gradient in momentum form, and the fourth term is the first-order moment estimate after bias correction, where β 1 t is beta 1 raised to the power of t, β 1 and β 2 They are the hyperparameters for attenuating the first-order and second-order matrices of the gradient, and the fifth item is the second-order moment estimate after deviation correction. The last term η and ε represent the learning rate and the constant added to prevent the denominator from being zero. Substituting into the formula to update the parameters, we get θ t+1 ;

[0073] The training loss function is adjusted according to the prediction results of the CLIP model. Specifically, the image encoder and text encoder of CLIP are used to extract the image in training and the provided text to obtain the corresponding prediction scores. The prediction scores of the images in training are added in combination with the IResNet model, and the loss function value is calculated. The model is trained and optimized by continuously adjusting the loss function value. The specific model training formula is:

[0074] F train =VisualEncoder(I M ) (14);

[0075] L train =OneHot(L N ) (15);

[0076]

[0077] Among them, VisualEnconder is the image encoder of the CLIP model, where M is the number of images in a batch, I represents the images in the batch; OneHot is the encoder function, which is used to encode N categories; L is the category text; F train To extract the features of the image; L train is the text encoding vector processed by the encoder; β and α are hyperparameters; f test is the image feature of the image to be classified; A is the cosine similarity between the image to be classified and the image in the IResNet model, T represents the transpose operation of the matrix; W represents the text vector obtained by the CLIP encoder for C detection categories; Logits is the loss function of the model during training.

[0078] By adopting the above technical solution, by combining the lightweight real-time detection model NanoDet-Plus and the powerful image understanding model CLIP, and introducing the improved image classification model ResNet50 to fine-tune CLIP, the deep fusion of multimodal information is achieved. This method can not only improve the accuracy of pedestrian fall detection in different scenes, but also enhance the semantic understanding ability of the model for specific tasks. Compared with the traditional closed set target detection method, the model of the present invention can effectively improve the detection accuracy. Since there are multiple pedestrians in the prediction process, the fine-tuned CLIP can only classify the image, and it is impossible to accurately classify each pedestrian in the image. Therefore, the present invention extracts the information of each pedestrian in the image by combining the lightweight real-time detection model NanoDet-Plus, and then uses the fine-tuned CLIP model for classification prediction, so as to achieve the prediction of multiple pedestrian postures. Compared with the YOLO series algorithms used in the existing pedestrian fall detection method, NanoDet-Plus has better real-time performance and recognition accuracy, and is easier to deploy on monitoring equipment. By combining CLIP's zero-shot component for pedestrian fall recognition, it has significant innovative advantages over traditional object detection and classification models. First, CLIP's multimodal training architecture gives it excellent flexibility. Users can customize text descriptions (such as "a pedestrian in yellow clothes fell"), and the model can directly understand and recognize them accordingly without additional training, but traditional object detection models and classification models cannot do this. This flexibility allows the model to quickly adapt to different scenarios and needs, improving the adaptability of applications. Secondly, traditional models usually require a large amount of labeled data for specific tasks. The zero-shot feature of CLIP means that effective recognition can be performed without having seen samples of related categories, which greatly saves training time and computing resources. In addition, the rich image and text features learned by CLIP in the pre-training stage enhance its recognition ability for complex scenes, making the recognition of subtle posture changes such as pedestrian falls more accurate. Finally, since it does not rely on specific training data, CLIP's zero-shot method also simplifies the maintenance and expansion of the model, making it easier to support more categories and scenarios. These advantages have jointly promoted the advancement of pedestrian fall recognition technology and achieved more efficient and flexible application scenarios.

[0079] Compared with the prior art, the present invention has the following beneficial effects: by combining the lightweight real-time detection model NanoDet-Plus and the powerful image understanding model CLIP, and introducing the improved image classification model ResNet50 to fine-tune CLIP, a deep fusion of multimodal information is achieved. This method can not only improve the accuracy of pedestrian fall detection in different scenarios, but also enhance the model's semantic understanding ability for specific tasks. Compared with the traditional closed-set target detection method, the pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP can effectively improve the detection accuracy, jointly promote the advancement of pedestrian fall recognition technology, and realize more efficient and flexible application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the overall process of the pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP of the present invention;

[0081] Figure 2 This is a flow chart of the detection in the pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP of the present invention. DETAILED DESCRIPTION

[0082] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0083] Embodiment: The known technical contents underlying the present invention include NanoDet-Plus target detection algorithm and CLIP multimodal model.

[0084] (1) NanoDet-Plus object detection algorithm

[0085] NanoDet-Plus is a lightweight object detection algorithm specially designed for high-speed and high-precision detection on devices with limited computing resources. Its main advantages include high efficiency, low model complexity, high accuracy, anchor-free design, flexibility and easy deployment. Compared with other algorithms such as YOLOv5, Faster R-CNN, EfficientDet and SSD, NanoDet-Plus significantly reduces the computing resource requirements and has stronger adaptability to diverse targets while maintaining high detection accuracy, which is particularly suitable for real-time and resource-constrained application scenarios.

[0086] (2) CLIP multimodal model

[0087] The CLIP (Contrastive Language-Image Pretraining) model is a multimodal learning model developed by OpenAI. It is pre-trained on a large-scale image-text pair dataset to enable it to understand and associate the relationship between images and text. Its main methods include contrastive learning, which aligns images and text by maximizing the similarity between correct image-text pairs and minimizing the similarity of errors; dual encoder structure, which uses convolutional neural networks or visual transformers to extract image features, and transformers to extract text features; joint embedding space, which generates a shared embedding space through training so that images and text descriptions have similar embedding vectors, thereby supporting multiple tasks such as image classification, retrieval, and generation. Pre-training is performed on a large-scale dataset containing about 400 million image-text pairs, which enhances the generalization ability of the model. The main applications include zero-shot learning, which can be applied to new tasks without task-specific annotated data; multimodal search, which is used for image, text, and cross-modal search; image generation, which combines generative models to generate images that match text descriptions.

[0088] like Figure 1 As shown, the pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP specifically includes the following steps:

[0089] S1: collect data and create datasets;

[0090] In step S1, Pycharm is used to crawl pedestrian posture image data of different categories, and the collected pedestrian video is frame sampled by the ffmpeg tool to obtain pedestrian posture image data; the specific steps are:

[0091] S11: Analyze the webpage structure of Baidu Image Library and the webpage loading method of 360 Image Library, and check the robots.txt protocol of both; write a script to collect pedestrian posture image data based on the analysis results;

[0092] S12: Use the ffmpeg tool to perform frame sampling on the collected pedestrian video, and select image data containing the pedestrian posture;

[0093] S13: Manually screen the collected pedestrian posture data to remove poor quality image data, such as image data that does not meet the definition of the detection task; obtain data to make a data set;

[0094] In this embodiment, the data set is prepared as follows:

[0095] First, use the "Octopus Collector" tool to analyze the Baidu Gallery and 360 Gallery in order to collect the download addresses of the image data of pedestrian postures in different scenarios; in the Octopus tool, create a new crawling task, enter the URL of the Baidu and 360 Gallery, and set the crawling rules to extract the download addresses of the images. In order to ensure that all images can be crawled, configure the paging rules, and check the HTML structure of the web page and the JavaScript dynamic loading method. After the crawling is completed, the Octopus tool aggregates all the image download addresses into a local file in CSV format for subsequent processing; then write a script to download these images in batches; in this process, read the previously generated local file, parse the download address of each image, and then use Python and the requests library to send HTTP requests to download these images one by one and save them to the specified local directory;

[0096] Secondly, in terms of video frame sampling and processing, the ffmpeg tool is used to extract key frames from the collected pedestrian videos. These frames usually contain higher image quality. The extracted frames will be saved as independent image files for subsequent processing and analysis;

[0097] Finally, in the data screening stage, the collected images of pedestrian postures in different scenes need to be manually screened to ensure the quality and relevance of the acquired images. The screening process includes removing blurry or improperly exposed images and ensuring that the images clearly show the pedestrian posture information. Through these steps, it is ensured that the final dataset meets both the quality standards and the predetermined task requirements;

[0098] This dataset covers various categories in different scenarios, including pedestrian posture images in different scenarios, to ensure that the model can perform effective detection in a variety of environments;

[0099] S2: Train the NanoDet-Plus target detection model and obtain the trained algorithm model weight file;

[0100] The specific steps of step S2 are:

[0101] S21: Segment the collected pedestrian posture image data, use the X-AnyLabeling tool to intelligently label the collected images, and conduct manual review to ensure that the target objects in each image are accurately labeled;

[0102] S22: After the labeling is completed, the data set is divided into a training set and a validation set to facilitate subsequent model training and evaluation; the data set includes different categories of pedestrian posture images in different scenes that need to be detected; the labeled data set is divided into two types, namely, a pedestrian data set and a pedestrian posture data set including pedestrian falls;

[0103] S23: Use the training set to train the NanoDet-Plus model, set the number of training rounds, and obtain the algorithm model weight file best.pt; in this embodiment, the number of training rounds is set to 200; through this process, the trained algorithm model weight file best.pt is obtained; this weight file contains the model parameters that have been trained and optimized, which can be used for subsequent target detection tasks to achieve accurate recognition of pedestrians;

[0104] S24: Use the validation set to evaluate the trained NanoDet-Plus model;

[0105] The evaluation indicators in step S24 include accuracy, average precision (AP), recall, and F1 score;

[0106] The accuracy calculation formula is:

[0107]

[0108] Among them, Accuracy is expressed as accuracy, TP (True Positive) means true positive, which means the number of samples that are actually positive and correctly predicted as positive by the model; TN (True Negative): true negative, which means the number of samples that are actually negative and correctly predicted as negative by the model; FP (False Positive): false positive, which means the number of samples that are actually negative but are incorrectly predicted as positive by the model (false positive); FN (False Negative): false negative, which means the number of samples that are actually positive but are incorrectly predicted as negative by the model (false negative);

[0109] The formula for calculating the average precision is:

[0110]

[0111] Among them, Precision is expressed as the average precision, TP (True Positive): True positive, which means the number of samples that are actually positive and correctly predicted as positive by the model; FP (False Positive): False positive, which means the number of samples that are actually negative but are incorrectly predicted as positive by the model (false positive);

[0112] The recall calculation formula is:

[0113]

[0114] Among them, Recall represents the recall rate, TP (True Positive): True positive, which means the number of samples that are actually positive and correctly predicted as positive by the model; FN (False Negative): False negative, which means the number of samples that are actually positive but incorrectly predicted as negative by the model (missed reports);

[0115] The F1 calculation formula is:

[0116]

[0117] Among them, Fmeasure is expressed as F1, Precision is the average precision, and Recall is the recall rate.

[0118] S3: Collect the image to be predicted and input it into the target detection model; use the target detection model NanoDet-Plus to perform pedestrian recognition on the image P, and save the recognized pedestrian image separately as an image file;

[0119] like Figure 2 As shown, the specific steps of step S3 are:

[0120] S31: Based on the deployed cameras, the area to be detected is monitored in real time, and continuous image acquisition is performed to obtain the image to be predicted; the algorithm can be deployed on fixed cameras, which can monitor the designated area in real time and perform continuous image acquisition; during this process, the algorithm will automatically identify pedestrians in the image and determine in real time whether a pedestrian has fallen. Once a pedestrian falls, the system will quickly record the relevant information and issue an alarm to facilitate timely intervention and rescue by staff;

[0121] S32: Load the model weight file best.pt and the model configuration file default.yaml, use the NanoDet-Plus algorithm to analyze the image to be detected, and obtain the rectangular frame position area of ​​the pedestrian;

[0122] The specific steps of analyzing the image P to be detected of 1000×750 pixels by using the NanoDet-Plus algorithm in step S32 are as follows:

[0123] S321: First, the size of the image P to be detected is adjusted to the input size required by the NanoDet-Plus model, which is 640×640 pixels, so that the image P to be detected meets the input requirements of the model;

[0124] S322: normalize the resized image and scale the pixel values ​​to the range of [0, 1] so that the model can process it effectively;

[0125] S323: The NanoDet-Plus model is used to recognize the image. During the detection process, for each pedestrian recognized, the predicted frame information of the pedestrian is extracted, and the predicted frame information of each pedestrian is saved as an independent picture for backup and subsequent processing; if no pedestrian is recognized, the detection is stopped;

[0126] Load the model weight file best.pt and the model configuration file default.yaml. When using the NanoDet-Plus algorithm to analyze the 1000×750 pixel image P to be detected, the main image processing method includes the following steps: First, the image P to be detected will be adjusted to the input size of 640×640 pixels required by the NanoDet-Plus model, so that the image P to be detected meets the input requirements of the model; then the adjusted image will be normalized to scale the pixel values ​​to the range of [0,1] so that the model can process it effectively. During the detection process, for each pedestrian identified, the predicted box information of each pedestrian will be extracted, and each pedestrian's predicted box information will be saved as a separate image for backup and subsequent processing;

[0127] NanoDet-Plus uses a lightweight backbone network (MobileNetV3) and a lightweight feature pyramid network (FPN), which enables it to effectively capture pedestrian features of different scales while reducing the number of parameters and the amount of computation. In contrast, YOLOv5 uses complex CSPNet and SPP components to improve feature extraction capabilities and context information integration through multi-layer feature fusion, but it is more complex. NanoDet-Plus can achieve lower latency and higher frame rates on edge devices due to its simplified structure, making it suitable for real-time detection tasks. Although YOLOv5 also has a speed-optimized version, it usually performs worse than NanoDet-Plus on resource-constrained devices. NanoDet-Plus focuses on computational efficiency, reduces complexity through a priori boxes and efficient anchor box mechanisms, and ensures that it can run effectively on low-computing devices. Due to its model complexity, YOLOv5's performance on low-configuration devices is relatively reduced. In pedestrian detection tasks, NanoDet-Plus's lightweight and optimized architecture can improve recall while ensuring reasonable accuracy, which is particularly suitable for real-time scenarios such as surveillance video analysis and intelligent transportation. NanoDet-Plus optimizes the loss function and post-processing to achieve better performance on a lightweight model. YOLOv5 enhances the detection capability of small targets through anchor-based methods and multi-scale prediction, but the overall complexity is also increased accordingly. Therefore, NanoDet-Plus is particularly suitable for environments with limited resources and requiring fast real-time detection compared to YOLO-v5;

[0128] S4: training the CLIP model and the improved Resnet50 model; that is, fine-tuning the CLIP model; the information of each pedestrian (i.e., the independent picture saved for each pedestrian in the image) detected and obtained through step S3 needs to be further classified and predicted using the fine-tuned CLIP model. In this embodiment, an improved Resnet50 and the zero-shot component of CLIP are introduced for fusion, so as to achieve the purpose of fine-tuning CLIP;

[0129] For CLIP's zero-shot capability:

[0130] CLIP's advantages in pedestrian fall detection are not only reflected in the design and functions of its specific components, but also closely related to the significant differences between traditional image classification models. First, CLIP's image encoder is based on a deep convolutional neural network (Vision Transformer), which can convert the input image into a high-dimensional feature vector and effectively extract key features in dynamic and changing scenes, such as pedestrian falls. In contrast, traditional image classification models (ResNet or VGG) usually only focus on single feature extraction when processing images and lack the ability to combine multimodal information. In addition, CLIP's text encoder uses a Transformer model to process natural language input, converting text descriptions into feature vectors, enabling it to classify fall behaviors through natural language, such as "a person in yellow clothes fell" or "an old man fell". This capability is not possible with traditional image classification models; on the other hand, the contrastive learning mechanism adopted by CLIP effectively aligns image and text features;

[0131] The specific steps of step S4 are:

[0132] S41: Training the CLIP model. The contrastive learning mechanism adopted by the CLIP model effectively aligns image and text features. The contrastive loss formula is:

[0133]

[0134] Among them, Loss is the value of the loss function, which is used to measure the performance of the model, and the goal is to minimize this value; N is the total number of sample pairs (that is, the number of pairs of images and texts); i is the index of the current sample pair, from 1 to N; j is used to traverse the index of all samples; u i Feature representation of the i-th image (image embedding vector); t i The feature representation of the i-th text (text embedding vector); sim(a,b) is a function that calculates the similarity between image a and text b (usually cosine similarity), which is used to adjust the distribution of similarity; For all texts jThe sum of the exponential similarity values ​​is used as the normalized denominator to represent the similarity between the i-th image and all texts; For all images u j The sum of the exponentially modified similarity values ​​is used as the normalized denominator to represent the similarity between the i-th text and all images; the temperature parameter τ is a hyperparameter used to adjust the smoothness of the similarity distribution; specifically, τ is introduced in the cross entropy calculation of the contrast loss function, and affects the training effect of the model by scaling the similarity of image-text pairs. When τ is small, the similarity distribution becomes sharper, and the model pays more attention to very similar sample pairs, making it easier to distinguish similar and dissimilar samples; when τ is large, the similarity distribution becomes smoother, and the model pays more attention to all sample pairs, but it may make it more difficult for the model to distinguish similar and dissimilar samples;

[0135] During training, the distance between the features of positive samples (images and their corresponding texts) is close, while the distance between negative samples (irrelevant images and texts) is far, which enables the model to accurately judge the relevance of pedestrian falls during inference. In contrast, traditional classification models rely on manually annotated category labels for training and cannot directly utilize multimodal information, resulting in limited performance when dealing with complex situations;

[0136] Secondly, CLIP's Zero-Shot reasoning capability allows it to directly perform reasoning without fine-tuning for specific tasks. Users only need to provide a natural language description, and the model can use the encoder to calculate the similarity between the image and the text to identify unseen fall situations. This feature is significantly better than the large amount of labeled data required by traditional models;

[0137] Finally, in contrast, although traditional target detection algorithms (YOLO series) have good real-time performance when performing target detection, their feature selection and classification capabilities are often based on a single, predefined category label. As the complexity of the scene increases, especially in dynamic environments, algorithms such as YOLO may encounter difficulties because they are not flexible enough to respond to unseen situations and have poor ability to capture subtle changes. This is because YOLO relies on a classifier trained on a specific dataset and has poor adaptability to new scenes or unknown fall situations;

[0138] S42: In order to further improve the ability of the additional model Resnet50 to identify pedestrian falls, an improved version of the IResNet model improved by Resnet50 is used; the training of the improved Resnet50 model, namely the IResNet model, includes three components: the information flow of the network layer, the residual block ResBlock and the projection shortcut; the residual block is the core structure in ResNet, which directly adds the input to the output of a certain network layer by introducing the skip connection, thereby solving the training difficulty problem in the deep network and allowing the network to learn the residual mapping between the input and the expected output; the projection shortcut is a special skip connection used to deal with the situation where the input and output dimensions are inconsistent. Generally, the dimension is adjusted through 1x1 convolution so that the input and output match. These two together promote the effective training of deep networks; specifically:

[0139] S421: The IResNet model optimizes the information transmission between network layers; the IResNet model optimizes the structure of ResBlock, that is, introduces more convolutional layers or different convolution operations; and introduces multi-scale feature fusion in ProjectionShortcut;

[0140] Specifically: First, IResNet significantly improves the quality of information flow by optimizing the way information is transmitted between network layers. This improvement not only reduces information loss and the introduction of noise, but also enhances the expressiveness of the information transmission process by introducing more nonlinear transformations in the jump connection. This optimization enables the information of the previous layer to be more accurately transmitted to deeper network layers, thereby improving the feature extraction ability of the model. In addition, the improved jump connection method effectively reduces the attenuation of the gradient during the back propagation process, making the model easier to train and converge;

[0141] Secondly, IResNet optimizes the structure of ResBlock more efficiently. By introducing more convolutional layers or different convolution operations (such as depthwise separable convolution), IResNet significantly improves the expressiveness of the model while reducing computational complexity. In addition, the introduction of the attention mechanism enables ResBlock to adaptively adjust the importance of different channels and improve the discriminative ability of features. These improvements not only improve the computational efficiency of the model while maintaining high performance, but also enhance the accuracy of feature extraction, thereby further improving the overall performance of the model;

[0142] Finally, IResNet makes flexible improvements to Projection Shortcut. The improved Projection Shortcut can more flexibly match the input and output dimensions between different layers, reducing the information loss caused by dimension mismatch. At the same time, by introducing multi-scale feature fusion in Projection Shortcut, IResNet can better utilize features at different levels and enhance the global feature expression ability of the model. This improvement not only solves the common dimension mismatch problem in traditional ResNet, but also significantly improves the overall expression and generalization ability of the model;

[0143] S422: Train the IResNet model, determine the hyperparameters of model training, set the number of training rounds, use Adam as the optimizer to update the model parameters, use ReLU as the activation function, and use the cross entropy loss function as the loss function, thereby obtaining the IResNet model; and evaluate the accuracy and average precision of the IResNet model; in this embodiment, in order to further improve the accuracy of pedestrian fall detection and reduce the training cost, this paper uses the zero-shot capability of the CLIP model for preliminary classification. Although this method has achieved certain results, in order to enhance the classification effect, this paper adopts the "additional model training" strategy, combining the zero-shot reasoning of CLIP with the classification prediction results of the improved ResNet50, and synthesizing the prediction results of the two models on the image through residual connection, thereby improving the accuracy of image classification. At the beginning of model training, the hyperparameters of the model are determined, where the number of data enhancement rounds is 20, the number of training rounds is 300, the number of sample categories in training is 16, the learning rate is 0.001, the initial value of the hyperparameter beta is 1, the initial value of the hyperparameter alpha is 1.17, the hyperparameter search range is [10, 10], and the hyperparameter search step is [200, 20]; the formula for the activation function ReLu in the IResNet model trained in step S422 is:

[0144]

[0145] Among them, x represents the input value, and max represents the maximum value function;

[0146] The formula for using cross entropy as the loss function is:

[0147]

[0148] Among them, Loss represents the loss function value, N is the number of images, and y n is the label of image n, p n is one of the element values ​​of the network output, and y nis corresponding;

[0149] The derived formula for updating model parameters using Adam as the optimizer is shown below.

[0150]

[0151] m t =β 1 m t-1 +(1-β 1 ) t (9);

[0152]

[0153] Among them, the first gradient g t It is the loss function For θ t Find the partial derivative, θ t is the optimized parameter, the second term m t is the first-order moment estimate of the gradient in momentum form at time t, and the third term u t is the second-order moment estimate of the gradient in momentum form, and the fourth term is the first-order moment estimate after bias correction, where β 1 t is beta 1 raised to the power of t, β 1 and β 2 They are the hyperparameters for attenuating the first-order and second-order matrices of the gradient, and the fifth item is the second-order moment estimate after deviation correction. The last term η and ε represent the learning rate and the constant added to prevent the denominator from being zero. Substituting into the formula to update the parameters, we get θ t+1 ;

[0154] In order to reduce costs, no parameters of the CLIP model are modified, and only the IResNet model is trained to enable it to recognize different pedestrian posture images. Unlike previous training, the loss function of the training is adjusted according to the prediction results of the CLIP model during the IResNet training. Specifically, the image encoder and text encoder of CLIP are used to extract the training images and the provided text to obtain the corresponding prediction scores, and the prediction scores of the training images are added in combination with the IResNet model, and the loss function value is calculated and the model is trained and optimized by continuously adjusting the loss function value. The specific model training formula is:

[0155] F train =VisualEncoder(I M ) (14);

[0156] L train=OneHot(L N ) (15);

[0157]

[0158] Among them, VisualEnconder is the image encoder of the CLIP model, where M is the number of images in a batch, I represents the images in the batch; OneHot is the encoder function, which is used to encode N categories; L is the category text; F train To extract the features of the image; L train is the text encoding vector processed by the encoder; β and α are hyperparameters; f test is the image feature of the image to be classified; A is the cosine similarity between the image to be classified and the image in the IResNet model, T represents the transpose operation of the matrix; W represents the text vector obtained by the CLIP encoder for C detection categories; Logits is the loss function of the model during training. For IResNet model training, the evaluation indicators are accuracy and average precision, as shown in formulas (1) and (2);

[0159] S5: using the CLIP model to analyze the image files one by one, that is, performing preliminary classification prediction on the pedestrian posture image, and then using the improved Resnet50 model to perform further classification prediction, connecting the classification prediction results obtained by the two models, so as to obtain the pedestrian posture prediction result; determining the current state of the pedestrian according to the posture prediction result, and taking corresponding feedback measures;

[0160] like Figure 2 As shown, the specific steps of step S5 are:

[0161] S51: First, load the pre-trained CLIP model weight file CLIP_best.pt. The CLIP model performs image classification through natural language supervision. After loading the CLIP model, switch the CLIP model to evaluation mode for subsequent classification prediction tasks. At the same time, perform pedestrian detection on the input image using the NanoDet-Plus model in step S3, and pass the pedestrian area extracted from the image to the fine-tuned CLIP model for further posture classification.

[0162] S52: Perform zero-sample classification prediction through the CLIP model; given the prompt text "a pedestrian with a falling posture", the CLIP model performs preliminary classification prediction on the extracted pedestrian image according to the natural language description to obtain a classification prediction result A; in the step S52, the CLIP model is used to encode the text prompt and image features, and the preliminary classification prediction result A is obtained by calculating the similarity between the two;

[0163] S53: In order to further improve the accuracy of classification, the IResNet classification model is introduced. This model is an improved deep residual network with excellent performance in feature extraction and classification tasks. The IResNet model is then used to further classify and predict pedestrian images, and the classification prediction result B is obtained;

[0164] S54: In order to comprehensively utilize the prediction information of different models, we use the residual connection method to fuse the classification prediction result A and the classification prediction result B to obtain a fused classification prediction result, that is, the pedestrian's posture prediction result; the residual connection method is used in step S54 to combine the classification prediction result A of the CLIP model and the prediction result B of the IResNet model to obtain a fused classification prediction result, that is, the pedestrian's posture prediction result; the residual connection can effectively fuse the features of two different models to obtain a more comprehensive prediction result. In order to further improve the accuracy of classification, we introduce the IResNet classification model, which is an improved deep residual network that performs well in feature extraction and classification tasks. Through the IResNet model, we perform further classification predictions on pedestrian images and obtain another set of prediction results B;

[0165] S55: Perform similarity evaluation on the comprehensive prediction results; if the similarity score exceeds a preset threshold of 0.75, the current pedestrian is determined to be in a falling posture, and corresponding feedback measures are initiated, such as alarming or notifying relevant personnel; otherwise, the detection is stopped;

[0166] In this embodiment, in the process of using the fine-tuned CLIP model to predict the exported image P, the classification prediction includes two parts, one part is predicted based on the value retrieved from the "cached model", and the other part is predicted based on the pre-trained CLIP, and the two are balanced by the weight parameter α; specifically:

[0167] First, use VisualEnconder to extract the image feature Ftest of image P. In this invention, we provide four types of text prompts, namely "a fallen pedestrian", "a sitting pedestrian", "a standing pedestrian", and "a bent pedestrian". By inputting these four types of text into the text encoder of the CLIP model, the feature set W of these four texts is obtained. The inner product of Ftest and W is used to obtain the zero-shot prediction result Y_A for image P. The text provided here can be flexibly adjusted according to specific scenarios;

[0168] Then, the IResNet model trained with the pedestrian posture dataset is used to extract the image features in the "cached model". The "cached model" here refers to the image data of different new people's postures, which are saved in separate variables for subsequent model processing. Before this, the IResNet model has been trained and has the ability to extract features from images of different pedestrian postures. Here, it is used to extract the image features F in the "cached model". train , and combined with L train 、F test Retrieve pedestrian posture knowledge from the "cached model", that is, the prediction result Y_B of pedestrian posture;

[0169] Finally, two prediction results Y_A and Y_B are obtained, and residual connection is performed to obtain the final prediction result Y_C, which is the final classification result of image P;

[0170] The advantages of this method are: first, integrating CLIP's zero-shot capability with the improved ResNet50 model can significantly improve the accuracy of pedestrian fall detection, because CLIP's wide scene adaptability combined with the improved ResNet50's detailed feature extraction capabilities provide a more comprehensive basis for judgment; second, by utilizing CLIP's zero-shot performance, there is no need for large-scale annotation for each scene, which greatly reduces the cost of data annotation and training, and only the improved ResNet50 needs to be trained to quickly adapt to specific task requirements; in addition, the use of residual connections enhances the model's anti-interference ability in complex environments or noisy data, enabling it to accurately capture the characteristics of pedestrian falls; finally, CLIP's multimodal understanding capabilities combined with the improved ResNet50's visual feature extraction capabilities achieve comprehensive feature fusion, further improving the model's classification accuracy in complex scenes. The specific advantages are:

[0171] First, the present invention combines the zero-shot capability of the CLIP model and IResNet (improved ResNet50) for pedestrian posture classification prediction. This strategy not only makes full use of the CLIP model's ability to perform fast and accurate category recognition without the need for a large amount of labeled data, but also further improves the capture of pedestrian posture details through the deep feature extraction capability of IResNet, thereby more accurately judging whether the pedestrian is in a falling state.

[0172] At the same time, the present invention introduces NanoDet-Plus for pedestrian information extraction. This lightweight target detection model can accurately locate and extract pedestrian information from images while maintaining high efficiency. Compared with traditional target detection algorithms such as YOLO, NanoDet-Plus is not only smaller in size and more computationally efficient, but also has stronger adaptability to multi-target detection and recognition tasks in complex traffic scenes, and can effectively reduce false detections and missed detections. After the pedestrian information is extracted, the present invention uses the fine-tuned CLIP model to perform one-to-one posture classification for each pedestrian. This approach avoids the problems of information transmission interruption and classification accuracy loss that may occur when traditional target detection is combined with image classification algorithms. By directly utilizing the rich pre-training knowledge and fine-tuned accuracy of the CLIP model, the present invention can efficiently and accurately classify the posture of each extracted pedestrian, thereby achieving accurate analysis of the postures of multiple pedestrians.

[0173] In addition, the CLIP model is a multimodal deep learning model that combines language and image information. It pre-trains data through large-scale image-text pairs, allowing the model to directly perform multiple visual tasks such as image classification and object detection without the need for task-specific labeled data. Traditional classification models usually rely on task-specific labeled data, which often requires a lot of time and labor costs to obtain. In contrast, the zero-shot learning ability of the CLIP model means that it can be directly applied to classification tasks of new categories without training data for specific tasks. This feature gives the CLIP model a significant advantage in the face of new or sparsely classified scenes, and it can quickly adapt and recognize. In terms of object detection, traditional object detection models such as YOLO and Faster R-CNN usually require specific bounding box labeled data for training, while the CLIP model can use its multimodal characteristics to directly use pre-trained image and text information for object detection and classification without these annotations. This not only reduces the workload of data annotation, but also improves the adaptability of the model in new scenes.

[0174] In summary, the present invention combines the zero-shot learning capability of the advanced CLIP model with the IResNet model, and realizes efficient and high-precision analysis of pedestrian posture in complex traffic scenes by optimizing the continuous process of target detection and image classification. Compared with traditional target detection and image classification algorithms, the present invention not only improves the processing speed and accuracy, but also reduces the model complexity and the consumption of computing resources, showing great potential in practical applications. The multimodal learning capability of the CLIP model enables it to better understand the contextual information of the image, which is particularly important in tasks combined with language descriptions. For example, when judging the posture of a pedestrian falling, CLIP can combine the language description in the context (such as "a pedestrian in red clothes fell") to more accurately classify the posture, which is difficult to achieve with traditional single-modal models. In general, the present invention not only improves the accuracy of pedestrian posture classification in complex traffic scenes by integrating the powerful functions of the CLIP model and the efficient detection capabilities of NanoDet-Plus, but also demonstrates its wide potential in real-world applications.

[0175] For ordinary technicians in this field, the specific embodiments are only illustrative descriptions of the present invention. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concepts and technical solutions of the present invention, or the concepts and technical solutions of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A pedestrian fall detection method based on NanoDet-Plus and fine-tuned CLIP, characterized in that: The specific steps include: S1: collect data and create datasets; S2: Train the NanoDet-Plus target detection model and obtain the trained algorithm model weight file; S3: Collect the image to be predicted and input it into the target detection model; use the target detection model NanoDet-Plus to perform pedestrian recognition on the image, and save the recognized pedestrian image separately as an image file; S4: Training CLIP model and improved Resnet50 model; S5: The CLIP model is used to make preliminary classification predictions on the pedestrian posture images, and then the improved Resnet50 model is used for further classification predictions. The classification prediction results obtained by the two models are connected to obtain the pedestrian posture prediction results; the current state of the pedestrian is determined according to the posture prediction results, and feedback measures are taken.

2. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 1, characterized in that: In step S1, Pycharm is used to crawl pedestrian posture image data of different categories, and the collected pedestrian video is frame sampled by the ffmpeg tool to obtain pedestrian posture image data; the specific steps are: S11: Analyze the loading methods of web pages in various websites, check the robots.txt protocols of web pages in various websites; collect pedestrian posture image data based on the analysis results; S12: Use the ffmpeg tool to perform frame sampling on the collected pedestrian video, and select image data containing the pedestrian posture; S13: Screening the collected pedestrian posture data to obtain data and create a data set.

3. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 2, characterized in that: The specific steps of step S2 are: S21: Annotate the collected images and review them to ensure that the target objects in each image are accurately marked; S22: After the labeling is completed, the dataset is divided into a training set and a validation set to facilitate subsequent model training and evaluation; S23: Use the training set to train the NanoDet-Plus model, set the number of training rounds, and obtain the algorithm model weight file best.pt; S24: Use the validation set to evaluate the trained NanoDet-Plus model.

4. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 2, characterized in that: The evaluation indicators in step S24 include accuracy, average precision, recall rate, and F1 score; The accuracy calculation formula is: Among them, Accuracy is expressed as accuracy, TP: True Positive, which means the number of samples that are actually positive and correctly predicted as positive by the model; TN: True Negative, which means the number of samples that are actually negative and correctly predicted as negative by the model; FP: False Positive, which means the number of samples that are actually negative but are incorrectly predicted as positive by the model, i.e. false positive; FN: False Negative, which means the number of samples that are actually positive but are incorrectly predicted as negative by the model, i.e. false negative; The formula for calculating the average precision is: Among them, Precision is expressed as the average precision, TP: True Positive, which means the number of samples that are actually positive and correctly predicted as positive by the model; FP: False Positive, which means the number of samples that are actually negative but are incorrectly predicted as positive by the model, i.e., false positive; The recall calculation formula is: Among them, Recall represents the recall rate, TP: True Positive, which means the number of samples that are actually positive and correctly predicted as positive by the model; FN: False Negative, which means the number of samples that are actually positive but are incorrectly predicted as negative by the model, i.e., missed reports; The F1 calculation formula is: Among them, Fmeasure is expressed as F1, Precision is the average precision, and Recall is the recall rate.

5. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 4, characterized in that: The specific steps of step S3 are: S31: According to the deployed cameras, the area to be detected is monitored in real time, and continuous image acquisition is performed to obtain the image to be predicted; S32: Load the model weight file and the model configuration file, use the NanoDet-Plus algorithm to analyze the image to be detected, and obtain the rectangular frame position area of ​​the pedestrian.

6. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 5, characterized in that: The specific steps of using the NanoDet-Plus algorithm to analyze the image to be detected in step S32 are as follows: S321: first adjust the size of the image to be detected; S322: normalizing the resized image and scaling the pixel values ​​to the range of [0, 1]; S323: The NanoDet-Plus model is used to recognize the image. During the detection process, for each pedestrian recognized, the predicted frame information of the pedestrian is extracted, and the predicted frame information of each pedestrian is saved as an independent picture for backup and subsequent processing; if no pedestrian is recognized, the detection is stopped.

7. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 5, characterized in that: The specific steps of step S4 are: S41: Training the CLIP model. The contrastive learning mechanism adopted by the CLIP model effectively aligns image and text features. The contrastive loss formula is: Among them, Loss is the value of the loss function, which is used to measure the performance of the model, and the goal is to minimize this value; N is the total number of sample pairs, that is, the number of pairs of images and texts; i is the index of the current sample pair, from 1 to N; j is used to traverse the index of all samples; u i The feature representation of the i-th image is the image embedding vector; t i is the feature representation of the i-th text, i.e., the text embedding vector; sim(a,b) is a function for calculating the similarity between image a and text b, which is used to adjust the distribution of similarity; For all texts j The sum of the exponential similarity values ​​is used as the normalized denominator to represent the similarity between the i-th image and all texts; For all images u j The sum of the exponentially modified similarity values ​​is used as the normalized denominator to represent the similarity between the i-th text and all images; the temperature parameter τ is a hyperparameter used to adjust the smoothness of the similarity distribution; S42: Training the improved Resnet50 model, namely the IResNet model, includes three components: information flow of the network layer, residual block ResBlock and projection shortcut; specifically: S421: The IResNet model optimizes the information transmission between network layers; the IResNet model optimizes the structure of the residual block ResBlock, that is, introduces more convolutional layers or different convolution operations; and introduces multi-scale feature fusion in the projection shortcut; S422: Train the IResNet model, determine the hyperparameters of model training, set the number of training rounds, use Adam as the optimizer to update the model parameters, use the activation function as ReLU, and the loss function as the cross entropy loss function to obtain the IResNet model; and evaluate the accuracy and average precision of the IResNet model.

8. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 7, characterized in that: The specific steps of step S5 are: S51: First, load the pre-trained CLIP model weight file. The CLIP model performs image classification through natural language supervision. After loading the CLIP model, switch the CLIP model to evaluation mode. At the same time, perform pedestrian detection on the input image using the NanoDet-Plus model in step S3. The pedestrian area in the extracted image is passed to the fine-tuned CLIP model for further posture classification. S52: Perform zero-shot classification prediction through the CLIP model; given the prompt text "a pedestrian with a falling posture", the CLIP model performs preliminary classification prediction on the extracted pedestrian image according to the natural language description, and obtains the classification prediction result A; S53: The IResNet model is used to further classify and predict the pedestrian image to obtain a classification prediction result B; S54: Fusing the classification prediction result A and the classification prediction result B to obtain a fused classification prediction result, that is, a pedestrian posture prediction result; S55: Perform similarity evaluation on the comprehensive prediction results; if the similarity score exceeds a preset threshold, the current pedestrian is determined to be in a falling posture, and corresponding feedback measures are initiated; otherwise, the detection is stopped.

9. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 7, characterized in that: In step S52, the CLIP model is used to encode text prompts and image features, and a preliminary classification prediction result A is obtained by calculating the similarity between the two. In step S54, the residual connection method is used to combine the classification prediction result A of the CLIP model and the prediction result B of the IResNet model to obtain a fused classification prediction result, that is, the pedestrian posture prediction result.

10. The pedestrian fall detection method based on NanoDet-Plus and fine-tuning CLIP according to claim 9, characterized in that: The formula of the activation function ReLu in the training IResNet model in step S422 is: Among them, x represents the input value, and max represents the maximum value function; The formula for using cross entropy as the loss function is: Among them, Loss represents the loss function value, N is the number of images, and y n is the label of image n, p n is one of the element values ​​of the network output, and y n is corresponding; The derived formula for updating model parameters using Adam as the optimizer is shown below. m t =β1m t-1 +(1-β1)g t (9); Among them, the first gradient g t It is the loss function For θ t Find the partial derivative, θ t is the optimized parameter, the second term m t is the first-order moment estimate of the gradient in momentum form at time t, and the third term u t is the second-order moment estimate of the gradient in momentum form, and the fourth term is the bias-corrected first-order moment estimate, where is beta1 to the power of t, β1 and β2 are the hyperparameters for attenuating the first-order and second-order matrices of the gradient, respectively. The fifth term is the second-order moment estimate after deviation correction. The last term η and ε represent the learning rate and the constant added to prevent the denominator from being zero. Substituting into the formula to update the parameters, we get θ t+1 ; The training loss function is adjusted according to the prediction results of the CLIP model. Specifically, the image encoder and text encoder of CLIP are used to extract the image in training and the provided text to obtain the corresponding prediction scores. The prediction scores of the images in training are added in combination with the IResNet model, and the loss function value is calculated. The model is trained and optimized by continuously adjusting the loss function value. The specific model training formula is: F train =VisualEncoder(I M ) (14); L train =OneHot(L N ) (15); Among them, VisualEnconder is the image encoder of the CLIP model, where M is the number of images in a batch, I represents the images in the batch; OneHot is the encoder function, which is used to encode N categories; L is the category text; F train To extract the features of the image; L train is the text encoding vector processed by the encoder; β and α are hyperparameters; f test is the image feature of the image to be classified; A is the cosine similarity between the image to be classified and the image in the IResNet model, T represents the transpose operation of the matrix; W represents the text vector obtained by the CLIP encoder for C detection categories; Logits is the loss function of the model during training.

Images