Pedestrian tumble detection method, apparatus and device, and storage medium
Through deep learning technology, the object detection model and the fall detection model are used, combined with the feature extraction and fusion of whole-body images and key point images, the problems of pedestrian fall detection in environmental factors and model training expansion are solved, and efficient and reliable fall detection effect is achieved.
Patent Information
- Application Number
- CN202411967858.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-15
AI Technical Summary
The prior art is susceptible to environmental factors when using deep learning to detect pedestrian falls, and model training is difficult to scale, affecting model performance.
A pedestrian fall detection method is adopted. By acquiring the target image and inputting it to the pretrained object detection model, the whole body image and key point image are extracted, and then these images are input to the pretrained fall detection model. The features are extracted and fused using a dual-stream network and feature fusion device, and the final output score results are performed for detection.
Under complex scenarios and uncertain factors, the reliability and real-time nature of pedestrian fall detection can be ensured, and low-cost, efficient and accurate technical means can be provided to monitor the safety status of pedestrians in public places.
Smart Images
Figure CN119832641A_ABST
Abstract
Description
Background Art
[0002] At present, the application of deep learning methods in the field of computer vision is becoming more and more popular. At the same time, with the widespread distribution of surveillance cameras in shopping malls, residential areas, stations, campuses and even streets, deep learning and convolutional neural networks (CNN) are increasingly used in applications such as pedestrian detection, tracking systems and fall detection. Neural networks are gradually becoming an ideal choice in the field of image classification and target detection.
[0003] However, in many real-world application scenarios, due to the diversity and complexity of human appearance features and postures, it is difficult to achieve ideal results by simply relying on appearance information to train convolutional neural network classifiers. Today's popular action recognition algorithms mainly recognize human motion trajectories. For example, optical flow data can distinguish dozens of postures. However, in certain specific scenarios, due to the influence of external environmental factors, such as the pedestrian's fall being blocked, resulting in the camera not capturing the pedestrian's fall trajectory, it is difficult to make a reliable judgment. At the same time, video clips of the fall process are difficult to obtain, resulting in the inability to expand model training, and limited training data will also damage model performance. Summary of the invention
[0004] This specification provides a pedestrian fall detection method to solve the problem that the existing technology is easily affected by environmental factors when using deep learning for depth detection, and the model training is difficult to expand, which affects the model performance.
[0005] To achieve the above object, the present invention adopts the following technical solution:
[0006] In a first aspect of the present specification, a pedestrian fall detection method is provided, comprising:
[0007] Obtain a target image of a pedestrian to be detected;
[0008] The target image is input into a pre-trained target detection model to obtain a full-body image and a key point image of the pedestrian to be detected, wherein the target detection model includes a backbone network and a target detection head and a key point detection head connected to the backbone network, wherein the backbone network is composed of a stack of residual modules composed of a convolutional neural network, and is used to extract features from the target image; the target detection head is composed of a three-way stacked convolutional network followed by a fully connected layer, and is used to predict features at different levels and determine the full-body image; and the key point prediction head is composed of a one-way stacked convolutional neural network, and is used to determine the key point image;
[0009] Inputting the whole body image and the key point image into a pre-trained fall detection model to obtain a scoring result output by the fall detection model, wherein the fall detection model includes a two-stream network and a feature fusion device, wherein the two-stream network is composed of two convolutional neural networks and is used to extract separable features of the whole body image and the key point image; the feature fusion device is composed of three-layer multi-layer perceptrons and is used to map the separable features into fusion features to determine the scoring result;
[0010] According to the scoring result, the detection result is output.
[0011] In some preferred embodiments, the training process of the target detection model includes:
[0012] Pre-training the initial detection model by presetting the first data set to determine the initial weights;
[0013] Obtain a key point detection data set, and divide the key point detection data set into a training set and a test set according to a preset data division ratio;
[0014] Based on the initial weights, the initial detection model is iteratively trained through the training set, and an evaluation result is determined through the test set, so as to iteratively adjust the hyperparameters of the initial detection model according to the training results until the evaluation result meets the preset expected values.
[0015] In some preferred embodiments, the step of obtaining a key point detection data set includes:
[0016] Acquire a second data set, wherein the second data set includes pedestrian images from multiple different sources, with different characteristics, and in different acquisition methods;
[0017] Preprocessing the second data set, wherein the preprocessing includes image enhancement based on Mosaic, affine transformation based on Random, image based on Mixup, and image transformation based on random parameter adjustment;
[0018] The preprocessed second data set is labeled to obtain the key point detection data set.
[0019] In some preferred embodiments, the dual-stream network includes an image stream network and a skeleton stream network, the image stream network and the skeleton stream network are composed of stacked convolutional neural networks, and are respectively used to extract separable features of the whole body image and the key point image, and the feature fusion device maps the separable features into fusion features by constructing a parallel mapping head;
[0020] The fall detection model also includes a discriminator, and the feature fusion device inputs the fused features into the discriminator so that the discriminator outputs a scoring result of the fused features.
[0021] In some preferred embodiments, the training process of the fall detection model includes:
[0022] Determining a training data set for the target detection model;
[0023] Extracting a full-body image of a pedestrian from a training data set of the target detection model;
[0024] Determine the coordinates of the pedestrian's full-body image using a preset key point detection algorithm, and colorize each limb part in the pedestrian's full-body image to determine a skeleton image;
[0025] Dividing the pedestrian full body image and the skeleton image into a falling image and a non-falling image respectively;
[0026] Based on the fall image and the non-fall image, the initial network is trained using a preset loss function, and the fall detection model is obtained after the training is completed.
[0027] In some preferred embodiments, the loss function of the initial network during training is:
[0028] FL(pt)=-(1-pt) Υ log(pt)
[0029] Where γ is a hyperparameter used to adjust the weight factor, and pt is the probability score of falling behavior.
[0030] In some preferred embodiments, determining the test result according to the scoring result includes:
[0031] Determine whether the scoring result is greater than a preset threshold;
[0032] If yes, it is determined that there is a pedestrian falling behavior in the target image, and a detection result of the pedestrian falling and an alarm signal are output;
[0033] If it is determined that a pedestrian falls in the target image, the target image is stored in a preset database, and the database is used to update a training data set of the fall detection model.
[0034] In a second aspect of the present specification, a pedestrian fall detection device is provided, comprising:
[0035] A data acquisition module, used to acquire a target image of a pedestrian to be detected;
[0036] A first detection module, used to input the target image into a pre-trained target detection model to obtain a full-body image and a key point image output by the target detection model, wherein the target detection model includes a backbone network and a prediction head, wherein the backbone network is used to extract features from the target image, and the prediction head includes a multi-level target detection head and a key point estimation head, which is used to perform prediction based on the result of feature extraction and output the full-body image and the key point image;
[0037] a second detection module, configured to input the whole body image and the key point image into a pre-trained fall detection model to obtain a scoring result output by the fall detection model, wherein the fall detection model includes a two-stream network and a feature fusion device, wherein the two-stream network is configured to extract separable features of the whole body image and the key point image, and the feature fusion device is configured to map the separable features into fused features to determine the scoring result;
[0038] A data output module is used to output the detection result according to the scoring result.
[0039] A third aspect of the present specification provides a pedestrian fall detection device, comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions being used to be executed by the processor to implement the pedestrian fall detection method described in the first aspect.
[0040] According to a fourth aspect of the present specification, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the pedestrian fall detection method according to the first aspect.
[0041] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:
[0042] This method uses deep learning technology and relies on deep training of massive raw image data to continuously optimize its recognition ability and response speed. Based on the pedestrian's full-body image and key point image, the method performs diversion detection, integrating the uniqueness and stability of the pedestrian's full-body appearance information and skeleton information, thereby ensuring that the fall state can be determined only by static images, and reliability and real-time performance can be guaranteed in complex scenes and uncertain factors. It can monitor the safety status of pedestrians in public places with low-cost, efficient and accurate technical means, especially in scenes where the elderly, children or people with physical disabilities may encounter the risk of falling, and provide timely assistance alarms. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0044] Figure 1 A flowchart of a pedestrian fall detection method provided in an embodiment of this specification;
[0045] Figure 2 A schematic diagram of the structure of a pedestrian fall detection device provided in one embodiment of this specification. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0047] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0048] Please refer to Figure 1 ,like Figure 1 As shown, the first embodiment of the present application proposes a pedestrian fall detection method, the method comprising:
[0049] Step S10, obtaining a target image of a pedestrian to be detected;
[0050] Step S20, inputting the target image into a pre-trained target detection model to obtain a full-body image and a key point image of the pedestrian to be detected, wherein the target detection model includes a backbone network and a target detection head and a key point detection head connected to the backbone network, wherein the backbone network is composed of a stack of residual modules composed of a convolutional neural network, and is used to extract features from the target image, and the target detection head is composed of a three-way stacked convolutional network followed by a fully connected layer, and is used to predict features at different levels and determine the full-body image, and the key point prediction head is composed of a one-way stacked convolutional neural network, and is used to determine the key point image;
[0051] In this embodiment, a target image containing the pedestrian to be detected is first obtained by using a suitable image acquisition device or by capturing a fixed frame from a video stream. This image serves as basic data for subsequent detection and analysis in this embodiment.
[0052] The target detection model in this embodiment includes a backbone network and a prediction head. The backbone network is responsible for extracting features of the target image, such as extracting representative feature information from the image through a series of convolution, pooling and other operations.
[0053] In this embodiment, each residual module is composed of 1*1, 3*3, 1*1, batch normalization and global pooling. The backbone network is followed by a target detection head and a key point detection head, which output the coordinates of the pedestrian in the image to be detected and the coordinates of the key points, respectively. The target detection prediction head is composed of a three-way stacked convolutional network followed by a fully connected layer, which is used to predict features at different levels; the key point prediction head is composed of a one-way stacked convolutional neural network, which outputs the coordinates of different key point positions of pedestrians.
[0054] The multi-level target detection head and key point estimation head in the prediction head will make predictions based on the feature results extracted by the backbone network, and finally output a full-body image and a key point image. The full-body image can present the overall appearance of the pedestrian, while the key point image clearly identifies the position information of the key skeleton, limbs and other parts of the human body, providing a key basis for further accurate judgment of the pedestrian's status. As an example, the key point image includes but is not limited to: nose, left eye, right eye, left ear, right ear, left shoulder, right button, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.
[0055] In this embodiment, after all the pedestrian cropped images and skeleton images are obtained by using the target detection model, the images that do not meet the quality standards are screened and the image blocks that are too small and smaller than the preset pixels (preferably, the preset pixels are set to 30,000) and the images that are severely blocked are discarded. It can be understood that the image blocks with too small pixel values are discarded because they may not provide sufficiently clear and complete pedestrian information; at the same time, the images that are severely blocked are also discarded because their key parts are blocked, which may cause deviations or inaccuracies in subsequent analysis.
[0056] Based on this, the data quality of subsequent processing can be effectively ensured and the accuracy and reliability of the overall detection and analysis can be improved.
[0057] Step S30, inputting the whole body image and the key point image into a pre-trained fall detection model to obtain a scoring result output by the fall detection model, wherein the fall detection model includes a two-stream network and a feature fusion device, wherein the two-stream network is composed of a two-way convolutional neural network, and is used to extract separable features of the whole body image and the key point image; the feature fusion device is composed of a three-layer multi-layer perceptron, and is used to map the separable features into fusion features to determine the scoring result;
[0058] In this step, the full-body image and key point image of the pedestrian obtained after preliminary processing are used as input data and sent to the pre-trained fall detection model. Since these images represent key information such as the appearance and posture of the pedestrian, they serve as an important basis for the fall detection model to judge fall behavior.
[0059] In the process of judging the falling behavior, the two-stream network in the fall detection model starts to work, and it performs independent feature extraction operations on the whole body image and the key point image, thereby obtaining separable features of two dimensions, including the information related to the fall contained in different levels, to ensure the comprehensiveness and effectiveness of feature extraction; then, the feature fusion device intervenes to map the separable features extracted by the two-stream network and transform them into a unique fusion feature; through this fusion process, the information of different dimensions is organically integrated to form a comprehensive feature representation, which provides a basis for the subsequent judgment of the falling behavior; finally, based on the fusion feature, the scoring result is output. The scoring result intuitively reflects the similarity between the current state of the pedestrian and the fall state, thus providing a basis and reference for subsequent behavioral decisions such as whether to trigger an alarm.
[0060] Step S40: output the detection result according to the scoring result.
[0061] In this embodiment, a judgment is made based on the scoring result to determine the content of the output detection result, specifically including:
[0062] Determine whether the scoring result is greater than a preset threshold; if so, determine that a pedestrian falls in the target image, and output a detection result of the pedestrian fall and an alarm signal; if it is determined that a pedestrian falls in the target image, store the target image in a preset database, and the database is used to update the training data set of the fall detection model.
[0063] Among them, the preset threshold is a critical value pre-set to distinguish between the normal state and the fall state of pedestrians. Its specific numerical setting is based on a large amount of experimental data and actual application scenario requirements, and comprehensively considers the generalization ability of the model to ensure the accuracy and reliability of detection.
[0064] Preferably, the preset threshold is set to 0.6.
[0065] It is easy to understand that if the scoring result is greater than the preset threshold, it indicates that the pedestrian's current state is highly suspected of falling. At this time, it is determined that there is a pedestrian falling behavior in the target image, and the detection result of the pedestrian falling should be output immediately. This result can be made known to the relevant monitoring system or staff in a timely manner, and an alarm signal will be issued at the same time so that appropriate rescue or intervention measures can be taken to ensure the safety of pedestrians.
[0066] When it is determined that a pedestrian falls in the target image, the target image is stored in a preset database, which is specifically used to store image data with fall behavior characteristics, so that the database can be used to update the training data set of the fall detection model in the later stage, making the data of the training data set more diverse, further optimizing the performance of the fall detection model, and improving its ability to distinguish different scenes and pedestrian states, thereby achieving continuous improvement and accuracy improvement of the model.
[0067] Furthermore, in the present embodiment, the training process of the target detection model specifically includes: pre-training the initial detection model by presetting a first data set to determine the initial weights; obtaining a key point detection data set, and dividing the key point detection data set into a training set and a test set according to a preset data division ratio; based on the initial weights, iteratively training the initial detection model through the training set, and determining the evaluation results through the test set, so as to iteratively adjust the hyperparameters of the initial detection model according to the training results until the evaluation results meet the preset expected values.
[0068] Among them, the step of obtaining a key point detection data set specifically includes: obtaining a second data set, the second data set includes pedestrian images from a variety of different sources, different characteristics, and different acquisition methods; preprocessing the second data set, the preprocessing includes Mosaic-based image enhancement, Random-based affine transformation, Mixup-based image, and image transformation based on random parameter adjustment; annotating the preprocessed second data set to obtain the key point detection data set.
[0069] Preferably, the data division ratio of the training set and the test set is 95% and 5%.
[0070] Specifically, the first data set mentioned above can be a relatively reliable public data set in the field. As an example, this embodiment uses the public COCO2016Keypoint data set for model pre-training, and after pre-training, it obtains a variety of different sources, different characteristics or different acquisition methods to expand the data set, covering different scenes such as indoor, outdoor, day and night changes, etc., different shooting equipment, different lighting conditions, different weather conditions, image and video data, and completes the annotation of the overall data set in a machine-assisted and manually reviewed manner. More source data contains more diverse samples, and these samples can cover a wider range of data distribution. Therefore, the model trained with multi-source data can learn more diverse feature representations, so that it has stronger generalization ability, enabling the model to better handle unseen data and show good performance in different tasks or scenarios.
[0071] In this embodiment, the pre-trained weights on the MsCOCO public dataset are used as the initial weights, which can significantly speed up the training of the model. By using the pre-trained weights, the initial detection model can determine the correct parameter direction more quickly, thereby accelerating the training process, significantly shortening the model training time, and improving the efficiency of research or application.
[0072] In addition, since the pre-trained weights are trained on a large amount of data, they already have a certain generalization ability. When these weights are used as initial weights, the model can more easily learn new features, thereby better adapting to the new data distribution and improving generalization ability. During the model training process, the learning rate gradually decreases. As the learning rate decreases, the step size of the model update in each iteration gradually decreases, which helps the model converge more stably in the later stages of training. The loss fluctuations during the training process are usually small, making it easier to converge to the optimal solution and improve the stability of the model. Gradually reducing the learning rate helps the model better generalize to unseen data in the later stages of training and avoid overfitting. It can achieve better performance in a shorter time and accelerate the training process. Using the strategy of gradually reducing the learning rate can save computing resources and reduce training costs. Using well-trained target detection models and classification models can realize the recognition of various hidden danger features and improve recognition accuracy.
[0073] When preprocessing the second data set, the preprocessing methods used include: Mosaic preprocessing, taking out four pictures, randomly cropping, scaling, cropping an area and rearranging them, and splicing four pictures into one picture; Random affine affine transformation randomly rotates, translates, scales, and cuts the pictures; MixUp changes the transparency of two pictures and merges them together; and other methods such as randomly adjusting the hue, saturation and transparency, and randomly flipping up and down and left and right. The performance and generalization ability of the model are improved, making it better adapted to state recognition in various complex application scenarios. It should be noted that if the picture is affine transformed, the corresponding human key point annotations are processed in the same way.
[0074] Furthermore, in this embodiment, the dual-stream network includes an image stream network and a skeleton stream network. The image stream network and the skeleton stream network are composed of stacked convolutional neural networks, which are respectively used to extract separable features of the whole body image and the key point image. The feature fuser maps the separable features into fused features by constructing a parallel mapping head. The fall detection model also includes a discriminator. The feature fuser inputs the fused features into the discriminator so that the discriminator outputs the scoring results of the fused features.
[0075] Further, in the present embodiment, the training process of the fall detection model includes: determining a training data set of the target detection model; extracting a full-body image of a pedestrian from the training data set of the target detection model; determining the coordinates of the full-body image of the pedestrian using a preset key point detection algorithm, and coloring each limb part in the full-body image of the pedestrian to determine a skeleton image; dividing the full-body image of the pedestrian and the skeleton image into a fall image and a non-fall image respectively; based on the fall image and the non-fall image, training the initial network using a preset loss function, and obtaining a fall detection model after the training is completed. In some embodiments, the loss function of the initial network during the training process is:
[0076] FL(pt)=-(1-pt) Υ log(pt)
[0077] Where γ is a hyperparameter used to adjust the weight factor, and pt is the probability score of falling behavior.
[0078] Among them, the dual-stream classification network includes an image stream and a skeleton stream network. Both networks are composed of stacked convolutional neural networks. ResNet-50 is selected as the stacking block. The full-body cropped image of the pedestrian and the key point skeleton are used as input respectively to extract separable features of two dimensions; then a parallel mapping head is constructed to map the features of the two dimensions to the same dimension and add them together. The parallel mapping is a 3-layer multi-layer perceptron with an intermediate dimension of 256, so as to obtain a unique fusion feature; finally, it is input into the fall discriminator with softmax as the activation function to obtain the probability distribution of falling or non-falling; since the number of non-fall data is much larger than the fall data, category imbalance is introduced. Therefore, this embodiment adopts the Focal loss loss function instead of the binary cross entropy optimization model, and γ in the Focalloss loss function is set to 2.
[0079] Specifically, in this embodiment, the training data set is divided into a training set and a test set in a ratio of 95% and 5%. The data enhancement for training the two-stream network only includes random horizontal flipping transformation and random processing of the entire chroma, saturation and transparency. After that, the pre-trained weights on the MsCOCO public data set are used as the initial weights, and the adam optimizer and stochastic gradient descent algorithm are used for algorithm training. The learning rate is reduced from 0.01 to 0.004 using a cosine curve to avoid gradient oscillation; the batch size is set to 32.
[0080] Please refer to Figure 2 The second embodiment of the present application proposes a pedestrian fall detection device, such as Figure 2 As shown, the pedestrian fall detection device includes:
[0081] A data acquisition module, used to acquire a target image of a pedestrian to be detected;
[0082] A first detection module, used to input the target image into a pre-trained target detection model to obtain a full-body image and a key point image output by the target detection model, wherein the target detection model includes a backbone network and a prediction head, wherein the backbone network is used to extract features from the target image, and the prediction head includes a multi-level target detection head and a key point estimation head, which is used to perform prediction based on the result of feature extraction and output the full-body image and the key point image;
[0083] a second detection module, configured to input the whole body image and the key point image into a pre-trained fall detection model to obtain a scoring result output by the fall detection model, wherein the fall detection model includes a two-stream network and a feature fusion device, wherein the two-stream network is configured to extract separable features of the whole body image and the key point image, and the feature fusion device is configured to map the separable features into fused features to determine the scoring result;
[0084] A data output module is used to output the detection result according to the scoring result.
[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of each module in the above-described device can refer to the corresponding process in the aforementioned method example, and will not be repeated here.
[0086] The third embodiment of the present application proposes a pedestrian fall detection device, comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned pedestrian fall detection method.
[0087] A fourth embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned pedestrian fall detection method.
[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process or relevant instructions of the pedestrian fall detection device described above and when the instructions in the computer-readable storage medium are executed can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0089] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A pedestrian fall detection method, characterized in that: The method comprises: Obtain a target image of a pedestrian to be detected; The target image is input into a pre-trained target detection model to obtain a full-body image and a key point image of the pedestrian to be detected, wherein the target detection model includes a backbone network and a target detection head and a key point detection head connected to the backbone network, wherein the backbone network is composed of a stack of residual modules composed of a convolutional neural network, and is used to extract features from the target image; the target detection head is composed of a three-way stacked convolutional network followed by a fully connected layer, and is used to predict features at different levels and determine the full-body image; and the key point prediction head is composed of a one-way stacked convolutional neural network, and is used to determine the key point image; The whole body image and the key point image are input into a pre-trained fall detection model to obtain a scoring result output by the fall detection model, wherein the fall detection model includes a two-stream network and a feature fusion device, wherein the two-stream network is composed of two convolutional neural networks and is used to extract separable features of the whole body image and the key point image; the feature fusion device is composed of three-layer multi-layer perceptrons and is used to map the separable features into fusion features to determine the scoring result; According to the scoring result, the detection result is output.
2. The pedestrian fall detection method according to claim 1, characterized in that: The training process of the target detection model includes: Pre-training the initial detection model by presetting the first data set to determine the initial weights; Obtain a key point detection data set, and divide the key point detection data set into a training set and a test set according to a preset data division ratio; Based on the initial weights, the initial detection model is iteratively trained through the training set, and an evaluation result is determined through the test set, so as to iteratively adjust the hyperparameters of the initial detection model according to the training results until the evaluation result meets the preset expected values.
3. The pedestrian fall detection method according to claim 2, characterized in that: The step of obtaining a key point detection data set includes: Acquire a second data set, wherein the second data set includes pedestrian images from multiple different sources, with different characteristics, and in different acquisition methods; Preprocessing the second data set, wherein the preprocessing includes image enhancement based on Mosaic, affine transformation based on Random, image based on Mixup, and image conversion based on random parameter adjustment; The preprocessed second data set is labeled to obtain the key point detection data set.
4. The pedestrian fall detection method according to claim 1, characterized in that: The dual-stream network includes an image stream network and a skeleton stream network, wherein the image stream network and the skeleton stream network are composed of stacked convolutional neural networks and are respectively used to extract separable features of the whole body image and the key point image, and the feature fusion device maps the separable features into fusion features by constructing a parallel mapping head; The fall detection model also includes a discriminator, and the feature fusion device inputs the fused features into the discriminator so that the discriminator outputs a scoring result of the fused features.
5. The pedestrian fall detection method according to claim 1, characterized in that: The training process of the fall detection model includes: Determining a training data set for the target detection model; Extracting a full-body image of a pedestrian from a training data set of the target detection model; Determine the coordinates of the pedestrian's full-body image using a preset key point detection algorithm, and colorize each limb part in the pedestrian's full-body image to determine a skeleton image; Dividing the pedestrian full body image and the skeleton image into a falling image and a non-falling image respectively; Based on the fall image and the non-fall image, the initial network is trained using a preset loss function, and the fall detection model is obtained after the training is completed.
6. The pedestrian fall detection method according to claim 5, characterized in that: The loss function of the initial network during training is: FL(pt)=-(1-pt) Υ log(pt) Where γ is a hyperparameter used to adjust the weight factor, and pt is the probability score of falling behavior.
7. The pedestrian fall detection method according to claim 1, characterized in that: Determining the detection result according to the scoring result includes: Determine whether the scoring result is greater than a preset threshold; If yes, it is determined that there is a pedestrian falling behavior in the target image, and a detection result of the pedestrian falling and an alarm signal are output; If it is determined that a pedestrian falls in the target image, the target image is stored in a preset database, and the database is used to update a training data set of the fall detection model.
8. A pedestrian fall detection device, characterized in that: The device comprises: A data acquisition module, used to acquire a target image of a pedestrian to be detected; A first detection module, used to input the target image into a pre-trained target detection model to obtain a full-body image and a key point image output by the target detection model, wherein the target detection model includes a backbone network and a prediction head, wherein the backbone network is used to extract features from the target image, and the prediction head includes a multi-level target detection head and a key point estimation head, which is used to perform prediction based on the result of feature extraction and output the full-body image and the key point image; a second detection module, configured to input the whole body image and the key point image into a pre-trained fall detection model to obtain a scoring result output by the fall detection model, wherein the fall detection model includes a two-stream network and a feature fusion device, wherein the two-stream network is configured to extract separable features of the whole body image and the key point image, and the feature fusion device is configured to map the separable features into fused features to determine the scoring result; A data output module is used to output the detection result according to the scoring result.
9. A pedestrian fall detection device, characterized in that: include: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the pedestrian fall detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the pedestrian fall detection method according to any one of claims 1 to 7.