Pedestrian falling detection method based on improved YOLOV8 algorithm

Abstract

The invention relates to the technical field of image processing, in particular to a pedestrian falling-to-ground detection method based on an improved YOLOV8 algorithm, and the method comprises the following steps: detection model training, detection model deployment and real-time reasoning, and pedestrian falling-to-ground identification and estimation. The detection model training is based on a Yov8-pose key point detection algorithm, and a c2f module is replaced by a c2f-cib module. Performing model training by defining the structure and parameters of the model, a proper loss function and an optimizer to obtain a training parameter weight of Yov8-pose; the detection model deployment and real-time reasoning comprises three parts of image preprocessing, model reasoning and prediction result post-processing, the preprocessing mainly comprises warpAffine affine transformation, and the post-processing mainly comprises two parts of code decoding and NMS. Pedestrian falling identification estimation adopts a method combining key point geometric information and a detection frame.

Description

Technical Field

[0001] The invention relates to a pedestrian fall detection method based on an improved YOLOV8 algorithm, and belongs to the technical field of image processing. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and Internet of Things technology, intelligent and automated systems have become a research and application hotspot, and computer vision has been widely used in various fields. In recent years, accidental falls among the elderly, construction workers and other groups are particularly common, and accidental falls may cause fractures, serious injuries or even death. Therefore, in the service and security industry, the use of computer vision to detect pedestrian falls can provide safety service guarantees for places where accidents are prone to occur, such as the elderly and construction workers. Detecting accidents in the first place through computer vision is of great significance to improving public safety. Summary of the invention

[0003] The purpose of the present invention is to design and implement a pedestrian fall detection method based on an improved YOLOV8 algorithm, which can detect the occurrence of accidents such as pedestrian falls in a variety of scenarios.

[0004] To achieve the above objectives, the present invention provides a pedestrian fall detection method based on an improved YOLOV8 algorithm, and the method includes the following processes: detection model training, detection model deployment and real-time reasoning, and pedestrian fall recognition estimation.

[0005] Furthermore, the detection model training is based on the YOLOV8-pose key point detection algorithm, the principle of which is to use a convolutional neural network (CNN) to extract features from the image, and then predict the position of the key points through a regression algorithm; before performing YOLOV8-pose training, it is necessary to prepare the data set and convert it into a YOLO format data set.

[0006] Furthermore, Yolov8-pose training includes the following steps:

[0007] Step 1: Load the dataset and divide it into training set, validation set and test set;

[0008] Step 2: Define the model, use the model definition code of YOLOV8-pose to define the structure and parameters of the model;

[0009] Step 3: Define the loss function and optimizer to optimize the model during training;

[0010] Step 4: Train the model, use the training set to train the model, and monitor indicators such as loss and accuracy during the training process in real time;

[0011] Step 5: Test the model, use the test set to test the trained model and evaluate its performance;

[0012] Furthermore, the detection model training obtains the parameter file (.pt file) of the trained model, and then generates a .plan file for building an inference engine on the robot through a series of conversions. TensorRT is used to build an engine that can quickly infer on the robot, and CUDA / C++ and GPU are used to perform parallel calculations on pre-processing / post-processing, further improving the detection and inference speed of YOLOV8.

[0013] Furthermore, after the construction of the inference engine is completed, some preprocessing and postprocessing parts are still needed to finally form a complete inference process. The complete YOLOV8 inference includes three parts: image preprocessing, model inference, and prediction result postprocessing. The preprocessing mainly includes warpAffine affine transformation, and the postprocessing mainly includes decode decoding and NMS. The GPU on the edge device is used to perform parallel computing on the deployed program to achieve the effect of accelerating the program.

[0014] Furthermore, the image preprocessing mentioned is to scale the image proportionally, then center it, and fill the excess part. This process is also called adding gray bars (i.e. letterbox). The whole process can be divided into three steps:

[0015] Step 1: Geometric scaling, implemented by matrix S (the scaling factor scale is the minimum value of the width ratio and height ratio of the target image to the source image);

[0016] Step 2: Translate the center of the image to the coordinate origin of the upper left corner, using matrix O;

[0017] Step 3: Translate the image to the center of the target location using matrix T;

[0018] The three matrices of scaling matrix S, translation matrix O, and translation matrix T can be combined into a matrix M (M=TOS), where the M matrix is ​​called an affine transformation matrix. The affine transformation matrix M is shown in the following formula:

[0019]

[0020]

[0021] Where Origin represents the source image, Dst represents the target image, (x, y) represents the coordinates of any pixel point on the source image, and (x', y') represents the coordinates of any pixel point on the target image.

[0022] The preprocessed image information is input into the inference engine for high-speed inference.

[0023] Furthermore, the post-processing of the prediction results mainly includes two parts: decoding of key points and prediction boxes and NMS non-maximum suppression; the programs of these two parts are accelerated by CUDA; the inference information will be distributed in the memory according to the structure of the onnx model, and we can read the information we need to decode from the memory according to the corresponding dimensions and arrangement order of the data.

[0024] Furthermore, the decoder adopts a top-down posture point estimation algorithm to first detect all human body detection frames in the image, and then recognize the posture according to each detection frame.

[0025] Furthermore, the pedestrian fall estimation method used in the case combines key point geometric information with detection frames. First, the pedestrians in the video stream are detected using the YOLO-v8 model to extract the pedestrian detection frames and key point information. In the key point recognition stage, even if some key points are not detected due to occlusion and other issues, the system can infer the approximate position of the missing key points through the identified key points, and use fitting techniques such as random forests to predict the possible positions of these points. The extracted features include the relative position of the key points, the angles between the body parts and the ground, and the relative distances between the body parts. These geometric information and the detection frames together constitute the input of the decision tree model; the decision tree model is trained to recognize three states: falldown, suspect, and person.

[0026] Furthermore, data enhancement strategies were adopted, such as simulating key point occlusion and increasing posture diversity, to ensure that the model can adapt to a variety of practical application scenarios; in addition, cross-validation techniques were used during model training to optimize parameter settings and verify the generalization ability of the model.

[0027] Furthermore, the fall recognition also adds some post-processing, adding logic or filters after the detection process, such as using time series analysis to reduce false positives. For example, a brief posture change may not be immediately judged as a fall.

[0028] The invention relates to the technical field of image processing, and in particular to a method for detecting a pedestrian falling down based on an improved YOLOV8 algorithm. The method comprises the following processes: detection model training, detection model deployment and real-time reasoning, and pedestrian falling down recognition estimation; the detection model training is based on a Yolov8-pose key point detection algorithm, and a c2f module is replaced by a c2f-cib module; the model training is performed by defining the structure and parameters of the model, a suitable loss function and an optimizer to obtain the training parameter weights of Yolov8-pose; the detection model deployment and real-time reasoning include three parts: image preprocessing, model reasoning, and prediction result post-processing, wherein the preprocessing mainly includes warpAffine affine transformation, and the post-processing mainly includes decode decoding and NMS; the pedestrian falling down recognition estimation adopts a method combining key point geometric information with a detection frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is the key point and detection box reasoning flow chart of the present invention.

[0031] Figure 2 It is a fall estimation flow chart of the present invention.

[0032] Figure 3 It is a precision-recall curve diagram of the human body bounding box detection of the present invention.

[0033] Figure 4 It is a graph of the posture detection precision-recall of the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0035] See also Figure 1 The present invention provides a pedestrian fall detection method based on the YOLOV8 algorithm, and the method includes the following processes: detection model training, detection model deployment and real-time reasoning, and pedestrian fall recognition estimation.

[0036] Yolov8-pose training includes the following steps:

[0037] Step 1: Load the dataset and divide it into training set, validation set and test set;

[0038] Step 2: Define the model. Use the model definition code of Yolov8-pose to define the structure and parameters of the model.

[0039] Step 3: Define the loss function and optimizer to optimize the model during training;

[0040] Step 4: Train the model, use the training set to train the model, and monitor indicators such as loss and accuracy during the training process in real time;

[0041] Step 5: Test the model. Use the test set to test the trained model and evaluate its performance.

[0042] Step 6: Detect model training to obtain the parameter file (.pt file) of the trained model, and then generate a .plan file for building an inference engine on the robot through a series of conversions. Use TensorRT to build an engine that can quickly infer on the robot, and use CUDA / C++ and GPU to perform parallel calculations for pre-processing / post-processing to further improve the detection and inference speed of YOLOv8.

[0043] The following is a further explanation based on the specific implementation steps:

[0044] Step S1: Image preprocessing is to scale the image proportionally, then center it, and fill the excess part. This process is also called adding gray bars (i.e., letterbox). The whole process can be divided into three steps:

[0045] Step S1-1: geometric scaling, implemented by matrix S (scaling factor Scale is the minimum value of the ratio of width and height between the target image and the source image);

[0046] Step S1-2: translate the center of the image to the coordinate origin of the upper left corner, implemented by matrix O;

[0047] Step S1-3: translate the image to the center of the target position, implemented by matrix T;

[0048] The three matrices of scaling matrix S, translation matrix O, and translation matrix T can be combined into a matrix M (M=TOS), where the M matrix is ​​called an affine transformation matrix. The affine transformation matrix M is shown in the following formula:

[0049]

[0050]

[0051] Among them, Origin represents the source image, Dst represents the target image, (x, y) is the coordinate of any pixel point on the source image, and (x', y') is the coordinate of any pixel point on the target image.

[0052] Step S2: The pre-processed image information is input into the constructed inference engine. The inference engine performs high-speed inference, and the information obtained still needs some post-processing to obtain the final required information.

[0053] Step S3: The post-processing of the prediction results mainly includes decoding of key points and prediction boxes and NMS non-maximum suppression. Both of these parts of the program are accelerated by CUDA. The inference information will be distributed in the memory according to the structure of the onnx model. We only need to read the information we need to decode from the memory according to the corresponding dimensions and arrangement order of the data.

[0054] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A pedestrian fall detection method based on YOLOV8 algorithm, characterized in that: The steps include: Step 1: Detection model training; Step 2: Detection model deployment and real-time inference; Step 3: Pedestrian fall recognition and estimation.

2. The pedestrian fall detection method based on the YOLOV8 algorithm as claimed in claim 1, characterized in that: The detection model training is based on the Yolov8-pose key point detection algorithm, which uses a convolutional neural network (CNN) to extract features from the image and then predicts the location of the key points through a regression algorithm. Before performing Yolov8-pose training, you need to prepare the data set and convert it into a YOLO format data set.

3. The Yolov8-pose training as claimed in claim 2, characterized in that The steps include: Step 1: Load the dataset and divide it into training set, validation set and test set; Step 2: Define the model. Use the model definition code of Yolov8-pose to define the structure and parameters of the model. Step 3: Define the loss function and optimizer to optimize the model during training; Step 4: Train the model, use the training set to train the model, and monitor indicators such as loss and accuracy during the training process in real time; Step 5: Test the model. Use the test set to test the trained model and evaluate its performance.

4. The pedestrian fall detection method based on the YOLOV8 algorithm as claimed in claim 1, characterized in that: Claim 3 obtains the parameter file of the trained model, uses TensorRT to build an engine that can quickly infer on the robot, and uses CUDA / C++ and GPU to perform parallel calculations on pre-processing / post-processing, further improving the detection and inference speed of YOLOv8.

5. The YOLOv8 reasoning as claimed in claim 4, characterized in that The complete YOLOv8 reasoning includes three parts: image preprocessing, model reasoning, and prediction result post-processing. The preprocessing mainly includes warpAffine affine transformation, and the post-processing mainly includes decode decoding and NMS.

6. The image preprocessing method according to claim 5, characterized in that: The image is scaled proportionally, then centered and the excess is filled. This process is also called adding a gray bar (i.e., letterbox). The whole process can be divided into three steps: Step 1: Geometric scaling, implemented by matrix S (the scaling factor Scale is the minimum value of the width ratio and height ratio of the target image to the source image); Step 2: Translate the center of the image to the coordinate origin of the upper left corner, using matrix O; Step 3: Translate the image to the center of the target location using matrix T; The three matrices of scaling matrix S, translation matrix O, and translation matrix T can be combined into a matrix M (M=TOS), where the M matrix is ​​called an affine transformation matrix. The affine transformation matrix M is shown in the following formula: Where, Origin represents the source image, Dst represents the target image, (x, y) is the coordinate of any pixel point on the source image, and (x', y') is the coordinate of any pixel point on the target image. The preprocessed image information is input into the inference engine for high-speed inference.

7. The prediction result post-processing as claimed in claim 5, characterized in that: It mainly includes two parts: decoding of key points and prediction boxes and NMS non-maximum suppression. Both parts of the program use CUDA for accelerated computing. The inference information will be distributed in the memory according to the structure of the onnx model. We only need to read the information we need to decode from the memory according to the corresponding dimensions and arrangement order of the data.

8. The decode as claimed in claim 7, characterized in that: A top-down posture point estimation algorithm is used to first detect all human detection frames in the image, and then recognize the posture based on each detection frame.

9. The pedestrian fall detection method based on the YOLOV8 algorithm as claimed in claim 1, characterized in that: The method of combining key point geometric information with detection frames is adopted. First, the pedestrians in the video stream are detected using the YOLO-v8 model to extract the detection frames and key point information of the pedestrians. In the key point recognition stage, even if some key points are not detected due to occlusion and other problems, the system can infer the approximate position of the missing key points through the recognized key points, and use fitting techniques such as random forests to predict the possible positions of these points; the extracted features include the relative positions of the key points, the angles between the body parts and the ground, and the relative distances between the body parts; these geometric information and the detection box together constitute the input of the decision tree model; the decision tree model is trained to recognize three states: fall down, suspect, and person.

10. The pedestrian fall detection method based on the YOLOV8 algorithm as claimed in claim 1, characterized in that: Data enhancement strategies, such as simulating key point occlusion and increasing posture diversity, were adopted to ensure that the model can adapt to a variety of practical application scenarios; in addition, cross-validation techniques were used during model training to optimize parameter settings and verify the generalization ability of the model.

Images