Video analytics algorithms and edge devices

Abstract

This invention provides a video analysis algorithm, belonging to the field of intelligent video analysis technology. It solves the problem that existing methods require building different models for different types of tasks. This video analysis algorithm includes the following steps: Step S1: Constructing a person recognition model; Step S2: Person attribute and behavior recognition model; Step S3: Connecting the person recognition model and the person attribute and behavior recognition model. This invention has advantages such as real-time prediction.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video analysis technology, and specifically relates to a video analysis algorithm and edge device. Background Technology

[0002] Intelligent video analytics is a crucial component of building smart factories. Identifying personnel attributes and behaviors can efficiently detect abnormal actions, issue alerts, and store useful information, thereby assisting security personnel in handling crises more quickly and minimizing missed and false alarms. When the intelligent analytics system identifies abnormal personnel behavior, it immediately pushes alarm information to the video surveillance management center, enabling timely response measures.

[0003] The video analytics system is based on on-site installed terminal cameras and uses AI vision algorithms deployed on edge devices to perform real-time detection and analysis of video stream content. It detects human behavior in the video stream using video analytics algorithms; if an anomaly is detected, it automatically generates an alert, saves images or videos of the abnormal behavior, and returns the detection results.

[0004] In video analytics systems for smart factories, personnel attributes and behaviors are complex and diverse. Personnel attributes include identifying hard hats, work clothes, and reflective vests, while personnel behaviors include identifying phone use, absence from work, and smoking. When modeling, training a separate model for each task presents challenges: a large number of models lead to deployment difficulties, and model scheduling becomes an issue. Therefore, to improve model inference and response time, it's necessary to integrate multiple different types of tasks into a single model. However, simply merging datasets from multiple tasks results in low model accuracy due to the small data size.

[0005] When analyzing surveillance video data, time efficiency must be considered. Furthermore, directly identifying objects in video is computationally challenging. Therefore, most algorithms extract key image frames from the video, transforming the video detection task into an image detection task, resulting in faster execution and higher performance. Currently, image object detection methods are broadly divided into two types. One is the Two-Stage object detection method, represented by R-CNN, which first generates candidate boxes, then classifies their content, and corrects their positions. The other is the One-Stage method, represented by YOLO, which directly predicts and classifies candidate boxes without pre-generating them. Two-Stage algorithms offer high accuracy but poor real-time performance and are less effective at detecting small targets, making them more suitable for high-precision object detection. One-Stage algorithms offer high real-time performance but relatively lower accuracy, making them suitable for scenarios with high real-time requirements.

[0006] Edge computing refers to performing data analysis and model prediction at network edge nodes, thereby accelerating processing speed and reducing the impact of latency and bandwidth limitations. Furthermore, using algorithms deployed on edge devices enables real-time predictions, improving the security of sensitive and private data and the reliability of business data. Summary of the Invention

[0007] The purpose of this invention is to address the aforementioned problems in the prior art by providing a video analysis algorithm and edge device for real-time detection of abnormal behavior of factory workers.

[0008] The first objective of this invention can be achieved through the following technical solution: a video analysis algorithm, characterized by comprising the following steps:

[0009] Step S1: Construct a personnel identification model:

[0010] Step S1.a: Train a YOLOv5 model on the MS COCO dataset, denoted as M. g ;

[0011] Step S1.b: M g Model inference filters out categories other than personnel, retaining only personnel, to obtain a personnel recognition model, denoted as M. p ;

[0012] Step S2: Personnel Attribute and Behavior Recognition Model:

[0013] Step S2.a: Perform scaling preprocessing on the original image V in the personnel attribute behavior recognition dataset to obtain a scaled image V′. The size of the original image V is the original size and is denoted as (w v h v The size of the scaled image V′ is the scaling size and is denoted as (w v′ h v′ Using the aforementioned personnel identification model M p The positions of people in the detected image are obtained, and the center point, width, height, and confidence information are recorded as (x′, y′, w′, h′, c). The original positions of the detected people in the original image V are restored using the original dimensions and recorded as (x′, y′, w′, h′, c). l , yl x r， y r c r For example, formulas (1), (2), and (3):

[0014]

[0015] (x l y l， x r yr )=(x′ l y l , x′ r ,y′)×(w v h v w v h v (2);

[0016] c∈[0,1] (3);

[0017] Among them, (x l， y l ), (x r， y r ), c represent the coordinates of the upper left corner, the lower right corner, and the confidence level of the person detected in the original image V, respectively;

[0018] Step S2.b: Based on the original location information (x) l y l x r y r c) Set the personnel detection confidence threshold to C. When the threshold is greater than C, the personnel detection is considered correct, and the personnel image V is cropped based on the original location information. p The size is denoted as (w) p h p );

[0019] Step S2.c: Record the labeled personnel attribute and behavior information in the personnel attribute and behavior recognition dataset as... And based on the personnel location (x) l y l x r y r Construct the personnel image V p The attribute behavior information in the text is recorded as l represents the personnel attribute behavior category. It is necessary to determine whether the personnel candidate box and the behavior candidate box have an intersection. The intersection-union comparison is used to determine whether labeling is required. When labeling is required, the conversion is performed according to formula (4):

[0020]

[0021] Step S2.d: Train a personnel attribute and behavior recognition model using YOLOv5 on an attribute and behavior recognition dataset labeled with personnel attribute and behavior information. The model input is a people image V cropped from the original image V. p The output is labeled personnel attribute and behavior information. The transformed (l, x p y pw p h p The conversion process is shown in formulas (5) and (6):

[0022]

[0023] (x p y p w p h p )=(x p′ y p′ w p′ h p′ )÷(w p h p w p h p (6);

[0024] Step S3: Connect the personnel identification model and the personnel attribute and behavior identification model:

[0025] Step S3.a: Read the video stream and obtain the original image V;

[0026] Step S3.b: Use the personnel recognition model trained in step S1.b to detect whether there are personnel in the original image V;

[0027] Step S3.c: When a person target is detected in step S1.b, based on the person's location (x l y l x r y r Crop out the personnel image V p ;

[0028] Step S3.d: Use the personnel attribute behavior recognition model from step S.2.d Image V of personnel p Predict attribute behavior category and location

[0029] Step S3.e: Combining the results of model detection in steps S3.c and S3.d, obtain the personnel attribute and behavior information on the original image V. And save the screenshot. The position coordinate transformation is shown in the following formula (7);

[0030]

[0031] The video analysis algorithm described above also includes the following steps:

[0032] Step S4: Deploy personnel identification models and personnel attribute behavior recognition models on edge devices for real-time prediction and automatic alerts.

[0033] Step S4.a: Convert the format of the personnel identification model and the personnel attribute and behavior identification model;

[0034] Step S4.b: Configure the environment on the edge device and deploy the personnel identification model and personnel attribute behavior identification model;

[0035] Step S4.c: When the edge device detects abnormal human behavior, it automatically generates an alarm message, takes a screenshot, and transmits it to the monitoring center along with the alarm message.

[0036] In the video analysis algorithm described above, step S3.a involves reading the video stream using OpenCV.

[0037] In the video analysis algorithm described above, step S4.a involves converting the PyTorch format models generated by the personnel recognition model and the personnel attribute behavior recognition model into ONNX format.

[0038] An edge device is characterized by comprising an edge device body, an input terminal, an output terminal, one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include methods for performing the video analysis algorithms described above.

[0039] Compared with existing technologies, the present invention has the advantage of real-time prediction. Attached Figure Description

[0040] Figure 1 This is a flowchart of a video analysis algorithm according to the present invention.

[0041] Figure 2 This is an alarm flowchart of a video analysis algorithm according to the present invention.

[0042] Figure 3 This is a flowchart illustrating a video analysis algorithm and edge device according to the present invention.

[0043] Figure 4 This is a flowchart of the recognition process of a video analysis algorithm according to the present invention.

[0044] Figure 5 This is a schematic diagram demonstrating a video analysis algorithm of the present invention. Detailed Implementation

[0045] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0046] like Figures 1-4 As shown, this video analysis algorithm includes the following steps:

[0047] Step S1: Construct a personnel identification model:

[0048] Step S1.a: Train a YOLOv5 model on the MS COCO dataset, denoted as M. g ;

[0049] Step S1.b: M g Model inference filters out categories other than personnel, retaining only personnel, to obtain a personnel recognition model, denoted as M. p ;

[0050] Step S2: Personnel Attribute and Behavior Recognition Model:

[0051] Step S2.a: Perform scaling preprocessing on the original image V in the personnel attribute behavior recognition dataset to obtain a scaled image V′. The size of the original image V′ is the original size and is denoted as (w v h v The size of the scaled image V′ is the scaling size and is denoted as (w v′ h v′ Using the aforementioned personnel identification model M p The positions of people in the detected image are obtained, and the center point, width, height, and confidence information are recorded as (x′, y′, w′, h′, c). The original positions of the detected people in the original image V are restored using the original dimensions and recorded as (x′, y′, w′, h′, c). l y l x r y r , c For example, formulas (1), (2), and (3):

[0052]

[0053] (x l y l x r y r )=(x′ l y′ l , x′ r y′ r )×(w v h v w v h v (2);

[0054] c∈[0,1] (3);

[0055] Among them, (x l y l ), (x r y r), c represent the coordinates of the upper left corner, the lower right corner, and the confidence level of the person detected in the original image V, respectively;

[0056] Step S2.b: Based on the original location information (x) l y l x r y r c) Set the personnel detection confidence threshold to C. When the threshold is greater than C, the personnel detection is considered correct, and the personnel image V is cropped based on the original location information. p The size is denoted as (w) p h p );

[0057] The threshold C for personnel detection confidence is set at 0.5. If it exceeds this value, according to (x l y l x r y r Cropping out the personnel image from the original image (V) p .

[0058] Step S2.c: Record the labeled personnel attribute and behavior information in the personnel attribute and behavior recognition dataset as... And based on the personnel location (x) l y l x r y r Construct the personnel image V p The attribute behavior information in the text is recorded as l represents the personnel attribute behavior category. It is necessary to determine whether the personnel candidate box and the behavior candidate box have an intersection. The intersection-union comparison is used to determine whether labeling is required. When labeling is required, the conversion is performed according to formula (4):

[0059]

[0060] Define a list of personnel attribute behavior categories, for example: [0: safety helmet, 1: mask, 2: reflective vest, 3: smoking, 4: using mobile phone]. Combine this with existing location annotations and add them to the cropped personnel image V. p The categories and locations of personnel attributes and behaviors are marked on the label.

[0061] Step S2.d: Train a personnel attribute and behavior recognition model using YOLOv5 on an attribute and behavior recognition dataset labeled with personnel attribute and behavior information. The model input is a people image V cropped from the original image V. p The output is labeled personnel attribute and behavior information. The transformed (l, x p yp w p h p The conversion process is shown in formulas (5) and (6):

[0062]

[0063] (x p y p w p h p )=(x p′ y p′ w p′ h p′ )÷(w p h p w p h p (6);

[0064] Step S3: Connect the personnel identification model and the personnel attribute and behavior identification model:

[0065] Step S3.a: Read the video stream and obtain the original image V;

[0066] Step S3.b: Use the personnel recognition model trained in step S1.b to detect whether there are personnel in the original image V;

[0067] Step S3.c: When a person target is detected in step S1.b, based on the person's location (x l y l x r y r Crop out the personnel image V p ;

[0068] Step S3.d: Use the personnel attribute behavior recognition model from step S.2.d Image V of personnel p Predict attribute behavior category and location

[0069] Step S3.e: Combining the results of model detection in steps S3.c and S3.d, obtain the personnel attribute and behavior information on the original image V. And save the screenshot. The position coordinate transformation is shown in the following formula (7);

[0070]

[0071] To elaborate further, it also includes the following steps:

[0072] Step S4: Deploy personnel identification models and personnel attribute behavior recognition models on edge devices for real-time prediction and automatic alerts.

[0073] Step S4.a: Convert the format of the personnel identification model and the personnel attribute and behavior identification model;

[0074] Use the model format conversion script provided by YOLOv5 to convert the PyTorch format model to the ONNX format model, making it readable by OpenCV.

[0075] Step S4.b: Configure the environment on the edge device and deploy the personnel identification model and personnel attribute behavior identification model;

[0076] Install Python and PyTorch environments on edge devices and deploy the two-stage personnel attribute recognition model on the edge devices.

[0077] Step S4.c: When the edge device detects abnormal human behavior, it automatically generates an alarm message, takes a screenshot, and transmits it to the monitoring center along with the alarm message.

[0078] To elaborate further, in step S3.a, the video stream is read using OpenCV.

[0079] To elaborate further, in step S4.a, the PyTorch format models generated by the personnel identification model and the personnel attribute and behavior identification model are converted into ONNX format.

[0080] This edge device includes an edge device body, an input terminal, an output terminal, one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, the programs including those for performing the video analysis algorithms described above.

[0081] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0082] Although this document uses a large number of technical terms, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of the invention; interpreting them as any additional limitation would be contrary to the spirit of the invention.

Claims

1. A video analysis algorithm, characterized in that, Includes the following steps: Step S1: Construct a personnel identification model: Step S1.a: Train a YOLOv5 model on the MS COCO dataset, denoted as M. g ; Step S1.b: M g Model inference filters out categories other than personnel, retaining only personnel, to obtain a personnel recognition model, denoted as M. p ; Step S2: Personnel Attribute and Behavior Recognition Model: Step S2.a: Perform scaling preprocessing on the original image V in the personnel attribute behavior recognition dataset to obtain a scaled image V′. The size of the original image V is the original size and is denoted as (w v h v The image is scaled to a new size, denoted as (w). v′ h v′ Using the aforementioned personnel identification model M p The positions of people in the detected image are obtained, and the center point, width, height, and confidence information are recorded as (x′, y′, w′, h′, c). The original positions of the detected people in the original image V are restored using the original dimensions and recorded as (x′, y′, w′, h′, c). l y l x r y r c), as in formulas (1), (2), and (3): (x l ,y l ,x r ,y r )=(x′ l ,y′ l ,x′ r ,y′ r )×(w v ,h v ,w v ,h v ) (2); c∈[0，1] (3)； Among them, (x l y l ), (x r y r ), c represent the coordinates of the upper left corner, the lower right corner, and the confidence level of the person detected in the original image V, respectively; Step S2.b: Based on the original location information (x) l y l x r y r c) Set the personnel detection confidence threshold to C. When the threshold is greater than C, the personnel detection is considered correct, and the personnel image V is cropped based on the original location information. p The size is denoted as (w) p h p ); Step S2.c: Record the labeled personnel attribute and behavior information in the personnel attribute and behavior recognition dataset as... And based on the personnel location (x) l y l x r y r Construct the personnel image V p The attribute behavior information in the text is recorded as l represents the personnel attribute behavior category. It is necessary to determine whether the personnel candidate box and the behavior candidate box have an intersection. The intersection-union comparison is used to determine whether labeling is required. When labeling is required, the conversion is performed according to formula (4): Step S2.d: Train a personnel attribute and behavior recognition model using YOLOv5 on an attribute and behavior recognition dataset labeled with personnel attribute and behavior information. The model input is a people image V cropped from the original image V. p The output is labeled personnel attribute and behavior information. The transformed (l, x p y p w p h p The conversion process is shown in formulas (5) and (6): (x p ,y p ,w p ,h p )=(x p′ ,y p′ ,w p′ ,h p′ )÷(w p ,h p ,w p ,h p ) (6); Step S3: Connect the personnel identification model and the personnel attribute and behavior identification model: Step S3.a: Read the video stream and obtain the original image V; Step S3.b: Use the personnel recognition model trained in step S1.b to detect whether there are personnel in the original image V; Step S3.c: When a person target is detected in step S1.b, based on the person's location (x l y l x r y r Crop out the personnel image V p ; Step S3.d: Use the personnel attribute behavior recognition model from step S.2.d Image V of personnel p Predict attribute behavior category and location Step S3.e: Combining the results of model detection in steps S3.c and S3.d, obtain the personnel attribute and behavior information on the original image V. And save the screenshot. The position coordinate transformation is shown in the following formula (7); 。 2. The video analysis algorithm according to claim 1, characterized in that, It also includes the following steps: Step S4: Deploy personnel identification models and personnel attribute behavior recognition models on edge devices for real-time prediction and automatic alerts. Step S4.a: Convert the format of the personnel identification model and the personnel attribute and behavior identification model; Step S4.b: Configure the environment on the edge device and deploy the personnel identification model and personnel attribute behavior identification model; Step S4.c: When the edge device detects abnormal human behavior, it automatically generates an alarm message, takes a screenshot, and transmits it to the monitoring center along with the alarm message.

3. The video analysis algorithm according to claim 1, characterized in that, Read the video stream using OpenCV.

4. The video analysis algorithm according to claim 1, characterized in that, Convert the PyTorch format models generated by the personnel identification model and the personnel attribute and behavior identification model into ONNX format.

5. An edge device comprising an edge device body, an input terminal, an output terminal, one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising an algorithm for performing a video analysis algorithm as described in any one of claims 1-4.

Images