Monitoring method and system for recognizing human activity behaviors

Abstract

The invention relates to the technical field of activity monitoring, in particular to a monitoring method and system for recognizing human activity behaviors. The method comprises the steps of building a dependent environment trained by target detection yo8 of a deep learning neural network, creating a data set according to the environment, setting data set configuration, using a pre-training model, and performing model training on training statements in the model. A tool is used in a recognition monitoring picture to judge human activity behaviors, the quality of a data set is effectively improved, the purpose of finally improving the precision of a neural network model is achieved, and a human damage environment can be stopped in time.

Description

Technical Field

[0001] The present application relates to the field of activity monitoring technology, and in particular to a monitoring method and system for identifying human activity behaviors. Background Art

[0002] Based on the behavioral activity background of human using tools, areas within the scope of human activities where the use of tools is prohibited are monitored, and the specific tools used are analyzed through monitoring. There is no clear distinction between axes, machetes, pull saws, oil saws, chain saws, and electric saws commonly used for illegal tree felling, and fishing boats, wooden boats, speedboats and other tools used for lake fishing. In the prior art, human behavior is analyzed through monitoring images, but similar situations always appear through human behavior, which is prone to misjudgment. The function of identifying the use of tools requires high computing power, and it is necessary to obtain a large amount of data information in the database for comparison. The recognition ability and recognition speed are limited, and it is not possible to identify the situation of human practical prohibited tools. Therefore, a monitoring method and system for identifying human activity behavior are urgently needed to monitor the above-mentioned behavior, solve the above-mentioned problems, and issue alarms and warnings for prohibited behaviors in the human activity area to achieve the purpose of reminder. Summary of the invention

[0003] The purpose of the present invention is to provide a monitoring method and system for identifying human activity behaviors, so as to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0004] Based on the behavioral activity background of human using tools, the tool targets appearing in the human activity scenes are analyzed by image analysis as the detection objects. Based on the target detection principle of deep learning, the common engineering vehicles such as sand trucks, dump trucks, excavators, loaders, etc. in the activity scene pictures, the axes, machetes, pull saws, oil saws, chainsaws, and electric saws commonly used for illegal tree felling, and the fishing boats, wooden boats, and speedboats used for lake fishing are classified and labeled; the deep learning framework pytorch is combined with the target detection yolov8 principle to train the target detection model for the data in the yolo annotation format, and the trained detection model is used to write a model application prediction program for the video images of the surveillance cameras in the actual scene. The input jpg pictures are detected and identified through the application service program. The target ships (fishing boats, wooden boats, speedboats), engineering vehicles (sand trucks, dump trucks, excavators, loaders), and illegal tree felling tools (axes, machetes, pull saws, oil saws, chainsaws, and electric saws) are output, so as to achieve the purpose of alarm and warning reminder.

[0005] A monitoring method for identifying human activity behavior, characterized in that the method includes building a dependent environment for target detection yolo8 training of a deep learning neural network, creating a data set according to the environment, setting the data set configuration, using a pre-trained model, and performing model training on the training statements in the model.

[0006] Furthermore, according to the monitoring screen segments acquired by the monitoring device for the acquired environment, the targets appearing in the human activity scene are taken as detection objects through the target detection image analysis and recognition technology.

[0007] The dataset is created (including data preprocessing, screening and cleaning, and target type annotation). The dataset is divided into training set train, test set test, and evaluation set val. It is divided according to the ratio of 0.9:0.05:0.05 or 0.8:0.1:0.1 as needed to obtain data training sets of different quantities. The dataset annotation format is converted into yolo format data, which is expressed in the txt text format of annotating the extracted target object box bbox information. Each image corresponds to a txt file. Each line in the txt file contains information about a target, including class, x_center, y_center, width, height format: after center normalization, it is as follows: 0 0.602168 0.601077 0.354276 0.395933 (Note: it represents the label label number, the ratio of the center horizontal coordinate to the image width, the ratio of the center vertical coordinate to the image height, the ratio of the bbox width to the image width, and the ratio of the bbox height to the image height).

[0008] Furthermore, the configuration of the dataset creates a yaml file to record the path location of the images and labels of the dataset and the corresponding number of categories and category lists. The dataset structure is as follows:

[0009]

[0010] Use the pre-trained model weight file yolov8n.pt as a starting parameter to participate in model training and set the initial parameters;

[0011] Use training statements to train the model: yolo task=detect mode=train model=yolov8n.yaml pretrained=yolov8n.pt data=classdata.yaml epochs=500batch=16. This training statement is to train the target detection model with yolov8n structure with a training batch of 500 rounds and a step size of 16 images into memory, using the actual self-annotated dataset Dataset; output the model file object_detect_best.pt.

[0012] Furthermore, the optical flow method is used to perform optical flow detection on N consecutive frames of video images;

[0013] The specific steps of using the optical flow method to detect the optical flow of video frames include:

[0014] S21: remove the optical flow vector whose intensity is less than the optical flow amplitude threshold T1 in each frame image;

[0015] S22: superimposing the optical flow information of N consecutive frames of images to obtain an optical flow intensity information map;

[0016] S23: Using the graph connectivity algorithm, obtain the optical flow intensity connected subgraph of the image;

[0017] S3: Use the pre-trained CSR-Net crowd counting network to extract the probability information of people distribution in N consecutive frames in the video;

[0018] The specific steps of extracting the probability information of personnel distribution of N consecutive frames in a video include:

[0019] S31: suppressing image regions in each frame where the distribution probability integral value is less than a preset threshold value T2;

[0020] S32: superimposing probability information of N consecutive frames to obtain personnel distribution probability information;

[0021] S4: Combine the optical flow intensity map with the person distribution probability map to obtain the tool information used in the video;

[0022] S5: Capture and output the tool information used in the video to form an enhanced training video dataset.

[0023] Furthermore, step S2 uses a global dense optical flow algorithm of image pyramid matching to calculate motion information; suppose an object with intensity I(x, y, t) has a constant intensity between two consecutive frames. When it moves a distance of dx and dy in one unit of time, it has a new intensity I(x+dx, y+dy, t+dt), that is, it satisfies the following formula: I(x, y, t) = I(x+dx, y+dy, t+dt)

[0024] x, y are the coordinates of the point, t is the current time; Taylor approximation is performed on the right side of the equation;

[0025] Image along time; the solution method is based on Gunnar Farneback's dense optical flow algorithm. The pixel intensity changes of all points between two frames are detected to obtain the optical flow information (u, v). In the specific algorithm of this patent, the scale relationship coefficient between the upper and lower layers of each pyramid layer pyr_scale = 0.5, the number of pyramid levels = 3, the mean window size winsize = 15, 3 iterations are performed for each image, the pixel neighborhood size poly_n = 5, the Gaussian standard deviation poly_sigma = 1.2, and the input flow is used as an approximation of the initial flow, and the number of consecutive frames N = 64.

[0026] Further, it has the following steps: S231: binarizing the optical flow intensity information map obtained in step S22, with a threshold of 128;

[0027] S232: Scan the binarized optical flow intensity information graph to obtain an optical flow intensity connected subgraph; P(i, j) represents the pixel value of the image at that point, starting from the image starting point P(0, 0), mark the connected subgraph, which includes the following specific steps: S233: Scan the image line by line, if P(r-1, c) = 0 and P(r, c) = 1, it is determined to be the starting point of the optical flow cluster, if P(r-1, c) = 1 and P(r, c) = 0, it is determined to be the end point of the optical flow cluster, and a unique identifier RI is set for it;

[0028] S234: For the optical flow clusters in all rows except the first row, if there is no overlap in r value between them and the existing clusters in the previous row, that is, the optical flow cluster P(r,c-1)≠P(r,c) holds everywhere, a new unique identifier RI is set for them;

[0029] S235: For the optical flow clusters in all rows except the first row, if there is only one region with r value overlapped with the previous row, then the clusters obtain the same identifier RI as the overlapping clusters in the previous row;

[0030] S236: For the optical flow clusters in all rows except the first row, if their r values ​​overlap with those of multiple regions in the previous row, the RI values ​​of the row area and all the overlapping clusters in the previous row are uniformly rewritten as the minimum RI value of these clusters, as a connected domain;

[0031] S237: summing up the optical flow intensities of the optical flow sub-graphs, and the sub-graphs with KRI ≥ 1000 are the areas to be judged using the tool.

[0032] Further, the coordinates of the tool judgment area selected in step S4 are normalized, the lower left corner of the video frame is taken as the (0,0) point, and the four poles of the tool judgment area, that is, the maximum and minimum values ​​of the x-axis and y-axis coordinates: Wmin, Wmax, Hmin and Hmax, are obtained; according to the input image ratio required for the subsequent neural network training using the data set, the center point (Cx, Cy) of the high-risk area is used as the center for length and width proportional expansion selection, and the selection box coordinates are expressed as (xmin, ymin, xmax, ymax), that is, the coordinates of the lower left corner and the upper right corner of the rectangular area;

[0033] When xmin≥Wmin and ymin≥Hmin and xmax≥Wmax and ymax≥Hmax, the selected area can be captured to form a new video clip.

[0034] The beneficial effects of the present invention are as follows: the method is based on real-life scenarios, identifies the use of tools in monitoring images to judge human activities, effectively improves the quality of data sets, and ultimately achieves the goal of improving the accuracy of neural network models, and human destruction of the environment can be stopped immediately. DETAILED DESCRIPTION

[0035] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0036] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0037] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand the advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0038] The purpose of the present invention is to provide a monitoring method and system for identifying human activity behaviors, so as to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0039] Based on the behavioral activity background of human using tools, the tool targets appearing in the human activity scenes are analyzed by image analysis as the detection objects. Based on the target detection principle of deep learning, the common engineering vehicles such as sand trucks, dump trucks, excavators, loaders, etc. in the activity scene pictures, the axes, machetes, pull saws, oil saws, chainsaws, and electric saws commonly used for illegal tree felling, and the fishing boats, wooden boats, and speedboats used for lake fishing are classified and labeled; the deep learning framework pytorch is combined with the target detection yolov8 principle to train the target detection model for the data in the yolo annotation format, and the trained detection model is used to write a model application prediction program for the video images of the surveillance cameras in the actual scene. The input jpg pictures are detected and identified through the application service program. The target ships (fishing boats, wooden boats, speedboats), engineering vehicles (sand trucks, dump trucks, excavators, loaders), and illegal tree felling tools (axes, machetes, pull saws, oil saws, chainsaws, and electric saws) are output, so as to achieve the purpose of alarm and warning reminder.

[0040] A monitoring method for identifying human activity behavior, characterized in that the method includes building a dependent environment for target detection yolo8 training of a deep learning neural network, creating a data set according to the environment, setting the data set configuration, using a pre-trained model, and performing model training on the training statements in the model.

[0041] Furthermore, according to the monitoring screen segments acquired by the monitoring device for the acquired environment, the targets appearing in the human activity scene are taken as detection objects through the target detection image analysis and recognition technology.

[0042] The dataset is created (including data preprocessing, screening and cleaning, and target type annotation). The dataset is divided into training set train, test set test, and evaluation set val. It is divided according to the ratio of 0.9:0.05:0.05 or 0.8:0.1:0.1 as needed to obtain data training sets of different quantities. The dataset annotation format is converted into yolo format data, which is expressed in the txt text format of annotating the extracted target object box bbox information. Each image corresponds to a txt file. Each line in the txt file contains information about a target, including class, x_center, y_center, width, height format: after center normalization, it is as follows: 0 0.602168 0.601077 0.354276 0.395933 (Note: it represents the label label number, the ratio of the center horizontal coordinate to the image width, the ratio of the center vertical coordinate to the image height, the ratio of the bbox width to the image width, and the ratio of the bbox height to the image height).

[0043] Furthermore, the configuration of the dataset creates a yaml file to record the path location of the images and labels of the dataset and the corresponding number of categories and category lists. The dataset structure is as follows:

[0044]

[0045] Use the pre-trained model weight file yolov8n.pt as a starting parameter to participate in model training and set the initial parameters;

[0046] Use training statements to train the model: yolo task=detect mode=train model=yolov8n.yaml pretrained=yolov8n.pt data=classdata.yaml epochs=500batch=16. This training statement is to train the target detection model with yolov8n structure with a training batch of 500 rounds and a step size of 16 images into memory, using the actual self-annotated dataset Dataset; output the model file object_detect_best.pt.

[0047] Furthermore, the optical flow method is used to perform optical flow detection on N consecutive frames of video images;

[0048] The specific steps of using the optical flow method to detect the optical flow of video frames include:

[0049] S21: remove the optical flow vector whose intensity is less than the optical flow amplitude threshold T1 in each frame image;

[0050] S22: superimposing the optical flow information of N consecutive frames of images to obtain an optical flow intensity information map;

[0051] S23: Using the graph connectivity algorithm, obtain the optical flow intensity connected subgraph of the image;

[0052] S3: Use the pre-trained CSR-Net crowd counting network to extract the probability information of people distribution in N consecutive frames in the video;

[0053] The specific steps of extracting the probability information of personnel distribution of N consecutive frames in a video include:

[0054] S31: suppressing image regions in each frame where the distribution probability integral value is less than a preset threshold value T2;

[0055] S32: superimposing probability information of N consecutive frames to obtain personnel distribution probability information;

[0056] S4: Combine the optical flow intensity map with the person distribution probability map to obtain the tool information used in the video;

[0057] S5: Capture and output the tool information used in the video to form an enhanced training video dataset.

[0058] Furthermore, step S2 uses a global dense optical flow algorithm of image pyramid matching to calculate motion information; suppose an object with intensity I(x, y, t) has a constant intensity between two consecutive frames. When it moves a distance of dx and dy in one unit of time, it has a new intensity I(x+dx, y+dy, t+dt), that is, it satisfies the following formula: I(x, y, t) = I(x+dx, y+dy, t+dt)

[0059] x, y are the coordinates of the point, t is the current time; Taylor approximation is performed on the right side of the equation;

[0060] Image along time; the solution method is based on Gunnar Farneback's dense optical flow algorithm. The pixel intensity changes of all points between two frames are detected to obtain the optical flow information (u, v). In the specific algorithm of this patent, the scale relationship coefficient between the upper and lower layers of each pyramid layer pyr_scale = 0.5, the number of pyramid levels = 3, the mean window size winsize = 15, 3 iterations are performed for each image, the pixel neighborhood size poly_n = 5, the Gaussian standard deviation poly_sigma = 1.2, and the input flow is used as an approximation of the initial flow, and the number of consecutive frames N = 64.

[0061] Further, it has the following steps: S231: binarizing the optical flow intensity information map obtained in step S22, with a threshold of 128;

[0062] S232: Scan the binarized optical flow intensity information graph to obtain an optical flow intensity connected subgraph; P(i, j) represents the pixel value of the image at that point, starting from the image starting point P(0, 0), mark the connected subgraph, which includes the following specific steps: S233: Scan the image line by line, if P(r-1, c) = 0 and P(r, c) = 1, it is determined to be the starting point of the optical flow cluster, if P(r-1, c) = 1 and P(r, c) = 0, it is determined to be the end point of the optical flow cluster, and a unique identifier RI is set for it;

[0063] S234: For the optical flow clusters in all rows except the first row, if there is no overlap in r value between them and the existing clusters in the previous row, that is, the optical flow cluster P(r,c-1)≠P(r,c) holds everywhere, a new unique identifier RI is set for them;

[0064] S235: For the optical flow clusters in all rows except the first row, if there is only one region with r value overlapped with the previous row, then the clusters obtain the same identifier RI as the overlapping clusters in the previous row;

[0065] S236: For the optical flow clusters in all rows except the first row, if their r values ​​overlap with those of multiple regions in the previous row, the RI values ​​of the row area and all the overlapping clusters in the previous row are uniformly rewritten as the minimum RI value of these clusters, as a connected domain;

[0066] S237: summing up the optical flow intensities of the optical flow sub-graphs, and the sub-graphs with KRI ≥ 1000 are the areas to be judged using the tool.

[0067] Further, the coordinates of the tool judgment area selected in step S4 are normalized, the lower left corner of the video frame is taken as the (0,0) point, and the four poles of the tool judgment area, that is, the maximum and minimum values ​​of the x-axis and y-axis coordinates: Wmin, Wmax, Hmin and Hmax, are obtained; according to the input image ratio required for the subsequent neural network training using the data set, the center point (Cx, Cy) of the high-risk area is used as the center for length and width proportional expansion selection, and the selection box coordinates are expressed as (xmin, ymin, xmax, ymax), that is, the coordinates of the lower left corner and the upper right corner of the rectangular area;

[0068] When xmin≥Wmin and ymin≥Hmin and xmax≥Wmax and ymax≥Hmax, the selected area can be captured to form a new video clip.

[0069] Although the description of the present invention has been quite detailed and has been described in particular with respect to several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, so as to effectively cover the intended scope of the present invention. In addition, the present invention is described above with the embodiments foreseeable by the inventors, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that are not currently foreseen may still represent equivalent changes of the present invention.

Claims

1. A monitoring method for identifying human activity behavior, characterized in that: The method includes building a dependent environment for YOLO8 training of target detection of a deep learning neural network, creating a data set according to the environment, setting the data set configuration, using a pre-trained model, and performing model training on the training statements in the model.

2. A monitoring method for identifying human activity behavior according to claim 1, characterized in that: According to the monitoring device, the monitoring screen segments acquired from the environment are collected and the targets appearing in the human activity scene are taken as detection objects through the target detection image analysis and recognition technology.

3. A monitoring method for identifying human activity behavior according to claim 1, characterized in that: Creation of a data set, wherein the steps of creating the data set include preprocessing, screening and cleaning of the data, labeling of target categories, and dividing the data set into a training set, a test set and an evaluation set, thereby obtaining data training sets of different quantities; The dataset annotation format is converted into YOLO format data, which is expressed in the txt text format of annotating the extracted target object box bbox information. Each image corresponds to a txt file. Each line in the txt file contains the information of an object, including class, x_center, y_center, width, height format: the center is normalized as follows: 0 0.602168 0.601077 0.354276 0.395933.

4. A monitoring method for identifying human activity behavior according to claim 1, characterized in that: Use the optical flow method to perform optical flow detection on N consecutive frames of video images; The specific steps of using the optical flow method to detect the optical flow of video frames include: S21: remove the optical flow vector whose intensity is less than the optical flow amplitude threshold T1 in each frame image; S22: superimposing the optical flow information of N consecutive frames of images to obtain an optical flow intensity information map; S23: Using the graph connectivity algorithm, obtain the optical flow intensity connected subgraph of the image; S3: Use the pre-trained CSR-Net crowd counting network to extract the probability information of people distribution in N consecutive frames in the video; The specific steps of extracting the probability information of personnel distribution of N consecutive frames in a video include: S31: suppressing image regions in each frame where the distribution probability integral value is less than a preset threshold value T2; S32: superimposing probability information of N consecutive frames to obtain personnel distribution probability information; S4: Combine the optical flow intensity map with the person distribution probability map to obtain the tool information used in the video; S5: Capture and output the tool information used in the video to form an enhanced training video dataset.

5. A monitoring method for identifying human activity behavior according to claim 4, characterized in that: Step S2 uses a global dense optical flow algorithm of image pyramid matching to calculate motion information; suppose an object with intensity I(x, y, t) has a constant intensity between two consecutive frames. When it moves a distance of dx and dy in one unit of time, it has a new intensity I(x+dx, y+dy, t+dt), that is, it satisfies the following formula: I(x, y, t) = I(x+dx, y+dy, t+dt) x, y are the coordinates of the point, t is the current time; Taylor approximation is performed on the right side of the equation; Image along time; the solution method is based on Gunnar Farneback's dense optical flow algorithm. The pixel intensity changes of all points between two frames are detected to obtain the optical flow information (u, v). In the specific algorithm of this patent, the scale relationship coefficient between the upper and lower layers of each pyramid layer pyr_scale = 0.5, the number of pyramid levels = 3, the mean window size winsize = 15, 3 iterations are performed for each image, the pixel neighborhood size poly_n = 5, the Gaussian standard deviation poly_sigma = 1.2, and the input flow is used as an approximation of the initial flow, and the number of consecutive frames N = 64.

6. A monitoring method for identifying human activity behavior according to claim 4, characterized in that: The method comprises the following steps: S231: binarizing the optical flow intensity information map obtained in step S22, with a threshold value of 128; S232: Scan the binarized optical flow intensity information graph to obtain an optical flow intensity connected subgraph; P(i,j) represents the pixel value of the image at that point. Starting from the image starting point P(0,0), the connected subgraph is marked, which includes the following specific steps: S233: Scan the image line by line, if P(r-1,c)=0 and P(r,c)=1, it is determined as the starting point of the optical flow cluster, if P(r-1,c)=1 and P(r,c)=0, it is determined as the ending point of the optical flow cluster, and a unique identifier RI is set for it; S234: For the optical flow clusters in all rows except the first row, if there is no overlap in r value between them and the existing clusters in the previous row, that is, the optical flow cluster P(r,c-1)≠P(r,c) holds everywhere, a new unique identifier RI is set for them; S235: For the optical flow clusters in all rows except the first row, if there is only one region with r value overlapped with the previous row, then the clusters obtain the same identifier RI as the overlapping clusters in the previous row; S236: For the optical flow clusters in all rows except the first row, if their r values ​​overlap with those of multiple regions in the previous row, the RI values ​​of the row area and all the overlapping clusters in the previous row are uniformly rewritten as the minimum RI value of these clusters, as a connected domain; S237: summing up the optical flow intensities of the optical flow sub-graphs, and the sub-graphs with KRI ≥ 1000 are the areas to be judged using the tool.

7. A monitoring method for identifying human activity behavior according to claim 4, characterized in that: The coordinates of the tool judgment area selected in step S4 are normalized, the lower left corner of the video frame is taken as the point (0,0), and the four poles of the tool judgment area, that is, the maximum and minimum values ​​of the x-axis and y-axis coordinates: Wmin, Wmax, Hmin and Hmax, are obtained; according to the input image ratio required for the subsequent neural network training using the data set, the length and width are proportionally expanded and selected with the center point (Cx, Cy) of the high-risk area as the center, and the selection box coordinates are expressed as (xmin, ymin, xmax, ymax), that is, the coordinates of the lower left corner and the upper right corner of the rectangular area; When xmin≥Wmin and ymin≥Hmin and xmax≥Wmax and ymax≥Hmax, the selected area can be captured to form a new video clip.

Images