Security Risk Early Warning Method Based on AIoT and Personnel Trajectory Analysis

Abstract

The present invention provides a security risk early warning method based on AIoT and personnel trajectory analysis. The method includes: using a camera to obtain real-time personnel images in a production operation scenario, obtaining a first key point heat map of the personnel based on the personnel images, and using a trained key point inference network to process the first key point heat map to infer a second key point heat map of the personnel; the first key point is the center point of the head, and the second key point is the center point of the line connecting the two feet; obtaining personnel trajectory information according to multiple frames of the second key point heat maps; the production operation scenario includes a production operation area. When the number of personnel in the production operation area reaches a preset quantity threshold and it is detected based on the personnel trajectory information that someone is approaching the production operation area, a risk early warning is carried out. The present invention takes into account the oblique downward imaging characteristics of the camera and accurately locates the foot key points based on the head key points of the personnel, improving the accuracy of positioning and overcoming the occlusion problem.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and specifically to a security risk early warning method based on AIOT and personnel trajectory analysis. Background Art

[0002] Existing personnel positioning methods usually use bounding boxes or human key points. Such methods have serious positioning deviations in two-dimensional image processing and are difficult to accurately position. Especially in the scenario of factory safety production, it is extremely easy to have misidentifications when using existing methods for personnel positioning. For example, when positioning personnel based on the extracted human key points, the detected key points will be missing due to personnel occlusion, further resulting in difficulty in matching multi-frame key points, difficulty in analyzing trajectories, and inability to conduct effective early warnings. Summary of the Invention

[0003] To solve the above problems, the present invention proposes a security risk early warning method based on AIOT and personnel trajectory analysis. The method includes:

[0004] Using a camera to continuously obtain personnel images in the production operation scenario, obtaining a first key point heat map of the personnel based on the personnel images, and using a trained key point inference network to process the first key point heat map to infer a second key point heat map of the personnel; the first key point is the center point of the head, and the second key point is the center point of the line connecting the two feet;

[0005] Obtaining personnel trajectory information based on multiple frames of the second key point heat maps; the production operation scenario includes a production operation area. When the number of personnel in the production operation area reaches a preset number threshold and it is detected based on the personnel trajectory information that someone is approaching the production operation area, a risk early warning is issued;

[0006] Among them, the training process of the key point inference network includes:

[0007] Obtaining valid key point matching pairs in the head and foot key point heat maps of each frame of the image set. A valid key point matching pair includes a first key point and a second key point; among them, the head and foot key point heat map includes the first key point and the second key point;

[0008] The first key point heat maps corresponding to each frame of the head and foot key point heat maps obtained based on the valid key point matching pairs form a training image set, and the second key point heat maps form a label image set; the key point inference network is trained using the training image set and the label image set.

[0009] Furthermore, the angle between the direction from the first key point to the second key point in the valid key point matching pair and the reference direction of the valid matching pair is the smallest, where the reference direction of the valid matching pair is the direction from the first key point to the convergence point, and the convergence point is the intersection point of the extension directions of all personnel bodies in the personnel image.

[0010] Further, the method for obtaining the convergence point is as follows:

[0011] Perform rough matching of the first key point and the second key point in the head and foot key point heat map to obtain rough key point matching pairs;

[0012] Generate a hypothetical image along the lower edge direction of the person image. Connect the center point of the head and foot key point heat map and the center point of the hypothetical image to obtain a first straight line. Connect the first key point and the second key point in the rough key point matching pairs to obtain multiple second straight lines. Cluster the intersection points of the multiple second straight lines and the first straight line, and the clustering center point is the convergence point.

[0013] Further, the process of the rough matching is as follows:

[0014] Generate a first vector from the first key point in the head and foot key point heat map to the center point of the hypothetical image, and generate a second vector from the first key point in the head and foot key point heat map to the second key point. Determine the second key point matched with each first key point based on the included angle between the first vector and the second vector to obtain rough key point matching pairs.

[0015] Further, obtain the movement starting point and the movement ending point of all persons, and obtain the trajectory line of the person based on the inferred multi-frame second key point heat maps;

[0016] Take the movement starting point as the search starting point and the movement ending point as the search ending point, and search for continuous pixel points of the trajectory line of the person to obtain all possible movement trajectories, forming a trajectory set;

[0017] Input the trajectory set into the trajectory acquisition network to obtain the movement trajectory of each person.

[0018] Further, the losses for supervising the training process of the trajectory acquisition network include:

[0019] Perform forgetting superposition on the trajectory points of each movement trajectory output by the trajectory acquisition network according to the preset person movement speed to obtain the simulated heat value of the trajectory points. Obtain the heat value loss according to the simulated heat value of the trajectory points on each movement trajectory output by the trajectory acquisition network and the actual heat value of the trajectory points on the trajectory segment in the corresponding person trajectory line.

[0020] Further, the losses for supervising the training process of the trajectory acquisition network also include: Obtain the distance loss according to the trajectory length of each movement trajectory output by the trajectory acquisition network and the shortest connection line length between the movement starting point and the movement ending point of this movement trajectory.

[0021] Further, the loss for supervising the training process of the trajectory acquisition network further includes: for each output movement trajectory, obtaining the minimum position difference between the output movement trajectory and each movement trajectory in the trajectory set, and obtaining a position loss according to the minimum position differences corresponding to all the output movement trajectories.

[0022] The beneficial effects of the present invention are as follows:

[0023] 1. The present invention takes into account the characteristics of camera oblique top-down imaging and accurately locates the foot key points based on the head key points of the person, improving the accuracy of positioning and overcoming the occlusion problem.

[0024] 2. The present invention effectively separates complex trajectory lines by using a neural network, avoids misjudgment when trajectories overlap, and improves the accuracy of trajectory recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of the method implementation of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to enable those skilled in the art to better understand the present invention, the present invention will be described in detail below with reference to the embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0027] The specific application scenario targeted by the present invention is: a factory production operation scenario, which includes a production operation area and a non-production operation area. Among them, there is a limit on the number of production operation personnel in the production operation area.

[0028] Deploy image acquisition devices in the factory production operation scenario to collect RGB images of personnel. In order to cover the entire factory production operation scenario, multiple image acquisition devices of the same specification can be deployed according to the actual situation, but it is necessary to ensure that each image acquisition device has an oblique top-down view, and there is a certain overlapping part in the imaging range of each image acquisition device to facilitate image stitching operations.

[0029] Embodiment 1:

[0030] This embodiment provides a security risk warning method based on AIOT and personnel trajectory analysis. The implementation process of this method is as Figure 1 shown. Specifically, this method includes:

[0031] Training the key point speculation network:

[0032] 1) Obtain in real time the personnel images in the production operation scenario collected from the top-down perspective of multiple cameras, detect the first key point and the second key point in the multiple personnel images, and the multiple head-and-foot key point heat maps obtained form an image set; wherein, the first key point is the center point of the head, the second key point is the center point of the connection line of the two feet, and the head-and-foot key point heat map includes the first key point and the second key point.

[0033] Obtain the effective key point matching pairs in each frame of the head-and-foot key point heat map of the image set. Among them, one effective key point matching pair includes a first key point and a second key point; the angle between the direction from the first key point to the second key point in the effective key point matching pair and the reference direction of the effective matching pair is the smallest. The reference direction of the effective matching pair is the direction from the first key point to the convergence point, and the convergence point is the intersection point on the extension direction of all personnel bodies in the personnel image; specifically, the method for obtaining the convergence point is as follows:

[0034] a) Conduct a rough matching of the first key point and the second key point in the head-and-foot key point heat map to obtain rough key point matching pairs.

[0035] Generate a hypothetical image along the lower edge direction of the personnel image, that is, generate a hypothetical image below the personnel image; generate a first vector from the first key point in the head-and-foot key point heat map to the center point of the hypothetical image, and generate a second vector from the first key point in the head-and-foot key point heat map to the second key point. Determine the second key point that matches each first key point based on the angle between the first vector and the second vector to obtain rough key point matching pairs. Specifically, for each first key point, obtain a first vector corresponding to the first key point and multiple second vectors, calculate the cosine value of the angle between the first vector corresponding to the first key point and each second vector respectively, and when the cosine value is the largest, the second key point corresponding to the second vector forms a rough key point matching pair with the first key point.

[0036] When there is a second key point that forms rough key point matching pairs with multiple first key points respectively, screen the first key points based on the cosine value of the angle between the vectors; until each first key point in all rough key point matching pairs corresponds to a different second key point respectively, proceed to the subsequent steps.

[0037] b) Connect the center point of the head-and-foot key point heat map and the center point of the hypothetical image to obtain a first straight line, connect the first key point and the second key point in each key point matching pair to obtain multiple second straight lines, and cluster the intersection points of the multiple second straight lines and the first straight line. The clustering center point is the convergence point. Specifically, in the embodiment, the K-means clustering algorithm is adopted, and the clustering set is set to 1.

[0038] It should be noted that the obtained convergence point is mostly the ground projection point of the camera.

[0039] After obtaining the convergence point, the first key point in the head and foot key point heat map points to the convergence point to generate a third vector, and the second key point matching each first key point is determined based on the included angle between the second vector and the third vector, obtaining valid key point matching pairs. Preferably, when the cosine value of the included angle between the second vector and the third vector is greater than or equal to 0.8, the first key point and the second key point corresponding to the second vector form a valid key point matching pair.

[0040] 2) The first key point heat maps corresponding to each frame of head and foot key point heat maps obtained based on the valid key point matching pairs form a training image set, and the second key point heat maps form a label image set. Specifically, the first key points in the valid key point matching pairs of each frame of head and foot key point heat maps in the image set form the first key point heat map, and the second key points form the second key point heat map; the obtained multiple frames of first key point heat maps form a training image set, and the multiple frames of second key point heat maps form a label image set;

[0041] 3) Use the training image set and the label image set to train the key point inference network, and use the mean square error loss function during training.

[0042] It should be noted that the key point inference network is installed on the local end of the image acquisition device. Therefore, for image acquisition devices at different positions, the key point inference network needs to be trained separately to avoid insufficient generalization ability of the key point inference network.

[0043] Preferably, the key point inference network is a network structure of Encoder-Decoder.

[0044] Apply the key point inference network:

[0045] Obtain the first key point heat map of the personnel in the production operation scene in real time, and use the trained key point inference network to process the first key point heat map to infer and obtain the second key point heat map of the personnel.

[0046] The reason for the present invention to infer the second key point based on the first key point is as follows: The first key point is easy to detect and usually there is no occlusion, but the positioning is inaccurate; the second key point can well represent the two-dimensional information of the personnel relative to the ground. Compared with the bounding box or the center point of the human body, there will be no positioning deviation due to the height difference between the key points of the personnel and the ground, and the positioning is accurate but easy to be occluded; therefore, the present invention constructs a mapping relationship that can obtain the second key point according to the first key point to improve the positioning accuracy.

[0047] Since the ReID function cannot be implemented in this type of scenario, there will be overlapping trajectories. For areas with simple trajectories, trajectory analysis is performed based on heat values to extract complete trajectory lines. However, for areas with frequent personnel movements, it is difficult to directly separate the trajectories based on the heat value information of the trajectories. Therefore, the present invention proposes a method for obtaining personnel trajectory information based on multiple frames of second key point heat maps. Specifically:

[0048] Obtain the moving starting points and moving ending points of all personnel. The implementer can determine the moving starting points and moving ending points of personnel according to the change of heat values of points on the trajectory line; obtain the personnel trajectory line based on the multiple frames of second key point heat maps obtained by speculation; use the moving starting point as the search starting point and the moving ending point as the search ending point, and perform continuous pixel point search on the personnel trajectory line to obtain all possible moving trajectories, forming a trajectory set; input the trajectory set into the trajectory acquisition network to obtain the moving trajectory of each personnel.

[0049] Among them, the trajectory line of personnel can be obtained by superimposing the multiple frames of second key point heat maps obtained by speculation based on the superposition method of forgetting coefficients, or the trajectory line of personnel can be determined according to the IOU of the outer rectangles of the second key points in the multiple frames of second key point heat maps. In the embodiment, the method of superimposing the multiple frames of second key point heat maps obtained by speculation is used to obtain the trajectory line of personnel.

[0050] It should be noted that the lengths of all moving trajectories in the trajectory set should be the same. Specifically, the purpose of making the lengths of each moving trajectory the same can be achieved by filling values.

[0051] Preferably, the losses for supervising the training process of the trajectory acquisition network include:

[0052] Performing forgetting superposition on the trajectory points of each moving trajectory output by the trajectory acquisition network according to the preset personnel moving speed to obtain the simulated heat value of the trajectory points on each output moving trajectory, and obtaining the heat value loss according to the simulated heat value of the trajectory points on each output moving trajectory and the actual heat value of the trajectory points on the trajectory segments in the corresponding personnel trajectory line. Specifically, the mean square error between the simulated heat value of the trajectory points on each output moving trajectory and the actual heat value of the trajectory points on the trajectory segments in the corresponding personnel trajectory line is used to obtain the heat value loss Loss H ;

[0053] Obtaining the distance loss according to the trajectory length of each moving trajectory output by the trajectory acquisition network and the shortest connection line length between the moving starting point and the moving ending point of this moving trajectory. Specifically, the difference between the trajectory length of each moving trajectory output by the trajectory acquisition network and the shortest connection line length between the moving starting point and the moving ending point of this moving trajectory is the distance loss Loss D ;

[0054] For each moving trajectory output by the trajectory acquisition network, obtain the minimum position difference between the output moving trajectory and each moving trajectory in the trajectory set, and obtain the position loss according to the minimum position differences corresponding to all output moving trajectories; specifically, calculate the mean square error between the coordinates of the trajectory points on the output moving trajectory and the coordinates of the trajectory points on each moving trajectory in the trajectory set, and the minimum value of the mean square error represents the minimum position difference corresponding to the output moving trajectory, and the sum of the minimum position differences corresponding to all output moving trajectories is the position loss Loss R 。

[0055] Preferably, the loss function used when training the trajectory acquisition network in the embodiment is Loss = Loss H +Loss D +Loss R 。

[0056] The production operation scenario includes a production operation area. When the number of people in the production operation area reaches a preset number threshold and someone approaches the production operation area, a risk warning is issued; specifically, when the number of people in the production operation area reaches the preset number threshold and it is determined according to the moving trajectories of the people that someone is walking towards the production operation area and the distance from the production operation area is less than the preset distance threshold, an infrared sensor is called for verification. When the sensor also determines that the distance is less than the preset distance threshold, it is confirmed that there is a safety risk, and a risk warning message is generated. The form of the risk warning message is not limited, and the implementer can choose warning forms such as text message prompts, alarm prompts or light prompts.

[0057] The above is intended to enable those skilled in the art to better understand the present invention and is not intended to limit the present invention. Any modifications and changes made within the spirit and principles of the present invention are within the protection scope of the present invention.

Claims

1. A security risk early warning method based on AIOT and personnel trajectory analysis, characterized in that, The method includes: Use a camera to obtain a person image in a production operation scene in real time, obtain a first key point heat map of the person based on the person image, and use a trained key point inference network to process the first key point heat map to infer a second key point heat map of the person; the first key point is the center point of the head, and the second key point is the center point of the line connecting the two feet; Obtaining personnel trajectory information according to a multi-frame second key point heat map; obtaining the moving starting point and moving ending point of all personnel, and obtaining the trajectory line of the personnel based on the inferred multi-frame second key point heat map; taking the moving starting point as the search starting point and the moving ending point as the search ending point, performing a continuous pixel search on the trajectory line of the personnel to obtain all possible movement trajectories to form a trajectory set; inputting the trajectory set into a trajectory acquisition network to obtain the movement trajectory of each person; the production operation scene includes a production operation area, and when the number of personnel in the production operation area reaches a preset number threshold, a risk warning is issued when it is detected that someone is approaching the production operation area based on the personnel trajectory information; Among them, the training process of the key point inference network includes: Obtain valid key point matching pairs in the head and foot key point heat map of each frame of the image set, wherein a valid key point matching pair includes a first key point and a second key point; wherein the head and foot key point heat map includes the first key point and the second key point; The first key point heat map corresponding to the head and foot key point heat map of each frame obtained based on the effective key point matching pairs constitutes a training image set, and the second key point heat map constitutes a label image set; the key point inference network is trained using the training image set and the label image set.

2. The method according to claim 1, wherein The angle between the direction from the first key point to the second key point in the effective key point matching pair and the reference direction of the effective matching pair is the smallest, wherein the reference direction of the effective matching pair is the direction from the first key point to the convergence point, and the convergence point is the intersection of the extension directions of the bodies of all persons in the person image.

3. The method according to claim 2, characterized in that, The method for obtaining the convergence point is: Perform a rough match between the first key point and the second key point in the head and foot key point heat map to obtain a rough key point matching pair; Generate an imaginary image along the lower edge of the person image, connect the center point of the head and foot key point heat map and the center point of the imaginary image to obtain the first straight line, connect the first key point and the second key point in the rough key point matching pair to obtain multiple second straight lines, cluster the intersections of the multiple second straight lines with the first straight line, and the cluster center point is the convergence point.

4. The method according to claim 3, wherein The process of the rough matching is as follows: The first key point in the head and foot key point heat map points to the center point of the imaginary image to generate a first vector, and the first key point in the head and foot key point heat map points to the second key point to generate a second vector. The second key point matching each first key point is determined based on the angle between the first vector and the second vector to obtain a rough key point matching pair.

5. The method according to claim 4, characterized in that, The losses that supervise the trajectory acquisition network training process include: For each trajectory point of each moving trajectory output by the trajectory acquisition network, perform forgetting superposition according to the preset personnel movement speed to obtain the simulated heat value of the trajectory point, and obtain the heat value loss based on the simulated heat value of the trajectory point on each moving trajectory output by the trajectory acquisition network and the actual heat value of the trajectory point on the trajectory segment in the corresponding personnel trajectory line.

6. The method according to claim 5, characterized in that, The loss for supervising the training process of the trajectory acquisition network further includes: obtaining a distance loss according to the trajectory length of each moving trajectory output by the trajectory acquisition network and the length of the shortest connection line between the moving starting point and the moving ending point of the moving trajectory.

7. The method according to claim 6, wherein The loss for supervising the training process of the trajectory acquisition network further includes: for each output moving trajectory, obtaining the minimum position difference between the output moving trajectory and each moving trajectory in the trajectory set, and obtaining a position loss based on the minimum position differences corresponding to all output moving trajectories.

Images