A LSTM-based edge protection warning method, device and medium

Abstract

The present application discloses an LSTM-based edge protection warning method, device and medium, which relates to the field of edge protection technology. The method includes: obtaining a video stream of a target person through a camera array, and obtaining three-dimensional point cloud data of the target person through a laser radar; performing background modeling on the video stream, and fusing the three-dimensional point cloud data to generate a motion trajectory with three-dimensional coordinates; receiving the motion trajectory, and predicting the motion path and speed within a preset time period through a stacked bidirectional LSTM network; integrating an audible and visual alarm, an intelligent terminal push and a mechanical braking device, and triggering a graded alarm based on the predicted path and speed, as well as the spatial relationship with the preset dangerous area. The present application uses the above method to achieve comprehensive perception, intelligent prediction and graded alarm of target personnel at the construction site by predicting the motion path and speed within a preset time period through a stacked bidirectional LSTM network.

Description

Technical Field

[0001] The present application relates to the field of edge protection technology, and in particular to an LSTM-based edge protection early warning method, device and medium. Background Art

[0002] In high-risk environments such as construction sites and factories, especially in high-altitude or edge working environments, edge protection is an important measure to ensure the safety of workers. Traditional edge protection methods mainly rely on physical guardrails and manual monitoring, but these methods have certain limitations, such as physical guardrails cannot be adjusted dynamically, and manual monitoring is easily affected by fatigue and distraction.

[0003] Although fences and safety belts are effective, traditional fences cannot dynamically adapt to changes in construction scenarios and lack the ability to monitor and warn workers' dynamic behaviors in real time. Existing warning systems usually use a single alarm method, such as sound and light alarms, and lack the ability to distinguish between different risk levels and corresponding response measures.

[0004] Existing edge protection systems usually rely on a single data source, such as video surveillance or lidar, which makes it difficult to fully capture the movement status and location information of the target personnel. The video surveillance system has a high false alarm rate and cannot distinguish between normal operations and dangerous behaviors. Single sensors are susceptible to environmental interference and lack multimodal data fusion capabilities. The early warning response is delayed, and existing algorithms cannot accurately predict the risk of sudden falls.

[0005] Through the above analysis, the problems and defects of the prior art are as follows:

[0006] The edge protection system in the existing technology collects single data, cannot achieve high-precision motion path and speed prediction, and lacks intelligent early warning. Summary of the invention

[0007] The embodiments of the present application provide an LSTM-based edge protection warning method, device and medium, which can solve the problems in the prior art that the edge protection system collects single data, cannot achieve high-precision motion path and speed prediction, and lacks intelligent warning.

[0008] In the first aspect, an embodiment of the present application provides an LSTM-based edge protection warning method, the method comprising: obtaining a video stream of a target person through a camera array, and obtaining three-dimensional point cloud data of the target person through a lidar; performing background modeling on the video stream, and fusing the three-dimensional point cloud data to generate a motion trajectory with three-dimensional coordinates; receiving the motion trajectory, and predicting the motion path and speed within a preset time period through a stacked bidirectional LSTM network; integrating an audible and visual alarm, a smart terminal push and a mechanical braking device, and triggering a graded alarm according to the predicted path and speed, as well as the spatial relationship with a preset danger zone.

[0009] In one implementation of the present application, background modeling is performed on the video stream, three-dimensional point cloud data is integrated, and a motion trajectory of a person with three-dimensional coordinates is constructed. The specific method includes: background modeling is performed on the video stream, and the motion contour of the target person is extracted based on the YOLOv algorithm; the three-dimensional point cloud data and the motion contour are aligned in time and space with the DeepSORT algorithm to obtain a motion trajectory with three-dimensional coordinates; based on the motion trajectory, the velocity vector and posture key point information of the target person are calculated; and the three-dimensional coordinates, velocity vector and posture key point information are integrated to generate a six-dimensional space-time feature vector.

[0010] In one implementation of the present application, the method also includes using the OpenPose framework to extract the three-dimensional coordinates of 17 key points of the human body in the motion trajectory; marking the coordinates of the safety warning line in the three-dimensional point cloud data, and calculating the kinematic parameters of the three-dimensional coordinates of the key points and the adjacent warning line, the kinematic parameters including the Euclidean distance and acceleration; calculating the acceleration variance of the motion trajectory of the key points of the hand to identify the waving action and trigger the distress sign; calculating the pitch angle of the key points of the head, and when the pitch angle is greater than a preset angle and exceeds a preset time, triggering the attention distraction sign.

[0011] In one implementation of the present application, motion trajectory data is received, and the motion path and motion speed are predicted through a stacked bidirectional LSTM network, specifically including: the stacked bidirectional LSTM network includes at least two LSTM layers, each LSTM layer is configured with a preset number of memory units, and a six-dimensional spatiotemporal feature vector is obtained through a sliding time window of the stacked bidirectional LSTM network; the features of each time step are weighted through an attention mechanism to predict the trajectory coordinates and speed probability distribution.

[0012] In one implementation of the present application, a graded alarm is triggered based on the predicted path and movement speed, as well as the spatial relationship with the preset danger zone, specifically including: calculating the similarity of the trajectory coordinates and speed probability distribution based on the current operation of the target personnel and the same operation in the database to obtain a first risk score; calculating the second risk score based on the trajectory coordinates and speed probability distribution; when the Euclidean distance is less than a preset threshold, marking it as a red high-risk level; when the Euclidean distance is greater than the preset threshold and the acceleration is greater than the acceleration threshold, marking it as an orange medium-risk level.

[0013] In one implementation of the present application, an sound and light alarm, smart terminal push and mechanical braking device are integrated to trigger graded alarms based on the spatial relationship between the predicted path and the preset danger zone, specifically including: when the first risk score does not reach the safety threshold, triggering the virtual warning light curtain; when the second risk score reaches the risk threshold, activating the electromagnetic lock of the guardrail; when it is a red high-risk level, starting the sound and light alarm; in the case of an orange medium-risk level, triggering vibration; when the second risk score reaches the risk threshold or the red high-risk level, synchronously adjusting the drone inspection path to the air above the warning area, and pushing the emergency plan to the personnel terminal.

[0014] In one implementation of the present application, the method also includes: constructing a digital twin construction site scene based on the BIM model, and establishing a two-way mapping channel with real-time monitoring data; in the digital twin construction site scene, using a reinforcement learning algorithm to iteratively optimize the camera array layout plan and warning threshold.

[0015] In one implementation of the present application, before the integrated sound and light alarm, intelligent terminal push and mechanical brake device trigger the graded alarm according to the spatial relationship between the predicted path and the preset danger zone, the method also includes: integrating a micro projector into the helmet to superimpose a virtual warning line on the user's field of view in real time; a built-in multi-axis sensor detects the motion state, and triggers a tactile warning when the acceleration exceeds a preset threshold.

[0016] In the second aspect, the embodiment of the present application also provides an LSTM-based edge protection warning device, the device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can: obtain a video stream of a target person through a camera array, and obtain three-dimensional point cloud data of the target person through a lidar; perform background modeling on the video stream, and fuse the three-dimensional point cloud data to generate a motion trajectory with three-dimensional coordinates; receive the motion trajectory, and predict the motion path and speed within a preset time period through a stacked bidirectional LSTM network; integrate sound and light alarms, smart terminal push and mechanical braking devices, and trigger graded alarms according to the predicted path and speed, as well as the spatial relationship with the preset danger zone.

[0017] On the third aspect, the embodiment of the present application also provides a non-volatile computer storage medium for edge protection warning based on LSTM, which stores computer executable instructions, and the computer executable instructions are set to: obtain a video stream of the target person through a camera array, and obtain three-dimensional point cloud data of the target person through a lidar; perform background modeling on the video stream, and fuse the three-dimensional point cloud data to generate a motion trajectory with three-dimensional coordinates; receive the motion trajectory, and predict the motion path and speed within a preset time period through a stacked bidirectional LSTM network; integrate sound and light alarms, smart terminal push and mechanical braking devices, and trigger graded alarms according to the predicted path and speed, as well as the spatial relationship with the preset danger zone.

[0018] The embodiments of the present application provide an LSTM-based edge protection warning method, device and medium, which integrate video streams and three-dimensional point cloud data to generate a motion trajectory with three-dimensional coordinates, thereby achieving comprehensive perception and precise tracking of target personnel, receiving motion trajectory data, and predicting the motion path and speed within a preset time period through a stacked bidirectional LSTM network. By using the powerful processing capability of deep learning, intelligent prediction of the future motion state of the target personnel is achieved; the graded alarm system can take corresponding warning measures according to different risk levels, which not only avoids the interference caused by excessive alarms, but also ensures timely response in high-risk situations; comprehensive perception, intelligent prediction and graded alarms of target personnel at the construction site are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 A flowchart of an LSTM-based edge protection warning method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the internal structure of an LSTM-based edge protection warning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0023] The embodiments of the present application provide an LSTM-based edge protection warning method, device and medium, which solve the problems in the prior art that the edge protection system collects single data, cannot achieve high-precision motion path and speed prediction, and lacks intelligent warning.

[0024] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0025] Figure 1 A flow chart of an LSTM-based edge protection warning method provided in an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides an LSTM-based edge protection warning method, which specifically includes the following steps:

[0026] Step 10: Obtain the video stream of the target person through the camera array, and obtain the three-dimensional point cloud data of the target person through the lidar.

[0027] In this step, at least three high-definition cameras with a resolution of 1080P are deployed in the edge area of ​​the construction site to form a network with no blind spots. The frame rate of each camera is not less than 30fps, and the wide dynamic range (WDR) technology is enabled. Two lidars are installed on the top of the edge guardrail with an inclination angle of 15° to 30°, a horizontal scanning angle of 120°, a vertical scanning angle of 40°, and a scanning frequency of ≥20Hz to generate real-time three-dimensional point cloud data. The vertical detection accuracy of the edge area is enhanced by tilting the LiDAR installation angle.

[0028] It is understandable that a light intensity sensor can be integrated into the camera array to monitor the illuminance value in real time. For example, when the illuminance value is lower than <50lux, the infrared fill light is started, and the mode is switched to multi-spectral imaging to perform Retinex image enhancement processing and dynamically adjust the density threshold of the three-dimensional point cloud data, for example, increase it to 200 points / ㎡ in a low-light environment.

[0029] Step 20: Perform background modeling on the video stream and fuse the 3D point cloud data to generate a motion trajectory with 3D coordinates.

[0030] As an optional embodiment, background modeling is performed on the video stream, three-dimensional point cloud data is integrated, and a motion trajectory of a person with three-dimensional coordinates is constructed. The specific method may include: step 201: background modeling is performed on the video stream, and a motion contour of a target person is extracted based on a YOLOv algorithm;

[0031] In this step, an improved ViBe (Visual Background Extractor) background modeling algorithm is used for the video stream, the update rate is adaptively adjusted, and the Gaussian Mixture Model (GMM) can effectively separate the foreground target from the background. YOLOv5 is used to detect workers and associate ReID (Re-identification) features to build cross-camera trajectories. For example, the 3D point cloud data point cloud can be voxelized into 5cm³ grids to calculate the point density in each grid.

[0032] Step 202: Using the DeepSORT algorithm, the three-dimensional point cloud data is spatially and temporally aligned with the motion contour to obtain a motion trajectory with three-dimensional coordinates.

[0033] In this step, the 2D trajectory coordinates and 3D point cloud density can be fused to generate a spatiotemporal probability distribution map. The best matching pair is found by calculating the similarity between the 3D point cloud data and the 2D motion profile. The Kalman filter technology in the DeepSORT (DeepSimple Online and Realtime Tracking, a multi-target tracking algorithm based on deep learning) algorithm is used to predict the position of the target person in the next frame, which helps to establish stable tracking between consecutive frames.

[0034] Step 203: Based on the motion trajectory, the velocity vector and posture key point information of the target person are calculated.

[0035] In this step, continuous time points are selected from the motion trajectory. These time points correspond to the position of the target person in the three-dimensional space. For each pair of adjacent time points, the displacement vector of the target person is calculated, that is, the three-dimensional space distance and direction between the two position points. The velocity vector of the target person in the time period is obtained by dividing the displacement vector by the time interval. The velocity vector contains the magnitude (i.e., rate) and direction information of the velocity. 17 key points are extracted. For example, the confidence threshold can be set to 0.7, and the graph convolutional network (GCN) is enabled for completion of low-confidence key points (such as ankles). After the three-dimensional posture is reconstructed, the three-dimensional coordinates of each key point on the target person's body and their relative positional relationship, such as joint angles, limb lengths, etc. can be obtained.

[0036] Step 204: Fuse the three-dimensional coordinates, velocity vector and posture key point information to generate a six-dimensional spatiotemporal feature vector.

[0037] As an optional embodiment, the method may further include: step 205: using the OpenPose framework to extract the three-dimensional coordinates of 17 key points of the human body in the motion trajectory; step 206: marking the coordinates of the safety warning line in the three-dimensional point cloud data, and calculating the kinematic parameters of the three-dimensional coordinates of the key points and the adjacent warning line, the kinematic parameters including the Euclidean distance and acceleration; step 207: calculating the acceleration variance of the motion trajectory of the key points of the hand to identify the waving action and trigger the distress sign; step 208: calculating the pitch angle of the key points of the head, and when the pitch angle is greater than a preset angle and exceeds a preset time, triggering the attention distraction sign.

[0038] In this step, key points usually include the head, shoulders, elbows, wrists, hips, knees and ankles, etc. These key points constitute the skeleton structure of the human body. The safety warning lines of the construction site are pre-marked in the three-dimensional point cloud data, such as the position of the edge guardrail, and the shortest Euclidean distance from the key points of the human body (such as feet and hands) to the warning line is calculated to evaluate whether the person is close to the dangerous area; the acceleration of the key points relative to the warning line is calculated by the position change of the key points in consecutive frames to detect whether the person is approaching or moving away from the warning line quickly, and the speed variance of the torso key points is calculated. The acceleration variance of the hand key points in consecutive frames is calculated. , a large variance indicates violent hand movement, which may be a waving action; when the acceleration variance exceeds the preset threshold, it is identified as a waving action, which may mean that the person needs help or issues an alarm; once a waving action is identified, a distress sign is triggered to notify relevant personnel or the system to take corresponding measures. For example, when the variance exceeds 2m² / s², it is marked as an unbalanced state; the three-dimensional coordinates of the key points of the head are used to calculate the pitch angle of the head relative to the horizontal plane; when the pitch angle is greater than the preset angle (such as the head is excessively lowered or raised), it may indicate that the person is distracted. If this state lasts for more than the preset time, the distraction sign is triggered.

[0039] Step 30: Receive the motion trajectory and predict the motion path and speed within a preset time period through a stacked bidirectional LSTM (Long Short-Term Memory) network.

[0040] As an optional embodiment, receiving motion trajectory data and predicting the motion path and motion speed through a stacked bidirectional LSTM network can specifically include: step 301: the stacked bidirectional LSTM network includes at least two LSTM layers, each LSTM layer is configured with a preset number of memory units, and a six-dimensional spatiotemporal feature vector is obtained through a sliding time window of the stacked bidirectional LSTM network; step 302: weighting the features of each time step through an attention mechanism to predict the trajectory coordinates and speed probability distribution.

[0041] For example, the stacked bidirectional LSTM network contains two LSTM layers, configured with 256 and 128 memory cells respectively. The input features include a six-dimensional vector [X, Y, Z, ΔX, ΔY, ΔZ], and the time window length is 20 frames. The output layer weights the features of each time step through the attention mechanism to predict the trajectory coordinates and speed probability distribution in the next 3 seconds. The first layer of bidirectional LSTM has 128 memory cells in each direction and outputs 256-dimensional features (bidirectional splicing), extracts local time features, and captures short-term motion trends. The second layer of bidirectional LSTM has 64 memory cells in each direction and outputs 128-dimensional features to capture global time dependencies and understand long-term motion patterns. The fully connected layer maps the LSTM output to a 6-dimensional vector (3 position coordinates + 3 speed components). The output layer regresses the task and directly outputs the predicted path and speed. Predicted path: the target's position coordinates in the next 3 seconds [X{t+1}, Y{t+1}, Z{t+1}], predicted speed: the target's speed components in the next 3 seconds [\Delta X_{t+1},\DeltaY_{t+1},\DeltaZ_{t+1}]. The output layer can be designed as a classifier or a regressor. The classifier is used to determine whether a dangerous behavior will occur (such as 0-no risk, 1-risk), while the regressor predicts specific position coordinates or other continuous variables. The predicted probability distribution indicates the possibility that the target person may appear in different locations and have different speeds within the preset time period. By selecting the coordinates and speed values ​​with the highest probability, the predicted movement path and movement speed of the target person can be obtained.

[0042] Step 40: Integrate sound and light alarms, intelligent terminal push and mechanical brake devices to trigger graded alarms based on the predicted path and movement speed, as well as the spatial relationship with the preset danger zone.

[0043] As an optional embodiment, triggering a graded alarm based on the predicted path and movement speed, as well as the spatial relationship with the preset danger zone may specifically include: Step 401: calculating the similarity of the trajectory coordinates and speed probability distribution with the same operation in the database based on the current operation of the target person to obtain a first risk score; Step 402: calculating the second risk score based on the trajectory coordinates and speed probability distribution; Step 403: when the Euclidean distance is less than a preset threshold, marking it as a red high-risk level; Step 404: when the Euclidean distance is greater than the preset threshold and the acceleration is greater than the acceleration threshold, marking it as an orange medium-risk level.

[0044] In this step, for example, when the distance D between the predicted position and the warning line is ≤0.5m, it is marked as a red high-risk level; if 0.5m<D≤1.5m and the speed v≥1.2m / s, it is marked as an orange medium-risk level; the Hausdorff distance D between the predicted position and the edge is calculated; the ratio R / v of the curvature radius R of the motion trajectory and the speed v is analyzed; when D<1m and R / v<0.5, it is determined to be a high risk of falling; when 1m≤D<2m and R / v<1, it is determined to be a potential risk.

[0045] As an optional embodiment, an audible and visual alarm, intelligent terminal push and mechanical braking device are integrated to trigger a graded alarm according to the spatial relationship between the predicted path and the preset danger zone, specifically including: Step 405: When the first risk score does not reach the safety threshold, the virtual warning light curtain is triggered; Step 406: When the second risk score reaches the risk threshold, the electromagnetic lock of the guardrail is activated; Step 407: When the red high-risk level is reached, the audible and visual alarm is started; Step 408: When the orange medium-risk level is reached, vibration is triggered; Step 409: When the second risk score reaches the risk threshold or the red high-risk level, the drone inspection path is synchronously adjusted to the air above the warning area, and the emergency plan is pushed to the personnel terminal.

[0046] In this step, when the speed at which workers approach the edge area exceeds a certain threshold, it should be considered high risk even if the distance is still far, and the sound and light alarm (120dB beep + red flash) should be activated for high-risk targets.

[0047] As an optional embodiment, the method may also include: constructing a digital twin construction site scene based on the BIM model, and establishing a two-way mapping channel with real-time monitoring data; in the digital twin construction site scene, using a reinforcement learning algorithm to iteratively optimize the camera array layout plan and warning threshold.

[0048] In this step, real trajectory data is imported to train the virtual risk prediction model, extreme working conditions are simulated in a virtual environment for stress testing, and the optimal camera layout plan and warning threshold recommendations are output.

[0049] As an optional embodiment, before integrating the sound and light alarm, smart terminal push and mechanical braking device, and triggering a graded alarm according to the spatial relationship between the predicted path and the preset danger zone, the method also includes: integrating a micro projector into the safety helmet to superimpose a virtual warning line on the user's field of view in real time; a built-in multi-axis sensor detects the motion state, and triggers a tactile warning when the acceleration exceeds a preset threshold.

[0050] In this step, the existing display screens on site can be used to display warning information. For workers working close to the edge, warning icons can be displayed directly in their field of vision through wearable devices such as smart glasses. For example, workers can wear vibrating bracelets to gradually increase the intensity of tactile feedback as the risk level increases. Safety belts can have integrated tactile arrays to encode vibration patterns to indicate the direction of risk. Smart safety helmets can display virtual guardrails superimposed with AR. When the hands continue to approach the dangerous area, the gloves generate reverse electrical stimulation. The built-in barometer detects sudden changes in altitude and triggers the deployment of the emergency airbag.

[0051] In addition, considering that each worker may have different working habits and personal preferences, personalized warning setting options are provided. For example, workers are allowed to choose their preferred warning method: sound, vibration, flashing lights, etc., and set different sensitivity thresholds; regularly collect feedback from workers on the warning system, continuously optimize the user experience, and ensure that warning information is neither ignored due to excessive frequency, nor delayed due to being too conservative.

[0052] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides an LSTM-based edge protection warning device, whose structure is as follows Figure 2 shown.

[0053] Figure 2 The internal structure diagram of an LSTM-based edge protection warning device provided in the embodiment of the present application. Figure 2 As shown, the device includes:

[0054] at least one processor 201;

[0055] and, a memory 202 communicatively connected to the at least one processor;

[0056] Among them, the memory 202 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 201, so that at least one processor 201 can: obtain a video stream of a target person through a camera array, and obtain three-dimensional point cloud data of the target person through a lidar; perform background modeling on the video stream, and fuse the three-dimensional point cloud data to generate a motion trajectory with three-dimensional coordinates; receive the motion trajectory, and predict the motion path and motion speed within a preset time period through a stacked bidirectional LSTM network; integrate an audible and visual alarm, an intelligent terminal push and a mechanical braking device, and trigger a graded alarm according to the predicted path and motion speed, as well as the spatial relationship with the preset danger zone.

[0057] Some embodiments of the present application provide corresponding Figure 1A non-volatile computer storage medium for edge protection warning based on LSTM stores computer executable instructions, wherein the computer executable instructions are configured to: obtain a video stream of a target person through a camera array, and obtain three-dimensional point cloud data of the target person through a laser radar; perform background modeling on the video stream, and fuse the three-dimensional point cloud data to generate a motion trajectory with three-dimensional coordinates; receive the motion trajectory, and predict the motion path and motion speed within a preset time period through a stacked bidirectional LSTM network; integrate an audible and visual alarm, an intelligent terminal push and a mechanical brake device, and trigger a graded alarm according to the predicted path and motion speed, as well as the spatial relationship with a preset dangerous area.

[0058] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0059] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0060] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0061] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0064] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0065] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0066] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0067] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0068] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A LSTM-based edge protection warning method, characterized in that: The method comprises: The video stream of the target person is obtained through the camera array, and the three-dimensional point cloud data of the target person is obtained through the laser radar; The video stream is subjected to background modeling, and the three-dimensional point cloud data is fused to generate a motion trajectory with three-dimensional coordinates. The specific method includes: Perform background modeling on the video stream and extract the motion contour of the target person based on the YOLOv algorithm; In combination with the DeepSORT algorithm, the three-dimensional point cloud data is spatially and temporally aligned with the motion profile to obtain a motion trajectory with three-dimensional coordinates; Based on the motion trajectory, calculating the velocity vector and posture key point information of the target person; Fusion of the three-dimensional coordinates, velocity vector and attitude key point information to generate a six-dimensional spatiotemporal feature vector; The OpenPose framework is used to extract the three-dimensional coordinates of 17 key points of the human body in the motion trajectory; Marking the coordinates of the safety warning line in the three-dimensional point cloud data, and calculating the kinematic parameters of the three-dimensional coordinates of the key points and the edge warning line, wherein the kinematic parameters include Euclidean distance and acceleration; Calculate the acceleration variance of the hand key point motion trajectory to identify the waving action and trigger the distress sign; Calculating the pitch angle of the key points of the head, and triggering the attention distraction flag when the pitch angle is greater than a preset angle and exceeds a preset time; Receiving the motion trajectory, and predicting the motion path and motion speed within a preset time period through a stacked bidirectional LSTM network; An integrated sound and light alarm, intelligent terminal push and mechanical braking device triggers a graded alarm according to the predicted path and movement speed, as well as the spatial relationship with the preset danger zone.

2. According to the LSTM-based edge protection early warning method of claim 1, it is characterized in that: Receiving the motion trajectory data, and predicting the motion path and motion speed through a stacked bidirectional LSTM network, specifically including: The stacked bidirectional LSTM network comprises at least two LSTM layers, each LSTM layer is configured with a preset number of memory units, and the six-dimensional spatiotemporal feature vector is obtained through a sliding time window of the stacked bidirectional LSTM network; The features of each time step are weighted through the attention mechanism to predict the trajectory coordinates and velocity probability distribution.

3. The LSTM-based edge protection early warning method according to claim 2 is characterized in that: According to the predicted path and movement speed, as well as the spatial relationship with the preset danger zone, a graded alarm is triggered, specifically including: Calculate the similarity of the trajectory coordinates and the speed probability distribution according to the current operation of the target person and the same operation in the database to obtain a first risk score; Calculating a second risk score based on the trajectory coordinates and the speed probability distribution; When the Euclidean distance is less than a preset threshold, it is marked as a red high-risk level; When the Euclidean distance is greater than a preset threshold and the acceleration is greater than the acceleration threshold, it is marked as an orange medium-risk level.

4. The LSTM-based edge protection early warning method according to claim 3 is characterized in that: Integrated sound and light alarm, intelligent terminal push and mechanical brake device, trigger graded alarm according to the spatial relationship between the predicted path and the preset danger zone, including: When the first risk score does not reach the safety threshold, triggering the virtual warning light curtain; When the second risk score reaches a risk threshold, activating the electromagnetic lock of the guardrail; At the red high-risk level, the sound and light alarm is activated; In the case of the orange medium-risk level, a vibration is triggered; When the second risk score reaches the risk threshold or the red high-risk level, the drone inspection path is synchronously adjusted to the airspace above the warning area, and the emergency plan is pushed to the personnel terminal.

5. The LSTM-based edge protection early warning method according to claim 1 is characterized in that: The method further comprises: Build a digital twin construction site scene based on the BIM model and establish a two-way mapping channel with real-time monitoring data; In the digital twin construction site scenario, a reinforcement learning algorithm is used to iteratively optimize the camera array layout plan and warning threshold.

6. The LSTM-based edge protection early warning method according to claim 4 is characterized in that: Before the integrated sound and light alarm, intelligent terminal push and mechanical brake device trigger the graded alarm according to the spatial relationship between the predicted path and the preset danger zone, the method further includes: A micro-projector is integrated into the helmet to superimpose a virtual warning line on the user's field of view in real time; The built-in multi-axis sensor detects motion status and triggers a tactile warning when the acceleration exceeds the preset threshold.

7. An LSTM-based edge protection warning device, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: Execute the steps of an LSTM-based edge protection warning method as described in any one of claims 1-6.

8. A non-volatile computer storage medium for edge protection warning based on LSTM, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Execute the steps of an LSTM-based edge protection warning method as described in any one of claims 1-6.

Images