Visual terminal behavior identification and safety prediction method and system

Abstract

The invention discloses a visual terminal behavior recognition and safety prediction method and system, which can detect and recognize abnormal behaviors in time through real-time monitoring and analysis of video data, and further predict an impending safety threat, so that measures can be taken in advance, thereby effectively preventing the occurrence of an accident and improving the safety of a user. The security of a public area or a specific monitoring scene is greatly improved, the video stream is processed through the edge computing node, the demand for transmitting a large amount of original video data to the central server is reduced, the network bandwidth pressure is reduced, the data processing speed is accelerated, and effective utilization of resources is realized.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance, and in particular to a method and system for visual terminal behavior recognition and safety prediction. Background Art

[0002] Research on pedestrian recognition technology began in the mid-1990s. With the rise of deep learning, detection accuracy and speed have significantly improved. Pedestrian recognition encompasses both pedestrian detection and person re-identification, a process that leverages computer vision techniques to automatically identify pedestrians from images or video sequences. Pedestrian detection uses machine learning or deep learning algorithms to analyze pixels in an image, extracting human features such as shape, size, and color, and then determining whether a pedestrian exists. Person re-identification, a subproblem of image retrieval, aims to address the visual limitations of fixed cameras and is widely used in intelligent video surveillance and security. Motion behavior prediction leverages observed historical trajectories and environmental information to predict a pedestrian's future location. Motion behavior prediction is crucial for autonomous driving, robot navigation, and video surveillance, significantly reducing collision risk and improving system efficiency. Traditional model-driven approaches struggle with complex and highly dynamic scenes, resulting in poor prediction results.

[0003] Existing technologies cannot simulate scenarios that occur after pedestrian movement prediction, cannot analyze and capture information in complex scenarios, cannot filter the collected information, and cannot warn pedestrians and users during movement. This leads to collisions with pedestrians and congestion during movement, wasting time, difficulty in capturing and identifying information that affects one's own movement route in complex scenarios, cumbersome and difficult to analyze collected information, privacy protection, distraction leading to collisions with pedestrians, and congestion caused by improper handling.

[0004] In summary, a visual terminal behavior recognition and safety prediction method and system are needed to address the shortcomings of the existing technology. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method and system for visual terminal behavior recognition and safety prediction, aiming to solve the above problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for visual terminal behavior recognition and safety prediction, comprising the following steps: Step S1: deploy visual terminal devices, collect visual data, process the video stream through edge nodes, and obtain a video database; Step S2: Analyze the video database, extract features from the detected targets, and obtain target features; Step S3: Detect and identify abnormal behaviors in real time based on historical data and behavior detection models; Step S4: Evaluate and analyze the identified abnormal behaviors, predict upcoming security threats, and trigger the early warning system.

[0007] Optionally, step S1 is implemented in the following manner: Step A1: Identify monitoring scenario requirements and build the overall system architecture based on the monitoring area, monitoring target, and the type of behavior to be identified; Step A2: Select visual terminal devices and configure edge computing nodes based on the overall system architecture; Step A3: The camera continuously collects video streams and transmits them to the edge computing node, which processes the video streams in real time.

[0008] Optionally, the target features are extracted in step S2 in the following manner: Step B1: Split the video into several image frames according to a certain frequency, standardize the image frames and enhance the image quality; Step B2: Identify the desired target from each image frame and output the bounding box and category label of each target; Step B3: Analyze the target behavior and motion trajectory across frames through the target tracking module and assign a target ID. Step B4: On the detected and tracked target area, further extract features for classification, recognition or prediction tasks, and vectorize the extracted features and store them in a structured manner.

[0009] Optionally, the target features include but are not limited to appearance features, posture features, motion features, context features and meta-information features; Appearance features, used to express the color, surface texture, and contour shape information of the target; Posture features, used to express the motion information of the target during movement; Motion features, used to express the direction, speed, and path of the target during its motion; Contextual features are used to express semantic information, target relative position, interaction information, and temporal evolution trend of behavior; Meta-information features are used to express the time, location, and shooting angle information of video frame acquisition.

[0010] Optionally, the abnormal behavior in step S3 is identified by: Step C1: Collect normal behavior samples and historical video data to obtain labeled abnormal behavior cases; Step C2: Label abnormal behaviors and define a multi-level labeling system; Step C3: The behavior detection model uses reconstruction errors to assign an abnormal score to abnormal behaviors, and identifies abnormal behaviors by matching the anomaly score with the multi-level label system.

[0011] Optionally, the abnormal behavior identified in step S4 is evaluated and analyzed in the following manner: Step D1: extracting the contour shape information from the appearance features of the abnormal behavior target; Step D2: Based on the distance between the abnormal behavior target and the camera, the target outline in each image frame in the video is triangulated or trapezoidally corrected so that the abnormal behavior target presents an orthographic projection posture within the image frame, thereby obtaining a corrected image frame; Step D3: Calculate the size information of the abnormal target based on the corrected image frame, and use the center point of the abnormal behavior target in the corrected image frame as the motion trajectory point; Step D4: extracting motion trajectory information from the abnormal behavior target motion features based on the corrected image frame; Step D5: Select the motion trajectories of two adjacent abnormal behavior targets, process the motion trajectories so that the two motion trajectories are on the same horizontal plane, and calculate the distance D between the two motion trajectories; Step D6: Based on the distance D and the sizes of the two abnormal behavior targets, determine whether a collision occurs when the two abnormal behavior targets intersect.

[0012] Optionally, in step D6, determining whether a collision occurs when two abnormal behavior targets intersect is performed in the following manner: Before two abnormal behavior targets intersect: Select the image frame before the two abnormal behavior targets intersect, use the motion trajectory points of the two abnormal behavior targets in the image frame as the origin, draw two parallel lines, and use the distance between the two parallel lines as the distance D between the two motion trajectories; When two abnormal behavior targets are intersecting or after intersecting, the distance D is the actual distance between their motion trajectories.

[0013] Optionally, the early warning system is triggered in step S4 in the following manner: If half of the sum of the widths of the two abnormal behavior targets is greater than the distance D, the two abnormal behavior targets are safely interleaved; If half of the sum of the widths of the two abnormal behavior targets is less than or equal to the distance D, the two abnormal behavior targets will scrape or collide, triggering the early warning system and issuing a prompt.

[0014] A visual terminal behavior recognition and safety prediction system adopts the visual terminal behavior recognition and safety prediction method, including a visual terminal device module, a video data analysis module, a feature extraction module, an abnormal behavior detection module and a prediction and warning module; The visual terminal device module is used to collect video data, transmit the collected video data to the edge computing node for real-time processing, and generate a video database; The video data analysis module is used to split the video into several image frames at a certain frequency, standardize these frames and enhance the image quality, identify the required target from each image frame, and output the bounding box and category label of each target. The target tracking module also analyzes the behavior and movement trajectory of the same target across frames and assigns a target ID. Feature extraction module, used to extract the appearance, posture, motion, context and meta-information features of abnormal behavior targets; Abnormal behavior detection module, which is used to mark abnormal behaviors based on normal behavior samples and historical video data, and to score abnormal behaviors; The prediction and warning module is used to predict and warn of impending security threats based on identified abnormal behaviors and scores.

[0015] Beneficial effects of the present invention: 1. In this invention, by real-time monitoring and analysis of video data, abnormal behavior can be detected and identified in a timely manner, and impending security threats can be predicted, so that measures can be taken in advance to effectively prevent accidents. This greatly improves the security of public areas or specific monitoring scenarios. By processing video streams through edge computing nodes, the need to transmit large amounts of raw video data to central servers is reduced, network bandwidth pressure is reduced, and data processing speed is accelerated, achieving efficient resource utilization. 2. The present invention uses multi-level feature extraction technology, including appearance features, posture features, motion features, context features, and meta-information features, to provide rich data support for the identification of abnormal behavior, significantly improving the accuracy of behavior recognition and prediction. The system can automatically complete the entire process from video acquisition to abnormal behavior identification, evaluation and analysis, and then to early warning triggering, greatly reducing the need for manual intervention and improving work efficiency. 3. In the present invention, real-time monitoring and analysis based on historical data and behavioral models contribute to the establishment of a more intelligent management system. It is not limited to security monitoring, but can also be extended to multiple fields such as crowd analysis and behavioral pattern research. The system configuration can be adjusted according to different monitoring needs. It has strong flexibility and adaptability and is suitable for a variety of different application scenarios, such as shopping malls, airports, stations and other crowded places. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention is a flow chart of a method.

[0017] Figure 2 This is an internal flow chart of step S1 of the present invention.

[0018] Figure 3 This is an internal flow chart of step S2 of the present invention.

[0019] Figure 4 This is an internal flow chart of step S3 of the present invention.

[0020] Figure 5 This is an internal flow chart of step S4 of the present invention.

[0021] Figure 6 This is an example diagram of behavioral target analysis of the present invention. DETAILED DESCRIPTION

[0022] In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] like Figures 1 to 6 As shown, a visual terminal behavior recognition and safety prediction method includes the following steps: Step S1: deploy visual terminal devices, collect visual data, process the video stream through edge nodes, and obtain a video database; Step A1: Identify monitoring scenario requirements and build the overall system architecture based on the monitoring area, monitoring target, and the type of behavior to be identified; Step A2: Select visual terminal devices and configure edge computing nodes based on the overall system architecture; Step A3: The camera continuously collects video streams and transmits them to the edge computing node, which processes the video streams in real time.

[0024] Step S2: Analyze the video database, extract features from the detected targets, and obtain target features; Extract target features by: Step B1: Split the video into several image frames according to a certain frequency, standardize the image frames and enhance the image quality; Step B2: Identify the desired target from each image frame and output the bounding box and category label of each target; Step B3: Analyze the target behavior and motion trajectory across frames through the target tracking module and assign a target ID. Step B4: On the detected and tracked target area, further extract features for classification, recognition or prediction tasks, and vectorize the extracted features and store them in a structured manner.

[0025] Target features include but are not limited to appearance features, posture features, motion features, context features, and meta-information features; Appearance features, used to express the color, surface texture, and contour shape information of the target; Posture features, used to express the motion information of the target during movement; Motion features, used to express the direction, speed, and path of the target during its motion; Contextual features are used to express semantic information, target relative position, interaction information, and temporal evolution trend of behavior; Meta-information features are used to express the time, location, and shooting angle information of video frame acquisition.

[0026] Step S3: Detect and identify abnormal behaviors in real time based on historical data and behavior detection models; Identification of abnormal behavior through the following methods: Step C1: Collect normal behavior samples and historical video data to obtain labeled abnormal behavior cases; Step C2: Label abnormal behaviors and define a multi-level labeling system; Step C3: The behavior detection model uses reconstruction errors to assign an abnormal score to abnormal behaviors, and identifies abnormal behaviors by matching the anomaly score with the multi-level label system.

[0027] Step S4: Evaluate and analyze the identified abnormal behaviors, predict upcoming security threats, and trigger an early warning system; Identified abnormal behaviors are evaluated and analyzed through the following methods: Step D1: extracting the contour shape information from the appearance features of the abnormal behavior target; Step D2: Based on the distance between the abnormal behavior target and the camera, the target outline in each image frame in the video is triangulated or trapezoidally corrected so that the abnormal behavior target presents an orthographic projection posture within the image frame, thereby obtaining a corrected image frame; Step D3: Calculate the size information of the abnormal target based on the corrected image frame, and use the center point of the abnormal behavior target in the corrected image frame as the motion trajectory point; Step D4: extracting motion trajectory information from the abnormal behavior target motion features based on the corrected image frame; Step D5: Select the motion trajectories of two adjacent abnormal behavior targets, process the motion trajectories so that the two motion trajectories are on the same horizontal plane, and calculate the distance D between the two motion trajectories; Step D6: Based on the distance D and the sizes of the two abnormal behavior targets, determine whether a collision occurs when the two abnormal behavior targets intersect.

[0028] Determine whether a collision occurs when two abnormal behavior targets intersect, using the following methods: Before two abnormal behavior targets intersect: Select the image frame before the two abnormal behavior targets intersect, use the motion trajectory points of the two abnormal behavior targets in the image frame as the origin, draw two parallel lines, and use the distance between the two parallel lines as the distance D between the two motion trajectories; When two abnormal behavior targets are intersecting or after intersecting, the distance D is the actual distance between their motion trajectories.

[0029] Trigger the early warning system in the following ways: If half of the sum of the widths of the two abnormal behavior targets is greater than the distance D, the two abnormal behavior targets are safely interleaved; If half of the sum of the widths of the two abnormal behavior targets is less than or equal to the distance D, the two abnormal behavior targets will scrape or collide, triggering the early warning system and issuing a prompt.

[0030] like Figure 6 As shown in the figure, the red line is the motion trajectory of the pickup truck, the yellow line is the outer contour dimension line of the pickup truck, the blue line is the predicted motion trajectory line of the car, and the green line is the outer contour dimension line of the car. By calculating the distance D between the red line and the blue line, and based on the dimensional data of the yellow and green dimension lines, it is determined whether the two cars will scratch or collide when they meet.

[0031] A visual terminal behavior recognition and safety prediction system adopts the visual terminal behavior recognition and safety prediction method, including a visual terminal device module, a video data analysis module, a feature extraction module, an abnormal behavior detection module and a prediction and warning module; The visual terminal device module is used to collect video data, transmit the collected video data to the edge computing node for real-time processing, and generate a video database; The video data analysis module is used to split the video into several image frames at a certain frequency, standardize these frames and enhance the image quality, identify the required target from each image frame, and output the bounding box and category label of each target. The target tracking module also analyzes the behavior and movement trajectory of the same target across frames and assigns a target ID. Feature extraction module, used to extract the appearance, posture, motion, context and meta-information features of abnormal behavior targets; Abnormal behavior detection module, which is used to mark abnormal behaviors based on normal behavior samples and historical video data, and to score abnormal behaviors; The prediction and warning module is used to predict and warn of impending security threats based on identified abnormal behaviors and scores.

[0032] By monitoring and analyzing video data in real time, the present invention can promptly detect and identify abnormal behavior, thereby predicting impending security threats and enabling measures to be taken in advance to effectively prevent accidents. This significantly improves the security of public areas or specific monitoring scenarios. By processing video streams through edge computing nodes, the need to transmit large amounts of raw video data to central servers is reduced, network bandwidth pressure is reduced, and data processing speed is accelerated, achieving efficient resource utilization. The system uses multi-level feature extraction technology, including appearance features, posture features, motion features, context features, and meta-information features, to provide rich data support for the identification of abnormal behavior, significantly improving the accuracy of behavior recognition and prediction. The system can automatically complete the entire process from video acquisition to abnormal behavior identification, evaluation and analysis, and then to early warning triggering, greatly reducing the need for manual intervention and improving work efficiency. Real-time monitoring and analysis based on historical data and behavioral models helps to establish a more intelligent management system. It is not limited to security monitoring, but can also be extended to multiple fields such as crowd analysis and behavioral pattern research. The system configuration can be adjusted according to different monitoring needs. It has strong flexibility and adaptability and is suitable for a variety of different application scenarios, such as shopping malls, airports, stations and other crowded places.

[0033] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A visual terminal behavior recognition and safety prediction method, characterized in that: The following steps are involved: Step S1: deploy visual terminal devices, collect visual data, process the video stream through edge nodes, and obtain a video database; Step S2: Analyze the video database, extract features from the detected targets, and obtain target features; Step S3: Detect and identify abnormal behaviors in real time based on historical data and behavior detection models; Step S4: Evaluate and analyze the identified abnormal behaviors, predict upcoming security threats, and trigger the early warning system.

2. The visual terminal behavior recognition and safety prediction method according to claim 1 is characterized in that: The step S1 is implemented in the following manner: Step A1: Identify monitoring scenario requirements and build the overall system architecture based on the monitoring area, monitoring target, and the type of behavior to be identified; Step A2: Select visual terminal devices and configure edge computing nodes based on the overall system architecture; Step A3: The camera continuously collects video streams and transmits them to the edge computing node, which processes the video streams in real time.

3. The visual terminal behavior recognition and safety prediction method according to claim 1 is characterized in that: The target features are extracted in step S2 by: Step B1: Split the video into several image frames according to a certain frequency, standardize the image frames and enhance the image quality; Step B2: Identify the desired target from each image frame and output the bounding box and category label of each target; Step B3: Analyze the target behavior and motion trajectory across frames through the target tracking module and assign a target ID. Step B4: On the detected and tracked target area, further extract features for classification, recognition or prediction tasks, and vectorize the extracted features and store them in a structured manner.

4. The method for visual terminal behavior recognition and safety prediction according to claim 3 is characterized in that: The target features include but are not limited to appearance features, posture features, motion features, context features and meta-information features; Appearance features, used to express the color, surface texture, and contour shape information of the target; Posture features, used to express the motion information of the target during movement; Motion features, used to express the direction, speed, and path of the target during its motion; Contextual features are used to express semantic information, target relative position, interaction information, and temporal evolution trend of behavior; Meta-information features are used to express the time, location, and shooting angle information of video frame acquisition.

5. The visual terminal behavior recognition and safety prediction method according to claim 1 is characterized in that: The identification of abnormal behavior in step S3 is carried out in the following manner: Step C1: Collect normal behavior samples and historical video data to obtain labeled abnormal behavior cases; Step C2: Label abnormal behaviors and define a multi-level labeling system; Step C3: The behavior detection model uses reconstruction errors to assign an abnormal score to abnormal behaviors, and identifies abnormal behaviors by matching the anomaly score with the multi-level label system.

6. The visual terminal behavior recognition and safety prediction method according to claim 1 is characterized in that: The abnormal behavior identified in step S4 is evaluated and analyzed in the following ways: Step D1: extracting the contour shape information from the appearance features of the abnormal behavior target; Step D2: Based on the distance between the abnormal behavior target and the camera, the target outline in each image frame in the video is triangulated or trapezoidally corrected so that the abnormal behavior target presents an orthographic projection posture within the image frame, thereby obtaining a corrected image frame; Step D3: Calculate the size information of the abnormal target based on the corrected image frame, and use the center point of the abnormal behavior target in the corrected image frame as the motion trajectory point; Step D4: extracting motion trajectory information from the abnormal behavior target motion features based on the corrected image frame; Step D5: Select the motion trajectories of two adjacent abnormal behavior targets, process the motion trajectories so that the two motion trajectories are on the same horizontal plane, and calculate the distance D between the two motion trajectories; Step D6: Based on the distance D and the sizes of the two abnormal behavior targets, determine whether a collision occurs when the two abnormal behavior targets intersect.

7. The method for visual terminal behavior recognition and safety prediction according to claim 6, characterized in that: In step D6, whether a collision occurs when two abnormal behavior targets intersect is determined by: Before two abnormal behavior targets intersect: Select the image frame before the two abnormal behavior targets intersect, use the motion trajectory points of the two abnormal behavior targets in the image frame as the origin, draw two parallel lines, and use the distance between the two parallel lines as the distance D between the two motion trajectories; When two abnormal behavior targets are intersecting or after intersecting, the distance D is the actual distance between their motion trajectories.

8. The method for visual terminal behavior recognition and safety prediction according to claim 7, characterized in that: The early warning system is triggered in step S4 in the following manner: If half of the sum of the widths of the two abnormal behavior targets is greater than the distance D, the two abnormal behavior targets are safely interleaved; If half of the sum of the widths of the two abnormal behavior targets is less than or equal to the distance D, the two abnormal behavior targets will scrape or collide, triggering the early warning system and issuing a prompt.

9. A visual terminal behavior recognition and safety prediction system, using the visual terminal behavior recognition and safety prediction method according to any one of claims 1 to 8, characterized in that: It includes visual terminal equipment module, video data analysis module, feature extraction module, abnormal behavior detection module and prediction and warning module; The visual terminal device module is used to collect video data, transmit the collected video data to the edge computing node for real-time processing, and generate a video database; The video data analysis module is used to split the video into several image frames at a certain frequency, standardize these frames and enhance the image quality, identify the required target from each image frame, and output the bounding box and category label of each target. The target tracking module also analyzes the behavior and movement trajectory of the same target across frames and assigns a target ID. Feature extraction module, used to extract the appearance, posture, motion, context and meta-information features of abnormal behavior targets; Abnormal behavior detection module, which is used to mark abnormal behaviors based on normal behavior samples and historical video data, and to score abnormal behaviors; The prediction and warning module is used to predict and warn of impending security threats based on identified abnormal behaviors and scores.

Images