A method and system for analyzing illegal behavior of grain storage operation personnel

Abstract

This invention relates to the field of safety monitoring technology, specifically disclosing a method and system for analyzing violations by grain storage workers. The method includes: S1: acquiring multiple video feeds of the storage operation environment to obtain multiple video frames; S2: spatially calibrating the multiple video frames to generate a semantic map of the storage scene; S3: identifying targets in the multiple video frames and associating the same target identified in consecutive video frames to generate a sequence of motion trajectories for each target; S4: extracting the coordinates of key skeletal points of workers within the targets, generating a sequence of motion evolution, and performing motion determination and velocity / acceleration marking; S5: mapping each target as a node to the semantic map of the storage scene to generate a dynamic target relationship graph; S6: generating a behavior determination result based on the dynamic target relationship graph. This invention can solve the problem of insufficient accuracy in existing grain storage safety supervision schemes regarding violations by workers.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring technology, specifically to a method and system for analyzing violations by grain storage workers. Background Technology

[0002] In grain storage operations, workers frequently move between various functional areas, including conveyor equipment areas, high-altitude work areas, and hazardous machinery areas, resulting in constantly changing spatial relationships between personnel, equipment, and hazardous areas. Existing warehouse safety monitoring solutions typically employ target detection methods based on video images to identify personnel and equipment targets and determine the presence of violations based on single-frame detection results. However, the same human posture can have different safety implications in different work areas. Existing solutions lack the analytical capability to correlate posture information with scene area attributes, leading to confusion between normal operations and violations. Furthermore, existing solutions use single-frame images as the analysis unit, failing to capture the evolution of actions and resulting in insufficient accuracy in identifying complex violations composed of multiple action stages. These deficiencies lead to technical problems in existing warehouse safety monitoring solutions, such as high false alarm rates and missed detection of complex violations in complex operational scenarios. Summary of the Invention

[0003] To address the aforementioned problems, this invention proposes a method for analyzing violations by grain storage workers, comprising: S1: Acquire multiple video streams of the warehouse operation environment to obtain multiple video frames; S2: Spatial calibration is performed on the multiple video frames. Based on the warehousing operation environment, the spatially calibrated area is divided into multiple different semantic regions. Corresponding risk attribute values ​​are added to generate a semantic map of the warehousing scene. S3: Identify targets in multiple video frames, associate the same target identified in consecutive video frames, and generate a motion trajectory sequence for each target. The targets include workers and work facilities. S4: Extract the coordinates of key skeletal points of the workers in the target, generate the motion evolution sequence, and perform motion determination and velocity and acceleration marking; S5: Using each target as a node, map it to the semantic map of the warehousing scene, use the edges between nodes to represent the distance relationship between targets, and generate a dynamic target relationship map based on the action evolution sequence of the operators and the movement trajectory sequence of the operating facilities; S6: Based on the target relationship dynamic graph, and in conjunction with the warehouse safety operation specifications, determine whether the operator's behavior is compliant, and generate a behavior judgment result.

[0004] Furthermore, step S2 also includes: Before spatial calibration of the multiple video frames, illumination compensation and defogging processing are performed on the multiple video frames based on different time periods and weather conditions.

[0005] Furthermore, in step S3, associating the same target identified in multiple consecutive video frames includes: S31: In the initial frame, extract the appearance feature vector and spatial calibration position of the target, and predict the position prediction value of the next frame based on Kalman filtering; S32: In the next frame, repeat the extraction and prediction operations. If the appearance feature vector and spatial calibration position of the target in the current frame match the appearance feature vector and position prediction value of a target in the previous frame, they are determined to be the same target and are associated. S33: Repeat steps S31 and S32 to generate the motion trajectory sequence of each target; When a target is occluded, the motion trajectory sequence of the target is retained. If a target matching the appearance feature vector is detected again within a preset number of frames, it is associated and continued to the original motion trajectory.

[0006] Furthermore, in step S4, generating the action evolution sequence includes: The coordinates of key skeletal points are extracted from the workers in the target, the joint angles are calculated based on the skeletal point coordinates, the angle changes of the same joint between all adjacent frames are counted, and the motion evolution sequence is generated based on the skeletal point displacements and joint angle changes of the same worker in consecutive frames.

[0007] Furthermore, in step S4, the motion determination includes: Within a preset time period, when the number of joints exceeds a preset proportion: When the absolute value of the angle change changes from less than the first preset threshold to greater than the first preset threshold, it is determined to be the starting stage of motion; When the absolute value of the angle change remains greater than the first preset threshold, it is determined to be in the motion continuous phase. When the absolute value of the angle change changes from being greater than the first preset threshold to being less than the first preset threshold, it is determined to be the end of the motion phase; The determination result is marked in real time in the action evolution sequence.

[0008] Furthermore, in step S4, the velocity and acceleration markers include: The motion evolution sequence is segmented using a sliding window method, and the velocity and acceleration characteristics of key skeletal points are calculated, including: Within each window, let the coordinates of the key skeleton point m in frame t be... Calculate the velocity characteristics of this bone point. and acceleration characteristics speed characteristics and acceleration characteristics Added to the action evolution sequence at the corresponding moment. The velocity characteristics at frame t are... The acceleration characteristics at frame t.

[0009] Furthermore, step S5 includes: S51: Map all targets in the current frame as nodes to the warehouse scene semantic map. The nodes are divided into worker nodes and facility nodes according to the target type. S52: Based on the action evolution sequence of the operators and the motion trajectory sequence of other types of targets, calculate the spatial distance between each operator node and each operation facility node and the boundary of the hazardous area, and use it as the edge attribute value to generate a dynamic target relationship graph, wherein the boundary of the hazardous area is determined according to the semantic map of the warehousing scene.

[0010] Furthermore, step S6 includes: Read the semantic region where the operator node is currently located and obtain the set of safety operation specification items corresponding to that semantic region; Based on the motion determination result, velocity characteristics, acceleration characteristics, and at least one of the edge attribute values, combined with the set of safety operation specification items, a behavior determination result is generated; the behavior determination result includes three levels: prompt, warning, and emergency.

[0011] Furthermore, the target relationship dynamic graph also includes establishing personnel relationships for the nodes of the operators: When two or more workers' nodes are in the same semantic region and move in the same direction, the corresponding nodes are marked as having a cooperative relationship.

[0012] Accordingly, this invention also proposes a system for analyzing violations by grain storage workers, comprising: The multi-channel video frame acquisition module acquires multiple video streams of the warehouse operation environment and obtains multiple video frames. The warehouse scene semantic map generation module spatially calibrates the multiple video frames, divides the spatially calibrated area into multiple different semantic regions based on the warehouse operation environment, adds corresponding risk attribute values, and generates a warehouse scene semantic map. The motion trajectory sequence generation module identifies targets in multiple video frames and associates the same target identified in consecutive video frames to generate motion trajectory sequences for each target, including personnel and equipment. The motion evolution sequence generation module extracts the coordinates of key skeletal points of the workers in the target, generates a motion evolution sequence, and performs motion determination and velocity and acceleration marking. The target relationship dynamic graph generation module uses each target as a node and maps it to the semantic map of the warehousing scene. The edges between nodes represent the distance relationship between targets. The module generates a target relationship dynamic graph based on the action evolution sequence of the operators and the movement trajectory sequence of the operating facilities. The behavior determination result generation module, based on the target relationship dynamic graph and combined with the warehouse safety operation specifications, determines whether the operator's behavior is compliant and generates a behavior determination result.

[0013] This invention spatially calibrates multiple video frames and divides them into semantic regions with risk attribute values ​​to generate a semantic map of the warehouse scene. This introduces scene region context into behavior determination, allowing the same posture to be treated differently based on the risk attributes of its location, avoiding confusion between normal operation and violation. Furthermore, it constructs a dynamic target relationship graph with targets as nodes and distance relationships as edges, integrating the dynamic spatial relationships between personnel actions, equipment, and hazardous areas to capture the evolution of actions. The behavior determination results are then output on the dynamic graph in conjunction with warehouse safety operation specifications, thereby simultaneously reducing false alarm rates and the missed detection rate of complex violations. Attached Figure Description

[0014] Figure 1 This is a flowchart of a method for analyzing violations by grain storage workers according to the present invention; Figure 2 This is an architecture diagram of a grain storage operation personnel violation analysis system according to the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of this application to enable the reader to better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0016] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0017] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0018] The embodiments of this application will be described in detail below with reference to the accompanying drawings to provide a clearer understanding of the purpose, features, and advantages of this application. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of this application, but are merely for illustrating the essential spirit of the technical solution of this application.

[0019] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0020] To address the shortcomings of existing grain storage safety supervision schemes in handling violations by operational personnel, this invention proposes a method and system for analyzing violations by grain storage operational personnel. like Figure 1 As shown, the method includes steps S1 to S6. Step S1: Acquire multiple video feeds of the warehouse operation environment to obtain multiple video frames; Specifically, multiple video acquisition devices are deployed at key locations in the grain storage site, including conveyor areas, elevated work areas, mechanical work areas, and entrances / exits. Each video acquisition device captures video at a uniform frame rate and decodes each video frame to obtain multiple video frames. Since the equipment inside the grain storage facility is densely packed, a single viewpoint inevitably has obstructions and blind spots. Therefore, by using multiple complementary viewpoints, continuous observation of personnel and facilities can be ensured.

[0021] Step S2: Spatial calibration is performed on the multiple video frames. Based on the warehousing operation environment, the spatially calibrated area is divided into multiple different semantic regions, corresponding risk attribute values ​​are added, and a semantic map of the warehousing scene is generated. Spatial calibration refers to the process of establishing a mapping relationship between the pixel coordinates of a video frame and the actual physical coordinates of the warehouse site. For example, using a calibration board or a reference object of known size on-site, the coordinate transformation matrix is ​​calculated so that any pixel in the video frame can be converted into a planar position on-site. A semantic region refers to a region on the site plan divided according to operational functions, such as a conveyor belt operation area, an elevated platform area, a power distribution hazard area, and a vehicle passageway area. Risk attribute values ​​are quantitative values ​​characterizing the degree of danger in each semantic region; the higher the degree of danger, the larger the risk attribute value. A warehouse scene semantic map is a planar map of the site that simultaneously carries area boundary information and risk attribute values.

[0022] By unifying all video feeds to the same real physical coordinate system, the same area observed by different video acquisition devices can be aligned and merged, ensuring a consistent positional reference when mapping the target onto a map. Risk attribute values ​​incorporate scene context information into behavior determination, allowing the same posture to be treated differently based on the risk attributes of its location. For example, rapid movement might be considered normal travel in a regular passageway but dangerous in a high-altitude platform area, thus preventing confusion between normal operations and violations and reducing false alarms at the source.

[0023] Step S3: Identify targets in multiple video frames and associate the same target identified in consecutive video frames to generate a motion trajectory sequence for each target. The targets include workers and work facilities. Specifically, multiple video frames are inspected frame by frame to obtain the category and location of each target, including operational facilities such as belt conveyors, forklifts, and hoists. Then, the same target in consecutive frames is associated, and the spatially calibrated positions of the same target in each frame are arranged in chronological order to generate a motion trajectory sequence. The motion trajectory sequence refers to the sequence formed by arranging the positions of the same target in multiple consecutive frames according to time.

[0024] Step S4: Extract the coordinates of key skeletal points of the workers in the target, generate the motion evolution sequence, and perform motion determination and velocity and acceleration marking.

[0025] Key skeletal points refer to joints that characterize human posture, such as the points corresponding to joints like the shoulder, elbow, wrist, hip, knee, and ankle. A motion evolution sequence is a sequence formed by arranging the coordinates of skeletal points, joint angles, and their changes in consecutive frames in chronological order, used to depict the complete process of a motion from start to finish. Motion determination refers to the process of dividing the motion process into the initiation phase, the duration phase, and the termination phase. Velocity and acceleration labeling refers to attaching the velocity and acceleration characteristics of skeletal point motion to the motion evolution sequence at the corresponding moment. Violations often involve specific kinematic patterns; for example, rapid running corresponds to a large velocity characteristic, while a sudden fall corresponds to a sudden change in acceleration characteristics. Static posture alone cannot distinguish these situations; introducing kinematic features such as velocity and acceleration can significantly improve the ability to identify complex violations.

[0026] Step S5: Using each target as a node, map it to the semantic map of the warehousing scene, use the edges between nodes to represent the distance relationship between targets, and generate a dynamic target relationship map based on the action evolution sequence of the operators and the movement trajectory sequence of the operating facilities.

[0027] A dynamic target relationship graph refers to a graph-structured data structure where targets are nodes and the distance relationships between targets are edges, updated frame-by-frame with video. Nodes carry attributes such as target type, location, action stage, and velocity / acceleration, while edges carry distance attributes between targets. "Dynamic" means that the graph refreshes with each frame, reflecting the real-time evolution of spatial relationships between personnel, equipment, and hazardous areas. Violations in warehousing scenarios often manifest as inappropriate changes in the spatial relationships between people and equipment or hazardous areas, such as personnel entering the hazardous area of ​​operating conveyor equipment. The graph structure can simultaneously express personnel action information and multi-target spatial relationship information within the same data structure, enabling subsequent judgments to perceive complex patterns combining actions and spatial relationships.

[0028] Step S6: Based on the target relationship dynamic graph and combined with the warehouse safety operation specifications, determine whether the operator's behavior is compliant and generate a behavior judgment result.

[0029] Specifically, the node and edge attributes in the dynamic graph of target relationships are compared with the warehouse safety operation specifications to determine whether the current behavior of the operators is compliant, and the behavior determination result is output. The behavior determination result can include the compliance conclusion, the type of violation, and the operators involved in the violation, semantic region, and time information, which facilitates the traceability and handling by the supervisors.

[0030] This invention spatially calibrates multiple video frames and divides them into semantic regions with risk attribute values ​​to generate a semantic map of the warehouse scene. This introduces scene region context into behavior determination, allowing the same posture to be treated differently based on the risk attributes of its location, avoiding confusion between normal operation and violation. Furthermore, it constructs a dynamic target relationship graph with targets as nodes and distance relationships as edges, integrating the dynamic spatial relationships between personnel actions, equipment, and hazardous areas to capture the evolution of actions. The behavior determination results are then output on the dynamic graph in conjunction with warehouse safety operation specifications, thereby simultaneously reducing false alarm rates and the missed detection rate of complex violations.

[0031] Furthermore, step S2 also includes: Before spatial calibration of the multiple video frames, illumination compensation and defogging processing are performed on the multiple video frames based on different time periods and weather conditions.

[0032] The dust concentration at grain storage sites is high, and lighting conditions fluctuate significantly with the time of day and weather. This can easily lead to problems such as fogging, uneven brightness, and decreased contrast in video images, directly affecting the accuracy of spatial calibration and target recognition. Therefore, illumination compensation and dehazing are performed before spatial calibration. Illumination compensation can be achieved through histogram equalization or gamma correction, while dehazing can be achieved through algorithms based on dark channel priors.

[0033] Furthermore, step S3 includes: Step S31: In the initial frame, extract the target's appearance feature vector and spatial calibration position, and predict the position value for the next frame based on Kalman filtering. The appearance feature vector refers to the numerical vector representing color, texture, and shape information extracted from the image region where the target is located, and can be generated by a feature extraction network. Kalman filtering is a recursive state estimation method that uses the target's position and velocity information in historical frames to estimate the most likely position of the target in the next frame.

[0034] Step S32: In the next frame, repeat the extraction and prediction operations. If the appearance feature vector and spatial calibration position of the target in the current frame match the appearance feature vector and position prediction value of a target in the previous frame, they are determined to be the same target and are associated.

[0035] Step S33: Repeat steps S31 and S32 to generate the motion trajectory sequence of each target.

[0036] Furthermore, when a target is occluded, the motion trajectory sequence of the target is preserved. If a target matching the appearance feature vector is detected again within a preset number of frames, it is associated with and continued to the original motion trajectory.

[0037] Appearance features are robust to target deformation and viewing angle changes, while position prediction provides motion priors. The complementarity of these two can reduce false matching when multiple targets intersect. Furthermore, mutual occlusion between personnel and equipment is very common in warehouses. If the same target is reassigned after occlusion, the action evolution sequence of the operator will be fragmented, and complex violations composed of multiple action stages will be difficult to identify. Therefore, trajectory continuity ensures the integrity of the action evolution sequence.

[0038] Furthermore, in step S4, generating the motion evolution sequence includes: extracting the coordinates of key skeletal points from the workers in the target, calculating the joint angles based on the skeletal point coordinates, counting the angle changes of the same joint between all adjacent frames, and generating the motion evolution sequence based on the skeletal point displacements and joint angle changes of the same workers in consecutive frames.

[0039] Joint angles are calculated from the angle between the lines connecting two adjacent bone segments of the same joint, and bone point displacements are obtained from the coordinate differences of the same bone point in adjacent frames. The changes in joint angles and bone point displacements are used to describe the rhythm, amplitude, and intensity of the action itself, thus enabling the same action to have similar characteristic expressions under different personnel and camera positions, improving the generalization ability of subsequent behavior determination.

[0040] Furthermore, in step S4, the motion determination includes: Within a preset time period, when the number of joints exceeds a preset proportion: When the absolute value of the angle change changes from less than the first preset threshold to greater than the first preset threshold, it is determined to be the starting stage of motion; When the absolute value of the angle change remains greater than the first preset threshold, it is determined to be in the motion continuous phase. When the absolute value of the angle change changes from being greater than the first preset threshold to being less than the first preset threshold, it is determined to be the end of the motion phase; The determination result is marked in real time in the action evolution sequence.

[0041] Human movement is characterized by the coordinated movement of multiple joints. Changes in the angle of a single joint may be caused by jitter in the coordinates of skeletal points or by recognition noise. Only consistent changes across multiple joints can accurately reflect the transition between movement stages, thus avoiding misjudgment of stages. Clearly defining the initiation, continuation, and termination stages of movement allows subsequent behavioral judgments to verify safe operating procedures based on complete movement units, such as the action chain of standing up, walking, turning, and putting down in handling operations.

[0042] Furthermore, in step S4, the velocity and acceleration markers include: within each window, assuming the coordinates of the key skeleton point m in frame t are... Calculate the velocity characteristics of this bone point. and acceleration characteristics speed characteristics and acceleration characteristics Added to the action evolution sequence at the corresponding moment. The velocity characteristics at frame t are... The acceleration characteristics at frame t.

[0043] The length of the sliding window can be set according to the video frame rate and the duration of the target action; however, this invention does not impose any limitations on this.

[0044] Furthermore, step S5 includes: S51: Map all targets in the current frame as nodes to the warehouse scene semantic map. The nodes are divided into worker nodes and facility nodes according to the target type. S52: Based on the action evolution sequence of the operators and the motion trajectory sequence of other types of targets, calculate the spatial distance between each operator node and each operation facility node and the boundary of the hazardous area, and use it as the edge attribute value to generate a dynamic target relationship graph, wherein the boundary of the hazardous area is determined according to the semantic map of the warehousing scene.

[0045] Furthermore, step S6 includes: Read the semantic region where the operator node is currently located and obtain the set of safety operation specification items corresponding to that semantic region; The set of safe operation procedure items refers to a safety rule library organized according to semantic regions. For example, the high-altitude operation area corresponds to the item prohibiting climbing over guardrails, and the conveyor equipment area corresponds to the items of maintaining a safe distance and prohibiting crossing the conveyor belt.

[0046] Based on the motion determination result, velocity characteristics, acceleration characteristics, and at least one of the edge attribute values, combined with the set of safety operation specification items, a behavior determination result is generated; the behavior determination result includes three levels: prompt, warning, and emergency.

[0047] The same action is subject to different safety requirements in different regions. Adaptive scenario-based judgment is achieved through the correspondence between regions and regulations, avoiding misjudgments caused by applying the same set of rules to all regions. Behavioral judgment results are divided into three levels: alert, warning, and emergency. The alert level corresponds to minor deviations from regulations, the warning level to explicit violations, and the emergency level to situations endangering personal safety requiring immediate action. This tiered output allows regulatory resources to be allocated according to risk priority.

[0048] Furthermore, when the nodes of two or more workers are located in the same semantic region and move in the same direction, the corresponding nodes are marked as having a collaborative relationship. Collaborative relationships are used to characterize the state of multiple workers working together, such as multiple people cooperating to move materials.

[0049] The reason for establishing collaborative relationship markers is that the safety implications of an individual worker's actions in a multi-person scenario are related to group behavior. By introducing collaborative relationships, the judgment process can analyze nodes within the same collaborative group as a whole. This avoids misjudging normal collaborative work as a violation of personnel gathering and also makes it easier to identify violations that require the cooperation of multiple people, thereby further improving the accuracy of behavior judgment results.

[0050] This invention also proposes a system for analyzing violations by grain storage workers, such as... Figure 2 As shown, it includes: The multi-channel video frame acquisition module acquires multiple video streams of the warehouse operation environment and obtains multiple video frames. The warehouse scene semantic map generation module spatially calibrates the multiple video frames, divides the spatially calibrated area into multiple different semantic regions based on the warehouse operation environment, adds corresponding risk attribute values, and generates a warehouse scene semantic map. The motion trajectory sequence generation module identifies targets in multiple video frames and associates the same target identified in consecutive video frames to generate motion trajectory sequences for each target, including personnel and equipment. The motion evolution sequence generation module extracts the coordinates of key skeletal points of the workers in the target, generates a motion evolution sequence, and performs motion determination and velocity and acceleration marking. The target relationship dynamic graph generation module uses each target as a node and maps it to the semantic map of the warehousing scene. The edges between nodes represent the distance relationship between targets. The module generates a target relationship dynamic graph based on the action evolution sequence of the operators and the movement trajectory sequence of the operating facilities. The behavior determination result generation module, based on the target relationship dynamic graph and combined with the warehouse safety operation specifications, determines whether the operator's behavior is compliant and generates a behavior determination result.

[0051] The implementation steps of the present invention will be described below with reference to a specific scenario: A grain reserve warehouse, hereinafter referred to as Warehouse A, implemented an intelligent monitoring upgrade for its storage area in a certain year. Warehouse A has four operating areas: a conveyor equipment area, a high-altitude operation area, a hazardous machinery area, and a personnel passage area. A total of six video capture devices were deployed to cover the entire area. Personnel were numbered from Personnel 001 to Personnel 006, and operating facilities were numbered from Equipment A to Equipment D. Initially, all personnel were in the personnel passage area, Equipment A was a conveyor belt, Equipment B was a hoist (both in operation), Equipment C was a maintenance platform elevator, and Equipment D was a grain unloading machine (both in standby mode).

[0052] In step S1, the six video acquisition devices acquire multiple video streams of the storage environment of Warehouse A at a uniform frame rate, and decode each video stream frame by frame to obtain multiple video frames.

[0053] In step S2, spatial calibration is performed on multiple video frames, mapping the image pixel coordinates to a unified warehousing plane coordinate system with the southwest corner of warehouse A as the origin. Then, based on the warehousing operating environment, the spatially calibrated area is divided into four different semantic regions, and corresponding risk attribute values ​​are added to generate a warehousing scene semantic map, as shown in Table 1. Because the image quality of warehouse A degrades at night and during periods of high dust concentration, the system automatically performs illumination compensation and dehazing processing on multiple video frames during these periods to ensure the consistency of visual features across all video frames.

[0054]

[0055] In step S3, the system identifies targets in multiple video frames, including personnel and equipment, and associates the same target identified in consecutive video frames. Specifically, in the starting frame, the system extracts the target's appearance feature vector and spatial calibration position, predicts the position value of the next frame based on Kalman filtering, and in subsequent frames, matches the target's appearance feature vector and spatial calibration position in the current frame with the appearance feature vector and position prediction value of each target in the previous frame. If a match is successful, the target is identified as the same target and associated. At 14:32 on a certain work day in that year, the system identified personnel 001 moving from area 04 to area 01, personnel 002 in area 02, and equipment A in operation. Personnel 001's trajectory was briefly interrupted from frame 48 to frame 52 due to dust obstruction. The system retained its motion trajectory sequence, and after re-detecting a target with a matching appearance feature vector in frame 54, the association was completed, and the original motion trajectory was resumed. The identification and trajectory information of some targets are shown in Table 2.

[0056]

[0057] In step S4, the system extracts the coordinates of seven key skeletal points—head, shoulder, elbow, wrist, hip, knee, and ankle—from the worker in the target. Based on these coordinates, joint angles are calculated, and the angle changes of the same joint between all adjacent frames are statistically analyzed. A motion evolution sequence is generated based on the skeletal point displacements and joint angle changes of the same worker in consecutive frames, and motion determination and velocity / acceleration marking are performed. Specifically, the system segments the motion evolution sequence using a sliding window approach, with a window length of 8 frames and adjacent windows overlapping by 4 frames. Within each window, the velocity and acceleration characteristics of the key skeletal points are calculated and appended to the motion evolution sequence at the corresponding time point.

[0058] Taking person 001 as an example, the angle of their right elbow joint changes from 142 degrees to 128 degrees in frame 55. The absolute value of the angle change of a number of joints exceeding a preset proportion changes from being less than the first preset threshold to being greater than the first preset threshold, which is determined to be the starting stage of the movement. In frame 56, the angle of the right elbow joint continues to change to 109 degrees. The absolute value of the angle change of a number of joints exceeding a preset proportion remains greater than the first preset threshold, which is determined to be the continuous stage of the movement. In frame 57, the angle of the right elbow joint changes to 107 degrees. The absolute value of the angle change changes from being greater than the first preset threshold to being less than the first preset threshold, which is determined to be the ending stage of the movement. The determination results are marked in real time in the action evolution sequence, as shown in Table 3. The action evolution sequence of person 002 shows that their right arm is continuously raised, and the absolute value of the angle change of multiple joints is continuously greater than the first preset threshold, which is determined to be the continuous stage of the movement.

[0059]

[0060] In step S5, the system executes steps S51 and S52. Step S51 maps all targets in the current frame as nodes to the warehouse scene semantic map. Nodes are categorized into worker nodes and facility nodes based on target type. Step S52 calculates the spatial distance between each worker node and each facility node, as well as the boundary of the hazardous area, based on the worker's action evolution sequence and the facility's motion trajectory sequence. This distance is used as an edge attribute value to generate a dynamic target relationship graph. The hazardous area boundary is determined based on the warehouse scene semantic map. Taking frame 56 as an example, the spatial distance between worker 001 and equipment A decreases continuously from 62 pixels in frame 54 to 38 pixels in frame 56. Furthermore, the nodes of worker 003 and worker 004 are located in the same region 04 and move in the same direction. The system marks the corresponding nodes as having a cooperative relationship. Some edge attributes of the target relationship dynamic graph in frame 56 are shown in Table 4.

[0061]

[0062] In step S6, the system reads the semantic region where the worker node is currently located, obtains the set of safety operation specification items corresponding to that semantic region, and generates a behavior judgment result based on at least one feature among the motion judgment result, velocity features, acceleration features, and edge attribute values, combined with the set of safety operation specification items. The behavior judgment result includes three levels: prompt, warning, and emergency. Specifically, the semantic region where worker 001 is currently located is region 01, and the corresponding specification item set includes items such as "prohibition of stopping while the equipment is running." Worker 001 is in the continuous motion phase, and the spatial distance between him and the running equipment A is continuously decreasing to 38 pixels. The system determines that worker 001 has a dangerous behavior of approaching the running equipment, and the behavior judgment result is emergency level. Worker 002 is in region 02, and his action evolution sequence shows that his right arm is continuously raised, and he is in the continuous motion phase. The system does not recognize that he is wearing a safety belt, and determines that it is an illegal climbing behavior, and the behavior judgment result is warning level. Workers 003 and 004 are marked as having a cooperative relationship. Their collaborative handling is a normal operation, and the behavior judgment result is prompt level, only recorded without alarm. The behavior judgment results are shown in Table 5.

[0063]

[0064] Data from each step maintains a consistent coordinate system and identifier standard throughout the entire process, ensuring that data references across steps are effective.

[0065] The solutions provided by the embodiments of this application have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0066] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0067] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

Claims

1. A method for analyzing violations by grain storage workers, characterized in that, include: S1: Acquire multiple video streams of the warehouse operation environment to obtain multiple video frames; S2: Spatial calibration is performed on the multiple video frames. Based on the warehousing operation environment, the spatially calibrated area is divided into multiple different semantic regions. Corresponding risk attribute values ​​are added to generate a semantic map of the warehousing scene. S3: Identify targets in multiple video frames, associate the same target identified in consecutive video frames, and generate a motion trajectory sequence for each target. The targets include workers and work facilities. S4: Extract the coordinates of key skeletal points of the workers in the target, generate the motion evolution sequence, and perform motion determination and velocity and acceleration marking; S5: Using each target as a node, map it to the semantic map of the warehousing scene, use the edges between nodes to represent the distance relationship between targets, and generate a dynamic target relationship map based on the action evolution sequence of the operators and the movement trajectory sequence of the operating facilities; S6: Based on the target relationship dynamic graph, and in conjunction with the warehouse safety operation specifications, determine whether the operator's behavior is compliant, and generate a behavior judgment result.

2. The method for analyzing violations by grain storage workers according to claim 1, characterized in that, Step S2 also includes: Before spatial calibration of the multiple video frames, illumination compensation and defogging processing are performed on the multiple video frames based on different time periods and weather conditions.

3. The method for analyzing violations by grain storage workers according to claim 1, characterized in that, In step S3, associating the same target identified in multiple consecutive video frames includes: S31: In the initial frame, extract the appearance feature vector and spatial calibration position of the target, and predict the position prediction value of the next frame based on Kalman filtering; S32: In the next frame, repeat the extraction and prediction operations. If the appearance feature vector and spatial calibration position of the target in the current frame match the appearance feature vector and position prediction value of a target in the previous frame, they are determined to be the same target and are associated. S33: Repeat steps S31 and S32 to generate the motion trajectory sequence of each target; When a target is occluded, the motion trajectory sequence of the target is retained. If a target matching the appearance feature vector is detected again within a preset number of frames, it is associated and continued to the original motion trajectory.

4. The method for analyzing violations by grain storage workers according to claim 1, characterized in that, In step S4, generating the action evolution sequence includes: The coordinates of key skeletal points are extracted from the workers in the target, the joint angles are calculated based on the skeletal point coordinates, the angle changes of the same joint between all adjacent frames are counted, and the motion evolution sequence is generated based on the skeletal point displacements and joint angle changes of the same worker in consecutive frames.

5. The method for analyzing violations by grain storage workers according to claim 4, characterized in that, In step S4, the motion determination includes: Within a preset time period, when the number of joints exceeds a preset proportion: When the absolute value of the angle change changes from less than the first preset threshold to greater than the first preset threshold, it is determined to be the starting stage of motion; When the absolute value of the angle change remains greater than the first preset threshold, it is determined to be in the motion continuous phase. When the absolute value of the angle change changes from being greater than the first preset threshold to being less than the first preset threshold, it is determined to be the end of the motion phase; The determination result is marked in real time in the action evolution sequence.

6. The method for analyzing violations by grain storage workers according to claim 5, characterized in that, In step S4, the velocity and acceleration markers include: The motion evolution sequence is segmented using a sliding window method, and the velocity and acceleration characteristics of key skeletal points are calculated, including: Within each window, let the coordinates of the key skeleton point m in frame t be... Calculate the velocity characteristics of this bone point. and acceleration characteristics speed characteristics and acceleration characteristics Added to the action evolution sequence at the corresponding moment. The velocity characteristics at frame t are... The acceleration characteristics at frame t.

7. The method for analyzing violations by grain storage workers according to claim 6, characterized in that, Step S5 includes: S51: Map all targets in the current frame as nodes to the warehouse scene semantic map. The nodes are divided into worker nodes and facility nodes according to the target type. S52: Based on the action evolution sequence of the operators and the motion trajectory sequence of other types of targets, calculate the spatial distance between each operator node and each operation facility node and the boundary of the hazardous area, and use it as the edge attribute value to generate a dynamic target relationship graph, wherein the boundary of the hazardous area is determined according to the semantic map of the warehousing scene.

8. The method for analyzing violations by grain storage workers according to claim 7, characterized in that, Step S6 includes: Read the semantic region where the operator node is currently located and obtain the set of safety operation specification items corresponding to that semantic region; Based on the motion determination result, velocity characteristics, acceleration characteristics, and at least one of the edge attribute values, combined with the set of safety operation specification items, a behavior determination result is generated; the behavior determination result includes three levels: prompt, warning, and emergency.

9. The method for analyzing violations by grain storage workers according to claim 1, characterized in that, The target relationship dynamic graph also includes establishing personnel relationships for the nodes of the operators: When two or more workers' nodes are in the same semantic region and move in the same direction, the corresponding nodes are marked as having a cooperative relationship.

10. A system for analyzing violations by grain storage workers, characterized in that, include: The multi-channel video frame acquisition module acquires multiple video streams of the warehouse operation environment and obtains multiple video frames. The warehouse scene semantic map generation module spatially calibrates the multiple video frames, divides the spatially calibrated area into multiple different semantic regions based on the warehouse operation environment, adds corresponding risk attribute values, and generates a warehouse scene semantic map. The motion trajectory sequence generation module identifies targets in multiple video frames and associates the same target identified in consecutive video frames to generate motion trajectory sequences for each target, including personnel and equipment. The motion evolution sequence generation module extracts the coordinates of key skeletal points of the workers in the target, generates a motion evolution sequence, and performs motion determination and velocity and acceleration marking. The target relationship dynamic graph generation module uses each target as a node and maps it to the semantic map of the warehousing scene. The edges between nodes represent the distance relationship between targets. The module generates a target relationship dynamic graph based on the action evolution sequence of the operators and the movement trajectory sequence of the operating facilities. The behavior determination result generation module, based on the target relationship dynamic graph and combined with the warehouse safety operation specifications, determines whether the operator's behavior is compliant and generates a behavior determination result.

Images