System and method for analyzing a work area

Abstract

This paper illustrates systems and methods for analyzing work areas. Embodiments of vision-based object perception systems for activity analysis, security monitoring, or both are provided. Embodiments of the perception subsystem detect multiple object classes (e.g., construction machines and humans) in real time, while simultaneously estimating the poses and actions of the detected objects. Security monitoring and object activity analysis embodiments can be based on the perception results. To evaluate the performance of the embodiments, a dataset including multiple object classes under different lighting conditions with human annotations was collected. Experimental results show that the proposed action recognition method outperforms existing methods by approximately 5.18% in initial accuracy.

Description

Technical Field

[0001] This disclosure generally relates to systems and methods for computer learning, which can provide improved computer performance, features, and uses. More specifically, this disclosure relates to systems and methods for vision-based security monitoring and object activity analysis. Background Technology

[0002] Research and development related to autonomous vehicles have increased dramatically in recent years. Autonomous vehicles have been studied for a variety of uses, including autonomous cars, autonomous trucks, autonomous robots, autonomous drones, and autonomous construction vehicles. The rationale for researching and developing autonomous vehicles varies depending on the application. For example, autonomous cars can assist commuters and drivers. Autonomous trucks reduce costs associated with transporting goods. Autonomous heavy equipment helps reduce costs and the need for humans to work in hazardous conditions.

[0003] Operating vehicles, such as excavators, in real-world environments can be challenging due to extreme conditions. The complexity of operating heavy equipment, combined with hazardous environments, leads to numerous fatal accidents each year. Safety is one of the primary requirements on construction sites. With advancements in deep learning and computer vision technologies, autonomous vehicle systems have been researched and have made substantial progress. However, systems and methods for safely operating heavy equipment and monitoring its performance still fall short of capturing the complexity of these tasks.

[0004] Therefore, there is a need for systems and methods for monitoring and analyzing object detection (e.g., devices such as autonomous vehicles, humans, etc.). Summary of the Invention

[0005] This document presents systems, methods, and computer-readable media, including instructions for embodiments of vision-based object perception for activity analysis, security monitoring, or both. Embodiments of the perception system are capable of detecting multiple object classes (e.g., construction machines and humans) in real time while estimating pose and motion. Embodiments of novel security monitoring and object activity analysis subsystems based on perception results are also proposed. To evaluate the performance of some embodiments, datasets were collected using an Autonomous Excavator System (AES), including multiple object classes including humans under different lighting conditions. Evaluation results of the embodiments show that the object detection model improves inference speed and accuracy while reducing model size. Although an excavator is used as an illustration, embodiments of the real-time security monitoring system and real-time activity / productivity analysis system are not limited to this equipment or environment (e.g., a solid waste scenario). The embodiments can be applied to other scenarios.

[0006] In one or more embodiments, a system for analyzing a working area may include one or more processors and one or more non-transitory computer-readable media including one or more instruction sets that, when executed by at least one of the one or more processors, cause execution steps. In one or more embodiments, the steps may include: segmenting the working area into one or more defined regions using a working area segmentation neural network subsystem, the working area segmentation neural network subsystem receiving image data from at least one camera and segmenting the working area into one or more defined sub-regions; detecting one or more objects in the working area using the image data from the at least one camera and an object detection neural network subsystem receiving the image data, to generate a classification of the detected objects and boundary region data of the detected objects for each detected object from a set of one or more detected objects from the image data; and detecting security issues using a security monitoring subsystem, which can be detected in real time. In one or more embodiments, the security monitoring subsystem may receive one or more defined sub-regions of the working area from the working area segmentation neural network subsystem, and for each detected object from a set of one or more detected objects, receive its boundary region data from the object detection neural network subsystem, determine whether a security issue exists based on the one or more defined sub-regions and boundary region data of the working area from the object detection neural network subsystem and one or more models, and respond to the existing security issue by issuing an alert.

[0007] It should be noted that the detected object can be a device (which can be autonomous) or other objects, such as a person.

[0008] In one or more embodiments, for a detected device, the system can use an action recognition subsystem to identify one or more action states of the device. The action recognition subsystem uses one or more models based on image data from at least one of one or more cameras to identify one or more action states of the device during the duration of the image data. In one or more embodiments, the action recognition subsystem may include a rule-based model, a deep learning-based model, or both.

[0009] In one or more embodiments that include an action recognition subsystem, the security monitoring subsystem may be further configured to receive a set of one or more action states and use one or more models to detect security issues by detecting abnormal action states or abnormal sequences of action states.

[0010] In one or more embodiments, the system can use boundary region data of the detected object and corresponding image data to generate a set of cropped images of the detected object, and use the set of cropped images of the detected object in an action neural network model to identify one or more action states of the detected object in the set of cropped images.

[0011] In one or more embodiments, the action recognition subsystem may include a set of rules for recognizing one or more action states of a detected object using a set of key points from a set of images in an image dataset, wherein the set of key points is obtained from a pose estimation subsystem that uses image data from an object detection neural network subsystem and boundary region data of the detected object to recognize key points of the detected object in the set of images.

[0012] In one or more embodiments, the productivity analysis subsystem may receive one or more action states from the action recognition subsystem for detected objects, and may determine the productivity of the detected objects based on a set of parameters including one or more object-related parameters.

[0013] In one or more embodiments, the system can refine boundary region data using a set of key points of the detected object, wherein the set of key points of the detected object is obtained from a pose estimation subsystem that identifies a set of key points using image data from an object detection neural network subsystem and boundary region data of the detected object. Furthermore, in one or more embodiments, the security monitoring subsystem can use the refined boundary region data as boundary region data of the detected object when determining a security issue.

[0014] In one or more embodiments, the security monitoring subsystem may determine a security issue by performing at least one step including the following steps: using boundary region data of a first detected object and boundary region data of a second detected object, to monitor security by determining whether the boundary region data of the first detected object is within a threshold of the boundary region data of the second detected object; using the boundary region data of the first detected object and at least one of one or more defined sub-regions, to monitor security by determining whether a threshold portion of the boundary region data of the first detected object is within one of the defined sub-regions; using the boundary region data of the first detected object, the boundary region data of the second detected object, and at least one of one or more defined sub-regions, to monitor security by determining whether a first threshold portion of the boundary region data of the first detected object and a second threshold portion of the boundary region data of the second detected object are within the same defined sub-region; and / or in response to detecting an abnormal key point orientation, to determine a security issue using at least some of a set of key points of the detected object.

[0015] In one or more embodiments, a system for analyzing a working area may include one or more processors, and one or more non-transitory computer-readable media including one or more instruction sets that, when executed by at least one of the one or more processors, cause execution steps. In one or more embodiments, the steps may include: segmenting the working area into one or more defined regions using a working area segmentation neural network subsystem, the working area segmentation neural network subsystem receiving image data from at least one camera and segmenting the working area into one or more defined sub-regions; detecting one or more objects in the working area using image data from at least one camera and an object detection neural network subsystem receiving the image data, to generate a classification of the detected object and boundary region data of the detected object for each detected object from a set of one or more detected objects in the image data; for the detected objects, identifying a set of one or more action states of the detected objects using an action recognition subsystem, the action recognition subsystem using one or more models based on image data from at least one camera to identify a set of one or more action states of the device during the duration of the image data; and using a productivity analysis subsystem, for the detected objects, receiving one or more action states from the action recognition subsystem and determining the productivity of the detected objects based on a set of parameters including one or more object-related parameters.

[0016] It should be noted that embodiments may be implemented as a system, a computer-implemented method, or instructions encoded on one or more non-transitory computer-readable media, which, when executed by at least one of one or more processors, cause the method to be performed.

[0017] Some features and advantages of embodiments of the present invention have been generally described in the Summary Section; however, additional features, advantages, and embodiments are presented herein, or will be apparent to those skilled in the art from the accompanying drawings, description, and claims. Therefore, it should be understood that the scope of the invention should not be limited to the specific embodiments disclosed in the Summary Section. Attached Figure Description

[0018] Reference will be made to embodiments of this disclosure, examples of which may be illustrated in the accompanying drawings. These drawings are intended to be illustrative and not restrictive. Although this disclosure has been generally described in the context of these embodiments, it should be understood that it is not intended to limit the scope of this disclosure to these particular embodiments. Items in the drawings may not be to scale.

[0019] Figure 1 A pipeline for object activity analysis and security monitoring system according to embodiments of the present disclosure is described.

[0020] Figure 2 A security monitoring and activity analysis method according to embodiments of the present disclosure is described.

[0021] Figure 3-7 A security monitoring system architecture according to embodiments of the present disclosure is described.

[0022] Figure 8 This is an activity analysis system according to embodiments of the present disclosure.

[0023] Figure 9 An example architecture for object detection according to embodiments of the present disclosure is described.

[0024] Figure 10 The Region Proposal Network (RPN) according to embodiments of the present disclosure is illustrated in the diagram.

[0025] Figure 11 An example network structure for pose estimation according to an embodiment of the present disclosure is illustrated.

[0026] Figure 12 The excavator and 10 parts of the excavator according to embodiments of the present disclosure are depicted, including 2 bucket end key points (bucket end 1, bucket end 2), bucket joint, boom joint, boom cylinder, boom base and 4 body key points (body 1, body 2, body 3, body 4).

[0027] Figure 13 An example region segmentation is depicted within a sub-region defined according to an embodiment of this disclosure.

[0028] Figure 14 An exemplary autonomous excavator work cycle according to embodiments of the present disclosure is depicted.

[0029] Figure 15 A network for action recognition according to embodiments of the present disclosure is described.

[0030] Figure 16 An autonomous excavator and loader, which may have potential safety issues according to embodiments of this disclosure, are depicted.

[0031] Figure 17 A workflow for determining cycle time according to embodiments of the present disclosure is described.

[0032] Figure 18 An alternative workflow for determining cycle time according to embodiments of the present disclosure is described.

[0033] Figure 19 Example excavator and loader inspection results according to embodiments of the present disclosure are depicted.

[0034] Figure 20A and 20B A comparison of detection results according to embodiments of the present disclosure is depicted.

[0035] Figure 21 An example excavator posture estimation result according to an embodiment of the present disclosure is depicted.

[0036] Figure 22 An example of long video motion detection results of an excavator according to an embodiment of the present disclosure is depicted.

[0037] Figure 23 A long video demonstration depicting motion recognition results in different architectural scenes according to embodiments of the present disclosure is provided.

[0038] Figure 24 A simplified block diagram of a computing device / information processing system according to embodiments of the present disclosure is depicted. Detailed Implementation

[0039] In the following description, specific details are set forth for purposes of explanation in order to provide an understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these details. Furthermore, those skilled in the art will recognize that the embodiments of this disclosure described below can be implemented in various ways, such as as processes, apparatuses, systems, devices, or methods on tangible computer-readable media.

[0040] The components or modules shown in the figures are illustrative of exemplary embodiments of this disclosure and are intended to avoid obscuring this disclosure. It should be understood that throughout the discussion, components can be described as individual functional units, which may include subunits; however, those skilled in the art will recognize that various components or portions thereof may be divided into individual components or may be integrated together, including, for example, in a single system or component. It should be noted that the functions or operations discussed herein can be implemented as components. Components can be implemented in software, hardware, or a combination thereof.

[0041] Furthermore, the connections between components or systems shown in the diagram are not intended to be limited to direct connections. Rather, data between these components can be modified, reformatted, or otherwise altered by intermediate components. Additionally, more or fewer connections may be used. It should also be noted that the terms “coupled,” “connection,” “communication coupling,” “interface,” “access,” or any derivative thereof should be understood to include direct connections, indirect connections via one or more intermediate devices, and wireless connections. It should also be noted that any communication, such as signals, responses, replies, acknowledgments, messages, queries, etc., may include one or more information exchanges.

[0042] In this specification, references to "one or more embodiments," "preferred embodiments," "an embodiment," or simply "embodiment" mean that a particular feature, structure, characteristic, or function described in connection with an embodiment is included in at least one embodiment of the invention and may be included in more than one embodiment. Furthermore, the above phrases appearing in various places in this specification do not necessarily refer to the same one or more embodiments.

[0043] The use of certain terms in different places in this specification is for illustrative purposes and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; the use of these terms may refer to a group of related services, functions, or resources that may be distributed or aggregated. The terms “including,” “contains,” “has,” or any variations thereof should be understood as open-ended terms, and any subsequent list of items is illustrative and does not imply limitation to the listed items. A “layer” may include one or more operations. The terms “optimal,” “optimized,” etc., refer to an improvement in a result or process and do not require that the specified result or process has reached an “optimal” or peak state. The use of memory, database, repository, data storage, table, hardware, cache, etc., in this document may refer to system components or parts that can input or otherwise record information.

[0044] In one or more embodiments, the stopping conditions may include: (1) a set number of iterations have been performed; (2) a certain processing time has been reached; (3) convergence (e.g., the difference between successive iterations is less than a first threshold); (4) divergence (e.g., performance degradation); (5) an acceptable result has been achieved; and (6) all data has been processed.

[0045] Those skilled in the art should recognize that: (1) certain steps may be selectively performed; (2) the steps may not be limited to the specific order specified herein; (3) certain steps may be performed in different orders; and (4) certain steps may be performed simultaneously.

[0046] Any headings used herein are for organizational purposes only and should not be used to limit the scope of the description or claims. Every reference / document mentioned in this patent document is incorporated herein by reference in its entirety.

[0047] It should be noted that any experiments and results provided herein are provided in an illustrative manner and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of disclosure of this patent document.

[0048] It should also be noted that although the embodiments described herein may be applicable to excavators or heavy equipment, the aspects of this disclosure are not limited thereto. Therefore, the aspects of this disclosure may be applied to or adapted for use with other machines or objects, and may be applied to or adapted for use in other situations.

[0049] A. General Introduction

[0050] With recent advancements in deep learning and computer vision, artificial intelligence (AI)-driven construction machinery, such as Autonomous Excavator Systems (AES), has made significant progress. In AES systems, excavators are assigned to load waste disposal materials into designated areas. However, safety remains one of the most critical aspects of modern construction, especially as construction machinery becomes increasingly automated.

[0051] Because it may be desirable to operate such systems 24 hours a day without any human intervention, one of the main concerns is safety, where the vehicle may collide with the environment or other machinery or vehicles. Therefore, the embodiments described herein address the safety issues related to excavators potentially colliding with the environment or other machinery or objects. The embodiments described herein include a camera-based safety monitoring system that detects excavator posture, the surrounding environment, and other construction machinery, and warns of any potential collisions. Furthermore, based on motion recognition of human activity, the embodiments include identifying excavator movements, which can also be used as part of an excavator productivity analysis system to analyze excavator activity. It should be noted that although the embodiments are discussed in the context of AES, they can also be generally applied to manned excavators and other vehicles or machinery.

[0052] As part of building a safety monitoring system for excavators, a perception system for the surrounding environment was developed. Implementations of the perception system include detection of the construction machine, posture estimation, and activity recognition. Real-time detection of the excavator's posture is a crucial feature for notifying workers and enabling autonomous operation. Vision-based (e.g., label-free and label-based) and sensor-based (e.g., inertial measurement unit (IMU) and ultra-wideband (UWB) sensor-based localization solutions) are two main methods for estimating robot posture. Label-based and sensor-based methods require some additional pre-installed sensors or labels, while label-free methods typically only require a field camera system, which is common on modern construction sites. Therefore, the embodiments in this paper employ a label-free method and utilize camera video input, leveraging existing deep learning methods.

[0053] This patent document proposes an embodiment of a deep learning-based excavator activity analysis and safety monitoring system, which can detect the surrounding environment, estimate posture, and identify excavator movements. Some contributions of this patent document include, but are not limited to, the following:

[0054] 1) An excavator dataset with truth annotations was created.

[0055] 2) An embodiment of a deep learning-based perception system for multi-target detection, pose estimation, and action recognition of construction machinery on construction sites was developed. Furthermore, the embodiment was tested against state-of-the-art (SOTA) systems, and the tested embodiment achieved SOTA results on an autonomous excavator system dataset and a benchmark construction dataset are shown in this paper.

[0056] 3) This paper also proposes an embodiment of a novel excavator safety monitoring and productivity analysis system based on the above-mentioned perception system.

[0057] B. Some related studies

[0058] This paper reviews some previous research related to security and productivity analytics. Some areas of interest include fundamental tasks in computer vision useful for activity analytics and security monitoring systems, including object detection, image segmentation, pose estimation, and action recognition. Vision-based activity analytics and security monitoring systems are also reviewed.

[0059] 1. Object detection

[0060] The first category is object detection. Recently, some researchers have used a region-based convolutional neural network (CNN) framework called Faster R-CNN (Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection With Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2016 (also available at arXiv:1506.01497v3, which is incorporated herein by reference in its entirety) to detect workers standing on scaffolding. A deep CNN then classifies whether the workers are wearing safety harnesses. Those who are not properly harnessed are identified to prevent falls from heights.

[0061] 2. Image segmentation

[0062] Others have used Mask R-CNN (H. Raoofi and A. Modamedi, Mask R-CNN Deep Learning-based Approach to Detect Construction Machinery on Jobsites, 37th International Symposium on Automation and Robotics in Construction (ISARC2020), Kitakyushu, Japan, October 2020, the entire contents of which are incorporated herein by reference) to detect construction machinery on job sites. More importantly, segmentation networks like Mask R-CNN can be used to determine areas such as excavation and dumping.

[0063] 3. Posture estimation

[0064] The second group of techniques is skeleton pose estimation. Pose estimation has been studied based on human pose estimation networks such as OpenPose. Soltani et al. (Mohammad Mostafa Soltani, Zhenhua Zhu and Amin Hammad. Skeleton Estimation Of Excavator By Detecting Its Parts. Automation in Construction, 82:1–15, 2017, the full text of which is incorporated herein by reference) proposed a method for estimating the skeleton parts of excavators.

[0065] 4. Action recognition

[0066] Learning-based action recognition methods have been proposed. For example, Feichtenhofer et al. (Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast Networks For Video Recognition. Proceedings of The IEEE / CVF International Conference On Computer Vision, pp. 6202-6211, 2019, the entire contents of which are incorporated herein by reference) proposed the SlowFast Network for video recognition. The model involves operating at a low frame rate to capture slow paths of spatial semantics, and operating at a high frame rate to capture fast paths of motion with fine temporal resolution. Others have proposed a non-convolutional video classification method that is entirely based on spatial and temporal self-attention.

[0067] 5. Activity analysis and safety monitoring

[0068] This paper briefly reviews recent vision-based methods for activity analysis and safety monitoring in the construction field. For example, some researchers have combined CNNs with Long Short-Term Memory (LSTM) to identify unsafe worker behaviors such as climbing ladders with objects in hand, facing backward, or reaching too far. While effectively identifying worker safety hazards, their methods only capture individual workers and do not consider multi-object analysis. On the other hand, Soltani et al. (cited above) used background subtraction to estimate excavator posture by detecting the three skeletal parts of the excavator separately, including the bucket, boom, and body. While knowing the operating status of construction equipment allows for close-range safety monitoring, the impact of the equipment on surrounding objects is not investigated. Others have proposed frameworks to automatically identify activities and analyze the productivity of multiple excavators. Still others have proposed methods to monitor and analyze interactions between workers and equipment by detecting their positions and trajectories and identifying hazardous areas using computer vision and deep learning techniques. However, their models do not consider the excavator's state. Some have proposed benchmark datasets; however, their action recognition models have lower accuracy compared to the deep learning-based implementations presented in this paper.

[0069] In general, previous research on using computer vision technology for activity analysis and security monitoring has mainly focused on different aspects, such as identifying the working status of construction equipment or estimating the pose of excavators. The embodiments in this paper demonstrate the advantages of state-of-the-art deep learning models for detection, pose estimation, and action recognition tasks.

[0070] C. Framework Implementation

[0071] Figure 1 This document illustrates an implementation framework for object activity recognition, safety monitoring, and productivity analysis. In one or more embodiments, the framework may include six main modules: an object detection subsystem 120, an excavator posture estimation subsystem 115, a work area segmentation subsystem 110, an activity recognition subsystem 125, a safety monitoring subsystem 130, and an activity / productivity analysis subsystem 135. The input to the system embodiment may be camera video 105 from one or more cameras. The embodiment is capable of detecting multiple types of objects (e.g., construction machines, people, etc.) in real time. It should be noted that various embodiments may include fewer and / or different subsystems—example embodiments are provided here.

[0072] Figure 2An example general method according to embodiments of the present disclosure is described. In one or more embodiments, a work area is first segmented (205) into one or more defined sub-regions (e.g., excavation and dumping areas) using a work area segmentation subsystem 110. One or more detection methods 120 of an object detection neural network subsystem (210) can be used to identify objects (e.g., equipment, people, etc.) in the video frame. In one or more embodiments, detection may also include classifying any detected objects—such as classifying the equipment type of a machine in the video frame, tagging people, etc. An excavator is used herein to aid in providing an example only for illustrative purposes; however, it should be noted that other equipment and non-equipment (e.g., people, animals, structures, etc.) may also be detected and classified. In one or more embodiments, objects may be identified or their locations may be further refined (215) using a pose estimation subsystem 115 through pose estimation and detection-based tracking.

[0073] In one or more embodiments, one or more models of the motion estimation subsystem 125 may also be used to identify (220) one or more motion states of the tracked object. For example, one or more keypoint models may be used in conjunction with a set of rules to define motion states and / or a motion neural network model may be used to determine motion states. In one or more embodiments, the security monitoring subsystem 130 may use motion states to help identify security issues. For example, if a device or a person is about to begin an unsafe action, an alarm may be issued (e.g., triggering one or more signals, such as lights and sounds, sending one or more messages to an operator or administrator, sending a command signal to the device to take some action or not to take action, etc.). For example, for autonomous devices, an alarm may be a signal to the device to stop its action. Additionally or alternatively, an alarm may be issued if abnormal action is detected. In one or more embodiments, the activity / productivity analysis subsystem 1135 may use motion states.

[0074] As explained in more detail with respect to the embodiments discussed below, the safety monitoring subsystem 130 may receive input from a variety of other subsystems to monitor (225) site safety, depending on the embodiment, based on object detection (e.g., their location), based on key points, based on activity identification results, or a combination thereof.

[0075] In one or more embodiments, a productivity analysis subsystem 135 may be used to determine the productivity of an object (230), the productivity analysis subsystem 135 receiving at least some action states from an action recognition subsystem for the detected object and determining the productivity of the detected object based on a set of parameters associated with the detected object.

[0076] It should be noted that different implementations can be formed. Note that, according to the implementation, security monitoring can be performed, activity / productivity analysis can be performed, or both can be performed. Figure 3-7 Various security monitoring systems according to various embodiments of the present disclosure are described. Figure 8 An example action or productivity analysis system according to embodiments of the present disclosure is described.

[0077] Detailed information about embodiments of each module in the framework is provided in the following sections.

[0078] 1. Object Detection Example

[0079] In one or more embodiments, the detection of building equipment can be achieved based on a Faster R-CNN network (Ren et al., 2016, cited above). Figure 9 An example architecture 900 according to an embodiment of the present disclosure is depicted. The architecture of Faster R-CNN includes (1) a backbone-convolutional network 910 for extracting image features 915; (2) a region proposal generation (RPN) network 920 for generating regions of interest (ROIs) 925; and (3) a classification network 935 for generating class scores and bounding boxes for objects.

[0080] To remove duplicate bounding boxes, Soft-NMS (non-maximum suppression) (e.g., Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-NMS—Improving Object Detection With One Line Of Code. Proceedings of the IEEE International Conference On Computer Vision, pp. 5561–5569, 2017, incorporated herein by reference in its entirety) can be used to limit the maximum bounding box of each object to 1.

[0081] Figure 5 The Region Proposal Network (RPN) according to an embodiment of this disclosure is illustrated graphically. The RPN acts as the "attention" of the Unified Faster R-CNN network. The RPN takes an image as input and outputs a set of rectangular object proposals, each with an objectivity score. To generate region proposals, a small network slides over the convolutional feature map output from the last shared convolutional layer. This small network takes an n×n spatial window of the input convolutional feature map as input, each sliding window being mapped to a low-dimensional feature. The features are fed into two sibling fully connected layers—a box regression layer (reg) and a box classification layer (cls), which can be 1×1 convolutional layers and... Figure 5 As shown in the image.

[0082] Note that at each sliding window position, the system can simultaneously predict multiple region proposals (up to k possible proposals per position). The `reg` layer can have 4k outputs encoding the coordinates of the k boxes, and the `cls` layer can output 2k scores to estimate the object probability of each proposal. The k proposals can be parameterized relative to k reference boxes, which can be called anchors. Anchors can be centered on the sliding window in question and can be associated with scale and aspect ratio. If there are 3 scales and 3 aspect ratios, then there are k = 9 anchors at each sliding position.

[0083] In one or more embodiments, the workflow of the Faster R-CNN model can be described as follows. Step 1 includes pre-training the CNN network on an image classification task.

[0084] Step 2 involves end-to-end fine-tuning of the Region Proposal Network (RPN) for the region proposal task, which can be initialized using a pre-trained image classifier. In one or more embodiments, the intersection-over-union (IoU) ratio for positive samples is >0.7, while that for negative samples is <0.3. Small n×n spatial windows can slide across the convolutional feature map of the entire image. At the center of each sliding window, multiple regions of various scales and ratios are predicted simultaneously. For example, anchors can be defined as combinations of (sliding window center, scale, ratio). For example, 3 scales × 3 ratios result in k = 9 anchors at each sliding position; however, it should be noted that different anchor values ​​can be used.

[0085] Step 3 involves training the Fast R-CNN object detection model using proposals generated by the current RPN.

[0086] Step 4 involves initializing RPN training using a Fast R-CNN network. In one or more embodiments, the shared convolutional layers may be fixed, while RPN-specific layers are fine-tuned. Note that at this stage, the RPN and the detection network share convolutional layers.

[0087] Step 5 involves fine-tuning the unique layers of Fast R-CNN. In one or more embodiments, steps 4 and 5 can be repeated to train RPN and Fast R-CNN alternately.

[0088] In one or more embodiments, Faster R-CNN can be optimized for a multi-task loss function. This multi-task loss function can combine the losses from classification and bounding box / boundary region regression.

[0089]

[0090] in It is a log loss function for two classes because by predicting whether a sample is the target object or not, multi-class classification can be easily converted into binary classification. i This represents the predicted probability that anchor i is an object. A binary truth label indicating whether anchor i is an object. i The four parametric coordinates representing the prediction, and Represents the true coordinates. N cls This represents the normalization term, which can be set to the mini-batch size (~256, although different values ​​can be used). N box The normalization term can be set to the number of anchor positions (~2400, although different values ​​can be used). Finally, λ represents the balance parameter, set to ~10 (although different values ​​can be used) so that L cls and L box The items have roughly equal weights. Indicates smoothed L1 loss:

[0091]

[0092] Alternatively, other models can be used instead of the Faster R-CNN model to detect building equipment. For example, YOLOv3 (Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement, 2018, available at arxiv.org / abs / 1804.02767, which is incorporated herein by reference in its entirety). YOLOv3 is an extremely fast, single-stage, state-of-the-art detector. The system architecture can be as follows:

[0093]

[0094] Alternatively, YOLOv5 (Glenn Jocher et al., Ultralytics / YOLOv5, version 3.1 available at zenodo.org / record / 4154370, version 6.1 available at zenodo.org / record / 6222936, the entire contents of which are incorporated herein by reference) can be used. YOLOv5 uses a Cross-Stage Partial Network (CSPNet) as its backbone. In one or more embodiments, to improve inference speed, reduce model size, and further improve detection accuracy, embodiments are based on YOLOv5 to achieve real-time detection of objects (e.g., construction machines and people). YOLOv5 has models of different sizes, including YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. Typically, YOLOv5 uses the architecture of CSPDarknet53, with SPP (Spatial Pyramid Pooling) layers as the backbone, PANet as the neck, and YOLO detection as the head. To further optimize the entire architecture, freebies and special offers are provided (e.g., see A. Bochkovskiy et al., “YOLOv4: Optimal speed and accuracy of object detection,” available in arXiv:2004.10934 (2020), cited in full here). Because it is the most significant and convenient single-stage detector, it is chosen as the benchmark for this embodiment. To improve human detection accuracy across all landscapes, the embodiment involves fine-tuning a pre-trained YOLOv5 model on a building dataset.

[0095] 2. Example of posture estimation

[0096] In one or more embodiments, the pose estimation subsystem may be based on the output bounding boxes from the detection. In one or more embodiments, the system for pose estimation may employ ResNet, one of the most common backbone networks used for image feature extraction and pose estimation. In one or more embodiments, a deconvolutional layer may be added on the last convolutional stage in the ResNet. Figure 11An example network structure 1100 according to an embodiment of the present disclosure is depicted graphically. In one or more embodiments, a system such as that described in Bin Xiao, Haiping Wu, and Yichen Wei, Simple Baselines for Human Pose Estimation and Tracking. Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 466-481 (hereinafter referred to as Xiao et al.) (incorporated herein by reference in its entirety).

[0097] In one or more embodiments, a labeling method can be used for objects. For example, a labeling method has been designed for a tracked excavator, using 10 key points. The key points for labeling excavator parts are as follows: Figure 12 As shown in the diagram. These 10 key points include: 2 bucket end key points, bucket joint, boom joint, boom cylinder, boom base, and 4 body key points. Unlike other posture marking methods that mark the bucket / excavator body as the midpoint, one or more embodiments of this paper mark corner points to improve accuracy. Other labels can be set for other objects, such as different equipment, people, etc.

[0098] 3. Example of work area segmentation

[0099] In one or more embodiments, image segmentation is used to determine one or more sub-regions or partitions, such as excavation and dumping areas. For example, Figure 13 The construction site, already segmented based on image segmentation, is depicted. As shown, region 1315 within the dashed line is the dumping area, and region 1310 within the solid line is the excavation area. In one or more embodiments, one or more additional regions or objects may also be identified. For example, excavator 1305 may be identified and segmented into a non-excavation and non-dumping area, i.e., region 1320 (or may be segmented because it is not any other identified region).

[0100] In one or more embodiments, the segmentation network may be a ResNet network or may be based on a ResNet network (Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016, the entire contents of which are incorporated herein by reference). The excavation area may be defined as a waste recycling area, which includes various toxic substances, and the dumping area may be a designated area for dumping waste.

[0101] 4. Action Recognition Example

[0102] One or more actions can be defined for each object, although some objects (such as structures) may not have actions defined for them. For illustration, consider an excavator, an example of which has been used here. In one or more embodiments, three actions can be defined for an excavator: (1) digging, (2) swinging, and (3) dumping. An autonomous excavator may have four defined states: (1) digging state, (2) post-digging swinging state, (3) dumping state, and (4) digging swinging state. More precisely, digging refers to loading the excavator bucket with the target material; post-digging swinging refers to swinging the excavator bucket to the dumping area; dumping refers to unloading material from the bucket to the dumping area; and digging swinging refers to swinging the bucket to the working area. In one or more embodiments, an optional idle state may also exist, for example, when the excavator is in manned mode, fault state, or other modes.

[0103] Figure 14 The working cycle of an excavator according to embodiments of the present disclosure is described. For example... Figure 14 As shown, the excavator can be in one of several states, including: digging 1405, digging and swinging 1410, tilting 1415, idle 1420, and digging and swinging 1425. It should be noted that different states can be defined and can be configured or arranged differently.

[0104] In one or more embodiments, action recognition can be implemented as a rule-based model, a deep learning-based model, or both. Furthermore, in one or more embodiments, keypoint and / or pose information can be used to help refine the boundary regions of an object. That is, given a set of keypoints, given a pose, or given keypoints and a pose, the pose estimation model can refine the boundary regions from object detection to provide more accurate or finer-grained boundary regions.

[0105] a) Rule-based model implementation

[0106] In one or more embodiments, to determine the motion state, the position of the object is determined based on key points obtained from pose estimation and image segmentation results. The pose can then be defined using consecutive frames of pose key points and one or more rules associated with those key points. For example, examining pose key point frames for vehicle bodies 1, 2, 3, and 4 can be used to determine whether the excavator is in a swinging state. A threshold for key point motion can be set: if the average value of each pose key point of vehicle bodies 1-4 is less than a set value, the excavator body can be considered stationary. Otherwise, the excavator body can be considered not stationary (i.e., the excavator body is in motion). The pose from the rule-based model can be used in the safety monitoring subsystem.

[0107] In one or more embodiments, the excavator's operating state can be defined as follows:

[0108] 1. Excavation status: The bucket / arm joint and key points 1-4 on the body of the excavator in the excavation area are fixed points (the excavator body is stationary).

[0109] 2. Swing State: The bucket / arm joint in the working area and one or more of the key points on the excavator body 1-4 are non-fixed points (the excavator body is not stationary). If this condition is met, the excavator is determined to be in either the "digging swing" state or the "digging post-swing" state based on the previous state. If the previous state was "tipping", the excavator will be in the "digging swing" state; otherwise, it will be in the "digging post-swing" state.

[0110] 3. Tilting state: The bucket / arm joint in the tilting area and key points 1-4 on the body are fixed points (excavator body is stationary).

[0111] 4. Idle state: The bucket / arm joint and bucket / arm joint / body 1-4 in the dumping area are fixed points (the excavator arm and body are stationary).

[0112] Those skilled in the art will recognize that other rule-based models can be set for other detected objects.

[0113] b) Neural Network-Based Model Implementation Examples

[0114] In one or more embodiments, a more general deep learning-based action recognition method, such as a SlowFast-based method (Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He, SlowFast Networks For Video Recognition, in Proceedings of the IEEE / CVF International Conference On Computer Vision, pp. 6202–6211, 2019, the entire contents of which are incorporated herein by reference) can be used for action recognition. Figure 15 As shown, in one or more embodiments, model 1500 includes (i) a slow path 1505, operating at a low frame rate to capture spatial semantics, and (ii) a fast path 1510, operating at a high frame rate to capture motion with fine temporal resolution. ResNet-50 can be used as the backbone. The fast path 1510 can be made very lightweight by reducing its channel capacity while learning useful temporal information for video recognition.

[0115] In one or more embodiments, the deep learning version of the pose estimation subsystem may include one or more neural network models that identify keypoints, poses, poses using keypoints, and / or actions (whether based on keypoints, poses, poses using keypoints, or directly determining actions).

[0116] Implementations of deep learning action recognition models can be used in conjunction with a security monitoring subsystem, a productivity analysis subsystem, or both.

[0117] 5. Safety Monitor – Example of Detecting Potential Construction Machinery Collisions

[0118] Safety issues are always a possibility at busy construction sites. The use of autonomous vehicles or other autonomous equipment exacerbates this problem. In solid waste recycling scenarios, excavators are often used in conjunction with other equipment such as loaders. For example, an excavator digs up waste and dumps it into a designated dumping area. When the excavation area is empty, a loader loads and dumps the waste into the excavation area. For example, as... Figure 16 As shown, the autonomous excavator 1605 and the loader 1610 may have a potential collision 1615. Therefore, detecting a potential collision is important because, from the loader's perspective, the loader may not know the current state of the excavator. A danger signal 1620 can be sent when both the autonomous excavator and the loader are detected operating in the excavation area or potentially in an overlapping area.

[0119] In one or more embodiments, object detection information can be used to determine hazards. For example, detection of excavator 1605 may include a boundary region 1625, which may be defined as a typical working area containing the detected object. Similarly, a loader has a boundary region 1630. If more than one machine (e.g., excavator 1615 or the working area) is detected in the same area, an alert may be issued to one or more users, or one or more autonomous vehicles may be suspended until the problem is resolved.

[0120] Some additional security rules may include (as examples and not limitations): (1) using boundary region data of a first object and boundary region data of a second object to monitor security by determining whether the boundary region data of the first object is within a threshold distance of the boundary region data of the second object; (2) using boundary region data of the first object and at least one of one or more defined sub-regions to monitor security by determining whether a threshold portion of the boundary region data of the first object is within one of the defined sub-regions (e.g., a person should never be in region X); (3) using boundary region data of the first object, boundary region data of the second object, and at least one of one or more defined sub-regions to monitor security by determining whether a first threshold portion of the boundary region data of the first object and a second threshold portion of the boundary region data of the second object are within the same defined sub-region; (4) whether the detected object is in an abnormal posture; and (5) whether the detected object is in an abnormal motion state or an abnormal motion state sequence. Those skilled in the art will recognize that many different security rules can be set and can be adapted according to embodiments (e.g., Figure 1 and Figure 3-7 The embodiments in the example (as an example) employ various inputs.

[0121] 6. Productivity Analysis Example

[0122] In one or more embodiments, the productivity of a detected object (e.g., a device or a person) can be based on activity recognition results. For an object, the productivity analysis subsystem can receive at least some action states from the action recognition subsystem and can determine the object's productivity based on a set of parameters including one or more object-related parameters and action states.

[0123] For example, an excavator's productivity can be calculated using cycle time, bucket volume, and average bucket full rate, as shown in Formula 1. Since the bucket volume is provided by the manufacturer, the goal of productivity calculation becomes determining the excavator's cycle time. To simplify the procedure, the two types of oscillations (post-digging oscillation and digging oscillation) can be disregarded.

[0124]

[0125] The time for each cycle is based on... Figure 17 The workflow is measured. In one or more embodiments, the action recognition module labels each video frame of the excavator with action tags. The action tags of two consecutive frames can be compared. If they are the same, it means the action remains unchanged. Therefore, the cumulative time of the current action increases by 1 / FPS (frames per second). If the tags are different, it means a new action has started, and the time of the newly identified activity has started. The time of the newly identified activity will increase by 1 / FPS. In one or more embodiments, the total time of a cycle can be defined as the difference between the start times of two adjacent excavation actions.

[0126] Figure 18 An alternative workflow for determining cycle time according to embodiments of this disclosure is described. Although not described in Figure 18 The document describes a process where, if the last decision is "no," the process can be the same as the process for the first "no" decision.

[0127] D. Experimental Results

[0128] It should be noted that these experiments and results are provided for illustrative purposes and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of disclosure of this patent document.

[0129] 1. Dataset

[0130] The excavator dataset was collected from an autonomous excavator system (AES) at a waste disposal and recycling site (Liangjun Zhang, Jinxin Zhao, Pinxin Long, Liyang Wang, Lingfeng Qian, Feixiang Lu, Xibin Song, and Dinesh Manocha, An autonomous excavator system for material loading tasks, Science Robotics, 6(55), 2021, the full text of which is incorporated herein by reference). The dataset consists of 10 hours of video, containing 9 object classes (excavator, loader, person, truck, crane, cone, hook, vehicle, shovel) across 5 data scenes (AES-line1, AES-line2, bird's-eye view construction site, crane construction site, and cone dataset). The dataset contains 6692 images with object detection bounding boxes, 601 images with excavator poses, and background segmentation.

[0131] 80% of the images were used for model training, and 20% for model validation and testing. 102 excavator video clips were tagged with three actions (digging, dumping, or swinging). These videos were captured at a resolution of 1920×1080 and shot at 25 frames per second.

[0132] The implementation was also tested against a benchmark building dataset, which included approximately 480 interactive videos of excavators and dump trucks performing earthmoving operations, with annotations for object detection, object tracking, and motion. These videos were captured at a resolution of 480×720 and filmed at 25 frames per second.

[0133] 2. Assessment

[0134] a) Target detection and evaluation

[0135] The evaluation metrics are based on object detection, segmentation, and keypoint detection datasets. Mean precision (AP) is used to evaluate the network's performance. Precision measures how many of the model's predictions are correct, while recall measures the extent to which the model finds all positive predictions. For a specific value of Intersection over Union (IoU), AP measures the precision / recall curve at recall values ​​(r1, r2, etc.) as the maximum precision decreases. AP can then be calculated as the area under the curve using numerical integration. The mean mean precision is the average AP across each object class. More precisely, AP can be defined as:

[0136]

[0137] b) Posture estimation and assessment

[0138] The implementation of the pose estimation evaluation matrix is ​​based on a dataset defining object keypoint similarity (OKS) and uses mean precision (AP) as the primary competing metric. OKS is calculated based on the distance between the predicted and ground truth points of an object.

[0139] c) Action recognition assessment

[0140] In one or more embodiments, the performance metric used is the mean Average Precision (mAP) for each object class, with a frame-level IoU threshold of 0.5.

[0141] 3. Accuracy

[0142] a) Accuracy of the detection model

[0143] Experiments were conducted on a Faster R-CNN model with ResNet-50-FPN and ResNet-152-FPN backbones. The tested model implementation achieved high detection accuracy for construction equipment. The mean accuracy (AP) reached 93.0% for excavators and 85.2% for loaders. The mAP of this model implementation was 90.1%, demonstrating promise for accurately detecting multiple types of construction equipment on construction sites.

[0144] The results were also compared with a YOLOv3 implementation. YOLOv3 is an extremely fast, single-stage advanced detector. In this study, the image input size was 416×416, and this method processed 20 images per second. Compared to some two-stage detectors, YOLOv3's performance is slightly lower, but its speed is much faster, which is important for real-time applications. The building detection dataset from the previous step was used to train YOLOv3, and the training process took 12 hours. YOLOv3's overall mAP on the test set was 73.2%, with an AP of 80.2% for the excavator class and 60.2% for the loader class. The results are presented in... Figure 19 The description shows an embodiment of the test capable of detecting multiple types of machines in real time.

[0145] To further improve model speed and detection accuracy (especially for humans), experiments were conducted on YOLOv5 model instances (small / medium / x large). The model was able to detect general construction sites. Results show that compared to the Faster R-CNN / YOLOv3 model instances, the YOLOv5 model instances improved inference speed by 8x (YOLOv5x-large) to 34x (YOLOv5 small). Furthermore, the accuracy of the YOLOv5 model instances improved by 0.7% to 2.7% (YOLOv5 medium, YOLOv5x-large), while the model size decreased by 3x (YOLOv5x-large) to 30x (YOLOv5 small). These results demonstrate that compared to the Faster R-CNN / YOLOv3 model instances, the YOLOv5 multi-class object detection model instances improve inference speed by 8x (YOLOv5x-large) to 34x (YOLOv5 small). Furthermore, the accuracy of the YOLOv5 model implementation improved by 2.7% (YOLOv5x-large), while the model size decreased by 63.9% (YOLOv5x-large) to 93.9% (YOLOv5 small). Detailed comparison results are shown in Table 1 below. Some YOLOv5 implementation results are... Figure 20A and 20B As shown in the image. Figure 20A From AES-line 1, and Figure 20BFrom AES-line 2. The perception system implementation is fine-tuned on the YOLOv5 pre-trained model implementations (1910 and 1915), and it is able to detect humans that are difficult to observe, compared to the Faster-RCNN model implementation (1905), which missed humans in both scenarios (1915).

[0146] Table 1. Accuracy of Construction Machinery Inspection

[0147] network Main trunk (proportion) mAP(%) Inference time (ms / frame) Model size (MB) Faster R-CNN Resnet-50-FPN 90.1 588 482 Faster R-CNN Resnet-101-FPN 92.3 588 482 YOLOv3 DarkNet-53(320) 78.0 313 492 YOLOv3 DarkNet-53(608) 75.7 344 492 YOLOv5s CSP-Darknet53(640) 88.9 9 14.9 YOLOv5m CSP-Darknet53(640) 93.0 14 42.9 YOLOv5x CSP-Darknet53(640) 95.0 39 174.2

[0148] b) Accuracy of pose estimation

[0149] SimpleBaseline (Xiao et al., cited above) was applied to an example of a pose estimation model, and the following results were obtained. Experiments were conducted on different backbone networks, including ResNet-50 and ResNet-152. Experiments with different image input sizes were also performed. Detailed comparison results are shown in Table 2. Example excavator pose estimation results are shown in... Figure 21 As shown in the image.

[0150] Table 2. Accuracy of the Pose Estimation Model Implementation Examples

[0151] network trunk Input size AP (%) SimpleBaseline Resnet-50 256×192 91.79 SimpleBaseline Resnet-50 384×288 94.19 SimpleBaseline Resnet-152 384×288 96.50

[0152] c) Accuracy of action recognition

[0153] The Slow-Fast implementation (Feichtenhofer et al., cited above) was applied to an action recognition model example, and the following results were obtained. Experiments were conducted on different networks, including SlowFast-101 and SlowFast-152. Experiments with different clip lengths were also performed. Detailed comparison results are shown in Table 3.

[0154] Table 3. Accuracy of action recognition model examples on the AES dataset and another dataset (dataset 2)

[0155]

[0156]

[0157] The prediction results for the first three actions are as follows Figure 22 As shown. Figure 22 The results of long video motion detection of an excavator according to embodiments of the present disclosure are depicted. Excavator video was input into a test system embodiment, which predicted the motion outcome in near real-time. The most probable predictions are shown in the first row. Here, the system embodiment predicts the motion as excavation with a 54% confidence level.

[0158] The tested embodiment was compared with the results of Roberts (Dominic Roberts and Mani Golparvar-Fard, End-To-End Vision-Based Detection, Tracking and Activity Analysis of Earthmoving Equipment Filmed at Ground Level, Automation in Construction, 105:102811, 2019, which is incorporated herein by reference in its entirety) on their dataset. The tested action recognition embodiment achieved approximately 5.18% higher accuracy than theirs. Some action recognition video results are available in... Figure 23 The results show that using a deep learning model on the action recognition task outperforms their Hidden Markov Model (HMM) + Gaussian Mixture Model (GMM) + Support Vector Machine (SVM) approach.

[0159] 4. Activity Analysis

[0160] An example was tested to estimate the excavator's productivity on a long video sequence containing 15 minutes of excavator operation. In the video, an XCMG 7.5-ton mini excavator (0.4 cubic meter bucket volume) completed 40 work cycles within 15 minutes. Based on manual measurement, the average bucket full-rate was 101%. Therefore, according to Formula 1, the excavation productivity was 64.64 m³. 3 The tested system implementation detected 39 work cycles in the video, with a productivity calculation accuracy of 97.5%. The test results demonstrate the feasibility of using the pipeline implementation to analyze real-world construction projects and monitor excavator operations.

[0161] 5. Additional implementation details and hardware

[0162] The detection module implementation is based on YOLOv5 and utilizes ultralytics, MMDetection, an MMSegmentation-based segmentation module, an MMPose-based pose estimation module, and an MMAction2 toolkit-based action recognition module. An NVIDIA M40 24GB GPU was used to train the network implementation. Testing was performed on a local NVIDIA 1080 GPU. An optimized implementation was then implemented on a remote solid waste site computer equipped with an Intel 9700 CPU (16GB) and an NVIDIA 1660 GPU (16GB).

[0163] 6. Training and reasoning time

[0164] YOLOv5 small, medium, and super-large model instances for detection were trained for 2, 3, and 4 hours respectively, and the pose estimation and action recognition subsystem modules were trained for 6 hours. The inference time of the YOLOv5 small detection network on an Nvidia m40 machine can reach 9 milliseconds (ms) / frame, the medium network can reach 14 ms / frame, and the super-large network can reach 39 ms / frame, as shown in Table 1 above.

[0165] In one or more embodiments, the object detection module, pose estimation module, working region segmentation module, and action recognition module are trained using supervised learning.

[0166] E. Some conclusions or observations

[0167] This patent document presents embodiments of a safety monitoring pipeline, a productivity system pipeline, and a combination of safety monitoring and productivity implementations. The embodiments are based on computer vision and incorporate deep learning techniques. In one or more embodiments, detection, pose estimation, and activity recognition modules are integrated into the system. Furthermore, a benchmark dataset is collected from an autonomous excavator system (AES), which includes multi-category objects under different lighting conditions. The embodiments are evaluated on a general construction dataset and achieve state-of-the-art results.

[0168] F. Computing System Implementation

[0169] In one or more embodiments, aspects of this patent document may be directed to, may include, or may be implemented on one or more information processing systems (or computing systems). An information processing system / computing system may include any tool or set of tools that can be used to calculate, compute, determine, classify, process, send, receive, retrieve, initiate, route, switch, store, display, communicate, detect, record, copy, process, or utilize information, intelligence, or data of any form. For example, a computing system may be or may include a personal computer (e.g., a laptop computer), a tablet computer, a mobile device (e.g., a personal digital assistant (PDA), a smartphone, a phablet, a tablet computer, etc.), a smartwatch, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and may vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, read-only memory (ROM), and / or other types of memory. Additional components of a computing system may include one or more drives (e.g., hard disk drives, solid-state drives, or both), one or more network ports for communicating with external devices, and various input and output (I / O) devices such as keyboards, mice, touchscreens, styluses, microphones, cameras, touchpads, displays, etc. The computing system may also include one or more buses operable to transmit communication between various hardware components.

[0170] Figure 24 A simplified block diagram of an information processing system (or computing system) according to embodiments of the present disclosure is depicted. It should be understood that the functionality shown by system 2400 can operate to support various embodiments of the computing system—although it should be understood that the computing system can be configured differently and include different components, including those with fewer or more components, such as… Figure 24 As depicted in the text.

[0171] like Figure 24 As shown, the computing system 2400 includes one or more CPUs 2401 that provide computing resources and control the computer. The CPU 2401 may be implemented using a microprocessor or the like, and may also include one or more graphics processing units (GPUs) 2402 and / or floating-point coprocessors for mathematical calculations. In one or more embodiments, the one or more GPUs 2402 may be incorporated into a display controller 2409, such as part of one or more graphics cards. The system 2400 may also include system memory 2419, which may include RAM, ROM, or both.

[0172] Multiple controllers and peripherals can also be provided, such as Figure 24As shown. Input controller 2403 represents an interface to various input devices 2404. The computing system 2400 may also include a storage controller 2407 for interfacing with one or more storage devices 2408, each storage device including a storage medium such as magnetic tape or a disk, or an optical medium that can be used to record instruction programs for operating systems, utilities, and applications, which may include embodiments of programs implementing various aspects of this disclosure. Storage device 2408 may also be used to store processed data or data to be processed according to this disclosure. System 2400 may also include a display controller 2409 for providing an interface to a display device 2411, which may be a cathode ray tube (CRT) display, a thin-film transistor (TFT) display, an organic light-emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. The computing system 2400 may also include one or more peripheral controllers or interfaces 2405 for one or more peripheral devices 2406. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, etc. The communication controller 2414 can interface with one or more communication devices 2415, enabling the system 2400 to connect to remote devices via any of a variety of networks, including the Internet, cloud resources (e.g., Ethernet cloud, Ethernet Fibre Channel (FCoE) / Data Center Bridge (DCB) cloud, etc.), local area network (LAN), wide area network (WAN), storage area network (SAN), or via any suitable electromagnetic carrier signal, including infrared signals. As shown in the depicted embodiment, the computing system 2400 includes one or more fans or fan trays 2418 and one or more cooling subsystem controllers 2417 that monitor the thermal temperature of the system 2400 (or its components) and operate the fans / fan trays 2418 to aid in temperature regulation.

[0173] In the illustrated system, all major system components can be connected to bus 2416, which can represent more than one physical bus. However, the various system components may or may not be physically close to each other. For example, input data and / or output data can be remotely transmitted from one physical location to another. Furthermore, programs implementing various aspects of this disclosure can be accessed from a remote location (e.g., a server) via a network. Such data and / or programs can be transmitted via any of a variety of machine-readable media, including, for example: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as optical discs (CDs) and holographic devices; magneto-optical media; and hardware devices specifically configured for storing or storing and executing program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices.

[0174] Various aspects of this disclosure may be encoded on one or more non-transitory computer-readable media having instructions for one or more processors or processing units to result in execution steps. It should be noted that the one or more non-transitory computer-readable media should include volatile and / or non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. The functionality of a hardware implementation can be implemented using an ASIC, a programmable array, digital signal processing circuitry, etc. Therefore, the term "apparatus" in any claim is intended to cover both software and hardware implementations. Similarly, the term "computer-readable medium" as used herein includes software and / or hardware having instructions thereon, or a combination thereof. In consideration of these alternative implementations, it should be understood that the accompanying drawings and description provide functional information necessary for those skilled in the art to write program code (i.e., software) and / or manufacture circuitry (i.e., hardware) to perform the desired processing.

[0175] It should be noted that embodiments of this disclosure may further relate to computer products having a non-transitory tangible computer-readable medium having computer code thereon for performing operations of various computer implementations. The medium and computer code may be those specifically designed and constructed for the purposes of this disclosure, or they may be of types known or available to those skilled in the art. Examples of tangible computer-readable media include, for example: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CDs and holographic devices; magneto-optical media; and hardware devices specifically configured for storing or storing and executing program code, such as ASICs, PLDs, flash memory devices, other non-volatile storage devices (such as 3DXPoint-based devices), and ROM and RAM devices. Examples of computer code include, for example, machine code generated by a compiler, and files containing higher-level code executed by a computer using an interpreter. Embodiments of this disclosure may be implemented wholly or partially as machine-executable instructions that may reside in program modules executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may reside physically in a local, remote, or both setting.

[0176] Those skilled in the art will recognize that the computing system or programming language is critical to the practice of this disclosure. They will also recognize that the aforementioned components can be physically and / or functionally separated into modules and / or submodules or combined together.

[0177] Those skilled in the art will understand that the foregoing examples and embodiments are exemplary and do not limit the scope of this disclosure. All arrangements, enhancements, equivalents, combinations, and modifications that will be apparent to those skilled in the art upon reading the specification and studying the accompanying drawings are intended to be included within the true spirit and scope of this disclosure. It should also be noted that the elements of any claim may be arranged differently, including having multiple dependencies, configurations, and combinations.

Claims

1. A system for analyzing a work area, the system comprising: One or more cameras that capture images of the work area; One or more processors; as well as One or more non-transitory computer-readable media, including one or more instruction sets, which, when executed by at least one of one or more processors, cause the following steps to be performed: A working region segmentation neural network subsystem is used to segment the working region into one or more defined regions. The working region segmentation neural network subsystem receives image data from at least one of one or more cameras and segments the working region into one or more defined sub-regions. Using image data from at least one of one or more cameras and an object detection neural network subsystem to detect one or more objects in a working area, the object detection neural network subsystem receives image data to generate a classification of the detected object and boundary region data of the detected object for each detected object from a set of one or more detected objects in the image data; as well as Security issues are detected using a security monitoring subsystem, which includes: The working region segmentation neural network subsystem receives one or more defined subregions of the working region, and for each detected object from a set of one or more detected objects, the object detection neural network subsystem receives its boundary region data. Based on one or more models, one or more defined sub-regions of the working area, and boundary region data from the object detection neural network subsystem, determine whether there are security issues; The security monitoring subsystem is further configured to receive the set of one or more action states and use one or more models to detect security issues by detecting abnormal action states or abnormal action state sequences; and An alert was issued in response to an existing security issue. The one or more non-transitory computer-readable media further include one or more instruction sets that, when executed by at least one of the one or more processors, cause the following steps to be performed: For one or more detected objects, an action recognition subsystem is used to identify one or more action states of the detected objects. The action recognition subsystem uses one or more models based on image data from at least one of one or more cameras to identify one or more action states of the detected objects during the duration of the image data. as well as Using the productivity analysis subsystem, for a detected object, one or more action states are received from the action recognition subsystem, and the productivity of the detected object is determined based on a set of parameters including one or more object-related parameters.

2. The system of claim 1, wherein, At least one of the detected objects is a device, and the one or more non-transitory computer-readable media further includes one or more instruction sets, which, when executed by at least one of the one or more processors, cause the following steps to be performed: For a detected device, an action recognition subsystem is used to identify one or more action states of the device. The action recognition subsystem uses one or more models based on image data from at least one of one or more cameras to identify one or more action states of the device during the duration of the image data.

3. The system of claim 2, wherein, The one or more non-transitory computer-readable media further include one or more instruction sets that, when executed by at least one of the one or more processors, cause the following steps to be performed: A set of cropped images of the detected device is generated using boundary region data and image data of the detected device; as well as A set of cropped images of the detected device is input into an action neural network model, which identifies one or more action states of the detected device on the set of cropped images.

4. The system according to claim 1, wherein at least one of the models of the action recognition subsystem comprises: A set of rules that uses a set of key points of a detected device across a set of images from image data to identify one or more action states of the detected device, wherein the set of key points is obtained from a pose estimation subsystem that uses boundary region data of the detected device and image data from an object detection neural network subsystem to identify key points of the detected device across a set of images.

5. The system of claim 1, wherein, The security monitoring subsystem determines security issues by performing at least one of the following steps: Security is monitored by using boundary region data of the first detected object and boundary region data of the second detected object to determine whether the boundary region data of the first detected object is within a threshold of the boundary region data of the second detected object. Security is monitored by using boundary region data of the first detected object and at least one of one or more defined sub-regions to determine whether a threshold portion of the boundary region data of the first detected object is within one of the defined sub-regions. as well as Security is monitored by using boundary region data of a first detected object, boundary region data of a second detected object, and at least one of one or more defined sub-regions, by determining whether a first threshold portion of the boundary region data of the first detected object and a second threshold portion of the boundary region data of the second detected object are within the same defined sub-region.

6. The system of claim 1, wherein, The one or more non-transitory computer-readable media further include one or more instruction sets that, when executed by at least one of the one or more processors, cause the following steps to be performed: For a detected object from one or more detected objects, the boundary region data is refined using a set of key points of the detected object, wherein the set of key points of the detected object is obtained from a pose estimation subsystem that uses the boundary region data of the detected object and image data from the object detection neural network subsystem to identify a set of key points.

7. The system of claim 1, wherein, The one or more non-transitory computer-readable media further include one or more instruction sets that, when executed by at least one of the one or more processors, cause the following steps to be performed: For detected objects from one or more detected objects: A set of key points for the detected object is obtained from a pose estimation neural network subsystem, which uses boundary region data of the detected object and image data from an object detection neural network subsystem to identify the set of key points; as well as In response to the detection of abnormal key point orientation, at least some of the key points of the detected object are used to determine the security issue.

8. A computer-implemented method for analyzing a work area, the method comprising: A working region segmentation neural network subsystem is used to segment the working region into one or more defined regions. The working region segmentation neural network subsystem receives image data from at least one of one or more cameras and segments the working region into one or more defined sub-regions. Using image data from at least one of one or more cameras and an object detection neural network subsystem to detect one or more objects in a working area, the object detection neural network subsystem receives image data to generate a classification of the detected object and boundary region data of the detected object for each detected object from a set of one or more detected objects in the image data; For one or more detected objects, an action recognition subsystem is used to identify one or more action states of the detected objects. The action recognition subsystem uses one or more models based on image data from at least one of one or more cameras to identify one or more action states of the detected objects during the duration of the image data. as well as Using the productivity analysis subsystem, for a detected object, one or more action states are received from the action recognition subsystem, and the productivity of the detected object is determined based on a set of parameters including one or more object-related parameters. as well as Security issues are detected using a security monitoring subsystem, which includes: The working region segmentation neural network subsystem receives one or more defined subregions of the working region, and for each detected object from a set of one or more detected objects, the object detection neural network subsystem receives its boundary region data. Based on one or more models, one or more defined sub-regions of the working area, and boundary region data from the object detection neural network subsystem, it is determined whether a security issue exists; the security monitoring subsystem is further configured to receive the set of one or more action states and use one or more models to detect security issues by detecting abnormal action states or abnormal action state sequences; and An alert was issued in response to an existing security issue.

9. The computer-implemented method according to claim 8, wherein: The action recognition subsystem obtains one or more action states of the detected object by performing the following steps: A set of cropped images of the detected objects is generated using boundary region data and image data of the detected objects. as well as A set of cropped images of the detected object is input into an action neural network model, which identifies one or more action states of the detected object across a set of cropped images. as well as The computer-implemented method further includes: In response to the detection of an abnormal action state or an abnormal sequence of action states, at least one or more action states are used to detect a security issue.

10. The computer-implemented method of claim 8, wherein, The security monitoring subsystem determines a security issue by performing at least one of the following steps: Security is monitored by using boundary region data of the first detected object and boundary region data of the second detected object to determine whether the boundary region data of the first detected object is within a threshold of the boundary region data of the second detected object. Security is monitored by using boundary region data of the first detected object and at least one of one or more defined sub-regions to determine whether a threshold portion of the boundary region data of the first detected object is within one of the defined sub-regions. as well as using at least one of the first detected object's boundary region data, the second detected object's boundary region data, and the one or more defined sub-regions, by determining whether a first threshold portion of the first detected object's boundary region data and a second threshold portion of the second detected object's boundary region data are within the same defined sub-region point to monitor safety.

11. The computer-implemented method of claim 8, wherein the step of generating boundary region data for the detected object further comprises: For one or more detected objects, the initial boundary region data of the detected objects is refined to obtain boundary region data by performing the following steps: A pose estimation subsystem is used to obtain a set of key points of the detected object. The pose estimation subsystem uses initial boundary region data of the detected object and image data from the object detection neural network subsystem to identify a set of key points of the detected object. as well as The pose of the detected object is determined using a set of key points of the detected object and a pose estimation subsystem. as well as Use poses to refine the initial boundary region data of the detected objects.

12. The computer-implemented method of claim 8, wherein at least one of the one or more models of the action recognition subsystem comprises: A set of rules that uses a set of key points of a detected object across a set of images from image data to identify one or more action states of the detected object, wherein the set of key points is obtained from a pose estimation subsystem that uses boundary region data of the detected object and image data from an object detection neural network subsystem to identify key points of the detected object across a set of images.

13. The computer-implemented method according to claim 8, further comprising: For detected objects from one or more detected objects: A set of key points of a detected object is obtained from a pose estimation neural network subsystem, which uses boundary region data of the detected object and image data from an object detection neural network subsystem to identify the set of key points; as well as In response to the detection of abnormal key point orientation, at least some of the key points of the detected object are used to determine the security issue.

14. A system for analyzing a work area, the system comprising: One or more processors; One or more non-transitory computer-readable media, including one or more instruction sets, which, when executed by at least one of one or more processors, cause the following steps to be performed: A working region segmentation neural network subsystem is used to segment the working region into one or more defined regions. The working region segmentation neural network subsystem receives image data from at least one camera and segments the working region into one or more defined sub-regions. Using image data from at least one camera and an object detection neural network subsystem to detect one or more objects in a working area, the object detection neural network subsystem receives image data to generate a classification of the detected object and boundary region data of the detected object for each detected object from a set of one or more detected objects in the image data. For one or more detected objects, an action recognition subsystem is used to identify one or more action states of the detected objects. The action recognition subsystem uses one or more models based on image data from at least one of one or more cameras to identify one or more action states of the detected objects during the duration of the image data. as well as Using the productivity analysis subsystem, for a detected object, one or more action states are received from the action recognition subsystem, and the productivity of the detected object is determined based on a set of parameters including one or more object-related parameters. as well as Security issues are monitored using a security monitoring subsystem, which includes: The working region segmentation neural network subsystem receives one or more defined subregions of the working region, and for each detected object from a set of one or more detected objects, the object detection neural network subsystem receives its boundary region data. as well as Based on one or more models, one or more defined sub-regions of the working area, and boundary region data from the object detection neural network subsystem, it is determined whether a security issue exists; the security monitoring subsystem is further configured to receive the set of one or more action states and use one or more models to detect security issues by detecting abnormal action states or abnormal action state sequences; and Responding to existing security issues leads to the issuance of alerts.

15. The system according to claim 14, wherein: The action recognition subsystem uses a rule-based model, a neural network-based model, or both to obtain a set of one or more action states of the detected object; and The one or more non-transitory computer-readable media further include one or more instruction sets that, when executed by at least one of the one or more processors, cause the following steps to be performed: Use at least one or more action states to detect security issues in response to the detection of abnormal action states or abnormal action state sequences.

16. The system of claim 14, wherein the security monitoring subsystem determines a security issue by performing the following steps: Security is monitored by using boundary region data of the first detected object and boundary region data of the second detected object to determine whether the boundary region data of the first detected object is within a threshold of the boundary region data of the second detected object. Security is monitored by using boundary region data of the first detected object and at least one of one or more defined sub-regions to determine whether a threshold portion of the boundary region data of the first detected object is within one of the defined sub-regions. as well as Security is monitored by using boundary region data of a first detected object, boundary region data of a second detected object, and at least one of one or more defined sub-regions, by determining whether a first threshold portion of the boundary region data of the first detected object and a second threshold portion of the boundary region data of the second detected object are within the same defined sub-region.

Images