Vision-based monitoring of on-site safety compliance based on worker re-identification and personal protective equipment classification

Abstract

A system for monitoring compliance of a worker wearing personal protective equipment includes a plurality of cameras and a computing system. The plurality of cameras are configured to obtain data, and the data includes a plurality of images associated with one or more objects. The computing system is configured to process the data to determine safety compliance of the one or more objects associated with the plurality of images based on an implementation of a first model trained for re-identification. The computing system is further configured to update the first model trained for re-identification based on a similarity loss.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 289,142, filed December 14, 2021, entitled “VISION-BASED MONITORING OF SITESAFETY COMPLIANCE BASED ON WORKER RE-IDENTIFICATION AND PERSONAL PROTECTIVEEQUIPMENT CLASSIFICATION,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and more specifically, to a system and method for monitoring security compliance. Background Technology

[0004] The construction industry is one of the most dangerous industries, with a recorded high incident rate in recent years. From 2013 to 2019, Hong Kong's construction industry had the highest annual fatality rate among 14 major industrial sectors. The United States also reported that approximately 20% of occupational deaths in 2019 occurred on construction sites. For example, on-site incidents can result in significant financial losses for stakeholders due to compensation claims, necessitating systematic practices for on-site safety monitoring. Construction sites involve highly dynamic interactions between multiple workers and equipment (such as excavators and trucks). In particular, the leading causes of construction injuries and deaths include collisions with moving objects and falls from heights. To avoid serious injuries, any unsafe behaviors of workers should be identified and corrected. Examples include analyzing worker movement on-site and checking the proper wearing of personal protective equipment (PPE), which can be recorded in regular site logs for review and behavioral analysis. Traditional on-site safety monitoring practices primarily involve manual observation of worker behavior and work operations by on-site safety personnel. However, continuous monitoring of workers dispersed across large construction sites is labor-intensive and error-prone, where unsafe behaviors may be overlooked.

[0005] To this end, automated processing of surveillance camera videos can potentially facilitate on-site monitoring. Surveillance cameras are commonly used for various purposes, for example, to identify suspects in public areas such as airports. By incorporating computer vision and deep learning techniques into video processing, visual content on video frames can be interpreted automatically. Using supervised learning as an example, a deep convolutional neural network (CNN) is trained with images whose desired outputs are manually labeled, such that the network learns to extract useful features from images for various tasks such as object detection and classification. These techniques have also been applied to civil engineering, for example, defect detection for structural health assessment, monitoring of construction sites such as location tracking and activity classification of workers and equipment on site, on-site event monitoring, and productivity assessment. Some studies also focus on identifying whether workers are wearing necessary PPE such as safety helmets and reflective vests.

[0006] However, existing studies only monitor individual workers within a small sub-area captured by a single camera. However, the movement of workers cannot be continuously tracked across the entire site. Since workers often move around in different areas of the site, continuously tracking workers over a large area would enable more comprehensive behavior analysis and safety monitoring. Therefore, there is a need to improve image processing techniques in order to monitor individual workers over a wider area range, as well as other applications. SUMMARY

[0007] In an example embodiment, the present disclosure provides a system for monitoring compliance of wearing personal protective equipment. The system for monitoring compliance of wearing personal protective equipment includes a plurality of cameras and a computing system. The plurality of cameras are configured to obtain data. The data includes a plurality of images associated with one or more objects. The computing system is configured to process the data to determine compliance of the one or more objects associated with the plurality of images in wearing personal protective equipment based on implementation of a first model for re-identification. The computing system is further configured to train the first model for re-identification. Training the first model for re-identification includes obtaining a first set of images, determining a similarity matrix including similarity scores of pairs of images in the first set of images based on features extracted from the images using the first model, determining a ground truth similarity matrix including ground truth similarity scores of the pairs of images in the first set of images based on labels marked on the first set of images, determining a comparison matrix by performing element-wise comparison between the similarity matrix and the ground truth similarity matrix, determining a similarity loss based on an element having a maximum value in the comparison matrix, and updating the first model based on the similarity loss. Each image in the first set of images includes an object and is labeled with an identity associated with the object.

[0008] In another example embodiment, the real similarity scores for pairs of images in the first set of images are determined based on a smoothing parameter P, where P is adjustable between 0 and 1.

[0009] In another example embodiment, each pair of images having the same identity is assigned P as the respective real similarity score, and each pair of images having different identities is assigned (1-P) as the respective real similarity score.

[0010] In another example embodiment, each pair of images in the first set of images includes a first image and a second image. Determining the similarity matrix including similarity scores for pairs of images in the first set of images further includes, for each pair of images, determining a first feature vector for the respective first image and a second feature vector for the respective second image, and determining a cosine similarity as the similarity score for the pair of images based on the first feature vector and the second feature vector. The cosine similarity is obtained by dividing a product of the first feature vector and the second feature vector by an absolute value of the first feature vector and the second feature vector.

[0011] In another example embodiment, determining the comparison matrix by performing an element-wise comparison between the similarity matrix and the real similarity matrix further includes determining an absolute difference or a squared difference between each similarity score in the similarity matrix and a corresponding real similarity score in the real similarity matrix. The respective similarity score and the corresponding real similarity score are associated with a same pair of images.

[0012] In another example embodiment, the computing system is further configured to train a second model for personal protective equipment classification. Training the second model for personal protective equipment classification includes receiving a second set of images associated with a plurality of classes, determining a total number of images in each class of the plurality of classes, determining a first class of the plurality of classes having a maximum number of images, determining a weight associated with the class based on a ratio of the maximum number of images in the first class to a number of images in a class of the plurality of classes, and updating the second model based on the weights associated with the plurality of classes each iteration. Each image in the second set of images is labeled with a class.

[0013] In another example embodiment, during training of the second model, the weight associated with a class is assigned to all images in the class. The loss calculation of each iteration is based on the weights associated with images in the second set of images.

[0014] In another example embodiment, processing the data to determine the compliance of the one or more objects associated with the plurality of images with the personal protective equipment further includes: identifying the one or more objects based on features extracted from the plurality of images in the data by using the first model; classifying the one or more objects based on the features extracted from the plurality of images in the data by using the second model; and determining the compliance of the one or more objects with the personal protective equipment based on the identifying and classifying results.

[0015] In another example embodiment, the identifying and classifying results include the plurality of images, and each image in the plurality of images is associated with a timestamp, an identification, and a safety status. The computing system is further configured to: determine that a subset of images in the plurality of images is associated with an event. The subset of images is in a category that indicates a non-compliant safety status. The subset of images is associated with a time period that is greater than or equal to a minimum buffer time threshold. The subset of images includes one or more time gaps. Each time gap is shorter than a time gap threshold.

[0016] In another example embodiment, the computing system is further configured to: generate a log including the detected event; and generate a warning corresponding to the detected event. The warning includes any of a warning message, an alarm, a flashing light.

[0017] In another example embodiment, the present disclosure provides a method for monitoring compliance with personal protective equipment. The method includes a computing system obtaining data from a plurality of cameras and processing the data to determine compliance of one or more objects associated with the plurality of images with personal protective equipment based on implementation of a first model trained for re-identification. The data includes a plurality of images associated with one or more objects. The first model is trained for re-identification based on: obtaining a first set of images; determining, using the first model, a similarity matrix including similarity scores for pairs of images in the first set of images based on features extracted from the images; determining a ground truth similarity matrix including ground truth similarity scores for the pairs of images in the first set of images based on the identifications labeled on the first set of images; determining a comparison matrix by performing an element-wise comparison between the similarity matrix and the ground truth similarity matrix; determining a similarity loss based on an element in the comparison matrix having a maximum value; and updating the first model based on the similarity loss. Each image in the first set of images includes an object and is labeled with an identification associated with the object.

[0018] In another example embodiment, the method further includes training a second model for personal protective equipment classification. Training the second model for personal protective equipment classification includes receiving a second set of images associated with a plurality of classes, determining a total number of images in each class of the plurality of classes, determining a first class of the plurality of classes having a maximum number of images, determining a weight associated with the class based on a ratio of the maximum number of images in the first class to a number of images in a class of the plurality of classes, and updating the second model based on the weights associated with the plurality of classes each iteration. Each image of the second set of images is labeled with a class.

[0019] In another example embodiment, during training of the second model, a weight associated with a class is assigned to all images in the class. A loss calculation for each iteration is based on the weights associated with images of the second set of images.

[0020] In another example embodiment, processing the data to determine compliance of the one or more objects wearing personal protective equipment associated with the plurality of images further includes identifying the one or more objects based on features extracted from the plurality of images in the data by using the first model, classifying the one or more objects based on the features extracted from the plurality of images in the data by using the second model, and determining compliance of the one or more objects wearing personal protective equipment based on the identifying and classifying results.

[0021] In another example embodiment, the identifying and classifying results include the plurality of images, and each image of the plurality of images is associated with a timestamp, an identification, and a safety status. The method further includes determining a subset of images of the plurality of images are associated with an event. The subset of images are in a class indicating a non-compliant safety status. The subset of images are associated with a time period greater than or equal to a minimum buffer time threshold. The subset of images include one or more time gaps. Each time gap is shorter than a time gap threshold.

[0022] In yet another example embodiment, the present disclosure provides a non-transitory computer-readable medium having stored thereon processor-executable instructions for monitoring compliance of wearing personal protective equipment using a computing system. The computing system executes the instructions to cause: obtaining data from a plurality of cameras; and processing the data to determine compliance of one or more objects associated with the plurality of images wearing personal protective equipment based on implementation of a first model trained for re-identification. The data includes a plurality of images associated with one or more objects. The first model is trained for re-identification based on: obtaining a first set of images; determining, using the first model, a similarity matrix comprising similarity scores of pairs of images in the first set of images based on features extracted from the images; determining a ground truth similarity matrix comprising ground truth similarity scores of the pairs of images in the first set of images based on labels marked on the first set of images; determining a comparison matrix by performing an element-wise comparison between the similarity matrix and the ground truth similarity matrix; determining a similarity loss based on an element having a maximum value in the comparison matrix; and updating the first model based on the similarity loss. Each image in the first set of images includes an object and is labeled with an identity associated with the object.

[0023] In another example embodiment, the computing system executes the instructions to further cause training a second model for personal protective equipment classification. Training the second model for personal protective equipment classification includes: receiving a second set of images associated with a plurality of classes; determining a total number of images in each class of the plurality of classes; determining a first class of the plurality of classes having a maximum number of images; determining a weight associated with the class based on a ratio of the maximum number of images in the first class to a number of images in a class of the plurality of classes; and updating the second model based on the weights associated with the plurality of classes each iteration. Each image in the second set of images is labeled with a class.

[0024] In another example embodiment, during the training, the weight associated with a class is assigned to all images in the class. The loss calculation of each iteration is based on the weights associated with images in the second set of images.

[0025] In another example embodiment, processing the data to determine compliance of the one or more objects associated with the plurality of images wearing personal protective equipment further includes: identifying the one or more objects based on features extracted from the plurality of images in the data by using the first model; classifying the one or more objects based on the features extracted from the plurality of images in the data by using the second model; and determining compliance of the one or more objects wearing personal protective equipment based on the identifying and classifying results.

[0026] In another example embodiment, the identification and classification results include the plurality of images, and each image of the plurality of images is associated with a timestamp, an identification, and a safety status. The computing system executes instructions to further cause: determining that a subset of images of the plurality of images are associated with an event. The subset of images are in a category that indicates non-compliance with the safety status. The subset of images are associated with a time period that is greater than or equal to a minimum buffer time threshold. The subset of images includes one or more time gaps. Each time gap is shorter than a time gap threshold. BRIEF DESCRIPTION OF DRAWINGS

[0027] Systems and methods for monitoring safety compliance are described in detail below with reference to the accompanying drawings, wherein:

[0028] Figure 1A A network environment for implementing the technology in this disclosure is shown in accordance with some embodiments.

[0029] Figure 1B is a block diagram of an example computing device configured to implement various functions in accordance with some embodiments.

[0030] Figure 2A is a flowchart of a method for monitoring site safety compliance in accordance with some embodiments.

[0031] Figure 2B An example of a method for monitoring site safety compliance is shown in accordance with some embodiments as Figure 2A

[0032] Figure 2C is a flowchart of a method for training a model for re-identification in accordance with some embodiments.

[0033] Figure 2D is a flowchart of a method for training a model for performing PPE classification in accordance with some embodiments.

[0034] Figure 2E is a flowchart of a method for monitoring site safety compliance in accordance with some embodiments.

[0035] Figure 3 A flowchart of obtaining images for monitoring site safety compliance is shown in accordance with some embodiments.

[0036] Figure 4 is a table of example data obtained for monitoring site safety compliance in accordance with some embodiments.

[0037] Figure 5 An example similarity matrix is shown in accordance with some embodiments.

[0038] ​Figure 6 An element-wise comparison between a similarity matrix generated by a ReID model and a ground truth similarity matrix is shown, according to some embodiments.

[0039] Figure 7 A workflow for PPE classification is shown, according to some embodiments.

[0040] Figure 8 is a table presenting information associated with a plurality of images for monitoring on-site safety compliance, according to some embodiments.

[0041] Figure 9A is a screenshot of an exemplary user interface 900 of a site monitoring software, according to some embodiments.

[0042] Figure 9B An exemplary process for event counting performed by a computing device is shown, according to some embodiments. DETAILED DESCRIPTION

[0043] Systems and methods are disclosed that relate to a vision-based framework for monitoring worker compliance in wearing personal protective equipment (hereinafter simply referred to as “safety compliance”). The framework combines algorithms for worker re-identification (ReID) and classification of personal protective equipment (PPE). Deep learning-based methods are implemented to address the challenges in the above two tasks. Regarding the task of ReID, a new loss function (called similarity loss) is designed to facilitate the deep learning model to learn more discriminative human features, thus enabling more robust tracking of individual workers. For the classification task, a weighted class strategy is used to mitigate model bias when given imbalanced samples among classes, so as to improve the performance of classifying the PPE status of the identified workers despite the limited training samples. By combining the ReID and PPE classification results, a workflow is developed to record any event where a worker is not wearing the necessary PPE.

[0044] When applied to analyze actual construction site datasets, exemplary embodiments of the present disclosure are able to achieve significantly improved accuracy of worker ReID and PPE classification over traditional techniques by facilitating on-site video analysis and inspection of worker on-site safety compliance.

[0045] Figure 1A A network environment 100 for implementing the techniques in the present disclosure is shown, according to some embodiments.

[0046] A network environment suitable for implementing embodiments of the present disclosure can include one or more cameras 120, computing devices 130, and / or other devices 140. By way of example and not limitation, a camera 120 can be embodied as an imaging device, such as a video camera, a monitoring device / system, etc., or an imaging module integrated in an electronic device / system, like a smartphone, an autonomous vehicle, a drone, a robot, etc. A computing device 130 can include one or more client devices, one or more servers, or a combination thereof. Examples of a computing device 130 can include, but are not limited to, a personal computer (PC), a laptop, a workstation, etc. Other devices 140 can include a network-attached storage focus (NAS), other backend devices, or any other suitable devices.

[0047] Components of the network environment 100 can communicate with each other via network(s) 110, which can be wired, wireless, or both. Network(s) 110 can include multiple networks or one of multiple networks. By way of example, network(s) 110 can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the public switched telephone network (PSTN), and / or one or more private networks. Where network(s) 110 include a wireless telecommunication network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0048] A compatible network environment can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein with respect to server(s) can be implemented on any number of client devices.

[0049] In at least one embodiment, network environment 100 can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functionality described herein (or one or more portions thereof). Any of these functionalities can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across states, regions, countries, globally, etc.). A cloud-based network environment can be private (e.g., limited to a single organization), can be public (e.g., available to many organizations), and / or combinations thereof (e.g., a hybrid cloud environment).

[0050] Figure 1BThis is a block diagram of an exemplary computing device 130 configured to perform various functions according to some embodiments. A computing system may include one or more computing devices 130 and other suitable devices / components.

[0051] like Figure 1B As shown, computing device 130 may include one or more processors 150, a communication interface 170, and memory 160. The processors 150 may be configured to perform operations according to instructions stored in memory 160. The processors 150 may include any suitable type of general-purpose or special-purpose microprocessor, such as a central processing unit (CPU), a graphics processing unit (GPU), a parallel processing unit (PPU), etc. Memory 160 may be configured to store computer-readable instructions that, when executed by the processors 150, enable the processors 150 to perform the various operations disclosed herein. Memory 160 may be any non-transitory type of mass storage device, such as volatile or non-volatile, magnetic, semiconductor-based, magnetic tape-based, optical, removable, non-removable, or other types of storage devices or tangible computer-readable media, including but not limited to read-only memory (“ROM”), flash memory, dynamic random access memory (“RAM”), and / or static RAM.

[0052] Communication interface 170 can be configured to connect computing device 130 to other devices or systems (such as, one or more cameras 120, another computing device 130, or one or more other devices 140, such as...) Figure 1A Information can be transmitted between (as shown). For example, communication interface 170 may include an Integrated Services Digital Network (“ISDN”) card, a cable modem, a satellite modem, or a modem that provides a data communication connection. As another example, communication interface 170 may include a Local Area Network (“LAN”) card to provide a data communication connection to a compatible LAN. As a further example, communication interface 170 may include a high-speed network adapter, such as a fiber optic network adapter, a 10G Ethernet adapter, etc. A wireless link may also be implemented by communication interface 170. In such an implementation, communication interface 170 can send and receive electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information via a network. The network may typically include a cellular communication network, a wireless local area network (“WLAN”), a wide area network (“WAN”), etc.

[0053] The communication interface 170 may also include various I / O devices, such as a keyboard, mouse, touchpad, touchscreen, microphone, camera, biosensor, etc. Users can input data to the computing device 130 (e.g., a terminal device) through the communication interface 170.

[0054] In some embodiments, the display can be integrated as part of the computing device 130, or can be provided as a separate device communicatively coupled to the computing device 130. The display can include a display device such as a liquid crystal display ("LCD"), a light emitting diode display ("LED"), a plasma display, or any other type of display, and provide a graphical user interface ("GUI") rendered on the display for user input and data portrayal. In some embodiments, the display can be integrated as part of the communication interface 170.

[0055] Figure 2A is a flowchart of a method 200 for monitoring safety compliance at a site in accordance with some embodiments. The method 200 can be performed by a system including a plurality of cameras 120, one or more computing devices 130, and other suitable devices as shown in Figure 1A / Figure 1B shown can be established in a network environment 100 as shown in Figure 1A It will be recognized that the method 200 can be performed in any suitable environment, and any of the blocks in Figure 2A may be performed in any suitable order. Figure 2B An exemplary process 242 is shown that performs the method 200 as shown in Figure 2A

[0056] At step 210, the computing device 130 obtains data from the one or more cameras 120.

[0057] The one or more cameras 120 can be distributed at a plurality of locations at the site. Each camera 120 can provide coverage of the site corresponding to a field of view of the camera 120. Figure 3 A flowchart 300 of obtaining images for monitoring safety compliance at a site in accordance with some embodiments is shown. An exemplary layout of a plurality of cameras of a site is shown in block 310. As shown in 320, a camera 320a is represented by a solid triangle, and a field of view 320b of the camera is represented by a hollow triangle. In this example, four cameras are placed at different locations of the site. Each camera can be assigned a unique camera number (or index), such as the number of camera 312a is Caml. Camera 312a (Caml) has a field of view 312b, camera 314a (Cam2) has a field of view 314b, camera 316a (Cam2) has a field of view 316b, and camera 318a (Cam2) has a field of view 318b. Each camera 120 can obtain a video stream of the corresponding location, which includes some raw images (video frames). The cameras 120 can send data including the video stream to the computing device 130 via the network environment 100. In addition, each raw image can be associated with a timestamp indicating a time instant at which the image is captured and a camera number indicating the camera that captures the image. Figure 3 ​The four original images 322, 324, 326, and 328 from the four cameras Caml, Cam2, Cam3, and Cam4, respectively, are shown in block 330. Each of the original images can capture one or more objects of interest, such as the worker in this example.

[0058] The computing device 130 can implement various object recognition algorithms to detect objects in the original images. In an embodiment, one or more bounding boxes can be overlaid on each of the original images to enclose the detected object(s) in the original images. For example, a bounding box can represent a detection of a person, which includes a subset of pixels in the original image. As shown, in image 322, the computing device 130 can detect a person and enclose the detected person with a bounding box 332. Similarly, the computing device 130 can generate bounding boxes 334, 336 in image 324, bounding box 338 in image 326, and bounding boxes 340, 342, and 344 in image 342. Figure 3

[0059] In a further embodiment, the computing device 130 can adjust the size of each bounding box by cropping the portion of the original image around the object (e.g., the worker) with minimal background clutter, thereby reducing background noise. This can be done automatically by first generating a rectangular bounding box around each object, and then cropping each bounding box to remove pixels in the original image that are not related to the object.

[0060] Hereinafter, the term "image" refers to a portion of the original image that is cropped to correspond to a bounding box.

[0061] The computing device 130 can perform feature extraction on the images to identify objects across multiple cameras and generate grouped images, as shown in block 350, which will be described in detail in later steps of the method 200.

[0062] In Figure 2B block 210 demonstrates exemplary results by performing step 210 of the method 200 using the computing device 130. In this example, the computing device 130 can obtain original images from Caml and Cam2. The computing device 130 can then generate images 212a and 212b of the identified objects in the original images from Caml, and generate images 214a, 214b, and 214c of the identified objects in the original images from Cam2. In another embodiment, the computing device 130 can associate information corresponding to the original images, such as a timestamp and a camera index, to the generated images. Figure 4 ​is a table 400 of exemplary data obtained in presenting to monitor on-site safety compliance. The table 400 includes images 410 generated by the computing device 130 and corresponding information of timestamps 420 and camera indices 430.

[0063] Referring back to Figure 2A At step 220, the computing device 130 identifies one or more objects based on the features extracted from the data. The computing device 130 can process the images obtained from step 210 to extract features related to each detected object. By finding similarities of the extracted features, the computing device 130 can group images with similar features in order to identify each object across multiple cameras 120. Figure 2B Block 220 in FIG. 2B illustrates an exemplary result of performing step 220 by grouping images generated for detected workers based on similarities of features and assigning a unique identification (ID) to each group, such as ID1 222 and ID2 224 in this example. In this way, the computing device 130 can identify a first worker corresponding to ID1 222 and a second worker corresponding to ID2 224 across multiple cameras 120.

[0064] Generally, tracking multiple people across multiple cameras can be formulated as a problem of person re-identification (ReID). Given videos from different cameras with non-overlapping fields of view, the computing device 130 can first detect people captured in each frame and generate images corresponding to the detected people, e.g., as bounding boxes cropped from the original images. The computing device 130 can then process the images by using a ReID model that extracts distinguishing human features from each person and determines feature similarities between images. The computing device 130 can match the same identities and distinguish different identities in images based on the feature similarities. In this way, the ReID model can be used to determine consistent identities across multiple cameras 120.

[0065] Previous research on ReID has encountered several technical challenges, which can be divided into two aspects. The first aspect is intra-person appearance variation. When the same person is captured at different locations, the appearance of the person can vary greatly due to changes in the viewpoint, clothing, lighting conditions, or partial occlusion of the human body. If the extracted features are sensitive to such variations, the identification can be lost. The second aspect is inter-person appearance ambiguity. When capturing an arbitrary group of people, they can have very similar visual features, such as clothing color, style, and certain items. If relying on some generic feature descriptors that are not distinctive enough, the identification can be confused. In view of these technical challenges, there is a need to develop robust ReID methods that can extract visual features that are discriminative for identification.

[0066] In one embodiment, the computing device 130 can implement a ReID model to perform feature extraction for each detected worker, from which discriminative human features can be utilized to match worker identities across multiple cameras and distinguish different workers. The computing device 130 can use the ReID model to determine a feature vector to represent the features in each image. The feature vector can include a plurality of elements, each of which can correspond to a feature map used in the ReID model for filtering out one particular feature from the image. In this way, the computing device 130 can perform a feature extraction process to obtain a feature vector for the images obtained from step 210 of the method 200. The computing device 130 can then compare the feature vectors in order to directly quantify the visual similarity between workers in the images for identification matching. Many researches have proposed different ReID models, with recent works mainly designing novel convolutional neural network (CNN) architectures to extract discriminative features. However, the development of loss functions to facilitate the model training process has been relatively underexplored, which is crucial for regulating the learning process of the feature representation.

[0067] In one embodiment, a loss function referred to as "similarity loss" can be implemented in the ReID model to facilitate an effective training process of the ReID model by learning more discriminative human features. In a manner that the loss function with similarity loss is implemented, the ReID model can be trained to minimize the error in determining the similarity between pairs of images. The pairs of images can be classified into two types. A positive pair of images can include two images associated with the same identity. A negative pair of images can include two images associated with different identities. The ReID model can be trained to generate a similarity score for each pair of images. Intuitively, the similarity score is expected to be high for positive pairs, while the similarity score is expected to be low for negative pairs. The similarity loss can be used to incorporate the computed similarity scores between different pairs of images directly into a training objective, such as a loss function. In this way, the ReID model can be guided to learn features that are capable of outputting reasonable similarity scores between positive and negative pairs. The formulation of the similarity loss will be described in detail below.

[0068] In an example training process, a set of training images and ground truth similarity labels (or ground truth similarities) for pairs of images in the training images can be provided to train a model to learn consistent similarities. The model can generate similarity scores for pairs of images, which are compared to the ground truth similarity labels in a stage where a loss is computed using a loss function with similarity loss. The learnable parameters in the model can be adjusted based on the computed loss. By learning similarities between the set of training images, the ground truth information can facilitate the model to output high similarity scores between positive pairs, and low scores between negative pairs.

[0069] In one example, the similarity score can be considered as a probability value ranging between 0 and 1, i.e., all positive pairs have a similarity score of 1.0, while negative pairs have a score of 0.0. This formulation of ground truth similarity can be used to force the model to learn a discriminative feature representation that maximizes the relative similarity difference between positive and negative pairs as much as possible. To this end, the model can learn discriminative human features that are capable of confidently matching the same worker with a similarity score of 1.0 and distinguishing different workers with a score of 0.0.

[0070] However, the above setting of similarity scores for positive and negative pairs can cause problems in some cases, where some redundant or even irrelevant features can be used to produce the ground truth similarity, leading to misleading results of identity matching. For example, if some background clutter such as rebar is noticed instead of the worker of interest, different identities can be wrongly matched due to similar background.

[0071] From another perspective, the labeling of similarity scores between positive and negative samples can be analogous to the labeling process for object classification, i.e., distributing true probabilities between positive and negative classes. For example, in K possible class labels for an image, a direct one-hot label (including a set of bits with a single "1" bit and all other low "0" bits) gives a probability of 1.0 to the positive class 1.0, while all other negative classes have a probability of 0.0. However, producing such a large relative disparity can lead to overfitting since the model is forced to learn features that strongly increase the confidence of its predicted class label. Thus, certain irrelevant features can be learned that can not generalize well from the training images.

[0072] In one example, the label smoothing technique described in Szegedy et al., "Speed / accuracy trade-offs for modern convolutional object detectors," published at Proc. -30 th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, 2017: pp. 3296-3305, the entire contents of which are hereby incorporated by reference, can be used to mitigate the opportunity to learn misleading features as described above. The label smoothing technique extracts a small fraction of probability and distributes it evenly to all negative classes. To illustrate by way of example, a dataset can include five classes, with one true class and the remaining classes being negative classes (e.g., including images associated with different identities). In this example, 10% of the total probability can be extracted from the true class, resulting in each of the four remaining negative classes receiving a 2.5% probability label. This can effectively adjust the training process such that the model can be prevented from learning misleading features.

[0073] In another example, as inspired by the label smoothing technique, a smoothing parameter P can be incorporated in the formulation of the similarity loss, allowing for a more flexible definition of the similarity label. Figure 5 is an example of a similarity matrix 500 including similarity labels generated between pairs of training images. Each training image 510 is associated with an ID 520. True similarity labels can be generated based on the IDs 520 associated with the training images 510. The similarity labels between pairs of images are presented in field 530 in the similarity matrix 500. As shown in field 530, no similarity label is generated for a pair of identical images, an image pair including two different images with the same ID is given P as a similarity label, and an image pair including two different images with different IDs is given (1-P) as a similarity label.

[0074] Similarly, in the pair-wise similarity matrix of a batch of B images, each positive pair is labeled with a similarity score P (a decimal number between 0 and 1), while each negative pair is labeled with a similarity score (1-P). It should be noted that different P values can be used in the similarity matrix of different image pairs. In this way, each similarity label has a finite range between 0 and 1. Basically, the smoothing parameter P can control the relative difference of the true similarity between positive pairs and negative pairs. For example, if P is reduced to 0.7, the positive pairs and negative pairs have true similarity of 0.7 and 0.3, respectively, resulting in much smaller relative difference than being labeled with 1.0 and 0.0. In addition, the smoothing parameter P can be adjusted to achieve different effects on model training. By defining P as the true similarity of positive pairs, the similarity loss is generic and applicable to different ReID scenarios when adjusting the smoothing parameter P.

[0075] By comparing whether the identity labels of each image pair are the same or different, the similarity labels for the training image set can be automatically generated. In this way, a pair-wise B x B similarity matrix for the batch of B images in the training image set can be formed, which can then constitute the ground truth of the batch during training, thereby guiding the ReID model to utilize features that best replicate the ground truth similarity scores in the training images.

[0076] With the generated ground truth similarity labels, the training images can be fed into the ReID model for feature representation learning. The paradigm of supervised learning for deep CNNs can be employed. During training, the ReID model can generate an output via forward propagation, and the deviation of the output from the ground truth is computed as a training loss, and then the learnable parameters in the model are iteratively updated via backpropagation. The CNN architecture for the ReID task can include a backbone model, which can include several convolutional layers, pooling layers, and fully connected layers, for utilizing hidden features from the mirror images. The backbone model can generate a feature vector for each image. The elements in the feature vector can represent features extracted from the image. The dimensionality of the feature vector, i.e., the number of elements in the feature vector, can depend on the model architecture of the CNN. In some cases, the backbone model can process each batch of B images, and generate B feature vectors corresponding to the images in the batch. Subsequently, the ReID model can evaluate the visual similarity between the B images by associating the corresponding B feature vectors. More specifically, a B x B pair-wise similarity matrix can be computed in the feature vectors, which can be compared with the ground truth similarity matrix.

[0077] Figure 6 The similarity matrix (M) 610 generated by the ReID model and the ground truth similarity matrix (M GTElement-wise comparisons 600 and 620 are performed. Element-wise comparison 600 compares the generated similarity label (or similarity score) for each image pair with the true similarity label for that image pair. Elements in the similarity matrix can be queried using the index set (i,j), where i is the row index and j is the column index. In this example, P is set to 1.0 for positive pairs in the true similarity matrix 620.

[0078] Various metrics can be applied to calculate similarity scores. In one formula, cosine similarity can be used, which benefits the calculation of similarity loss by forcing the similarity score within a finite range between 0 and 1. This ensures that the range of similarity scores matches the defined range of similarity labels. Each element (i,j) in the similarity matrix can be calculated using the following formula:

[0079] If i ≠ j, then cosine similarity Otherwise, it is zero (Formula 1).

[0080] Among them, f i and f j Let f represent the feature vectors corresponding to the i-th and j-th images arranged in the similarity matrix, respectively. The feature vector f can be calculated by applying Equation 1. i and f j The cosine similarity between them. Since all elements along the diagonal axis represent the self-similarity of the feature vectors themselves (which is always 1.0 due to equivalence), these elements are ignored when calculating the similarity loss. Thus, during the training of a ReID model with similarity loss, two similarity matrices can be compared, for example... Figure 6 The M610 and M shown GT 620.

[0081] The following illustrates an example of training a ReID model implemented in computing device 130. It will be appreciated that computing device 130 can perform the steps described in this example in any suitable environment and in any suitable order. First, computing device 130 can compute matrix M610 via forward propagation of the ReID model. Second, computing device 130 can generate matrix M... GT 620 is used as the true similarity matrix. Third, the computing device 130 can calculate the similarity loss by applying the following formula:

[0082] Similarity loss = max((MM) GT ) 2 ) (Formula 2)

[0083] Next, the computing device 130 can perform backpropagation for model updating based on the similarity loss calculated according to Equation 2. Specifically, the computing device 130 can generate matrices M610 and M GTThe element-wise squared differences between 620 can form a matrix with the same shape as M610 and M. GT A comparison matrix of dimension 620 is used. The comparison matrix captures how much the generated / predicted similarity score of an image pair deviates from its corresponding true value. The computing device 130 can then extract the maximum value in the comparison matrix by applying the max() operation. Therefore, the computing device 130 obtains a similarity loss value for a batch of training images, defined as the maximum deviation in similarity scores. The formulation of the loss function guides the ReID model to learn feature representations that progressively reconstruct the true similarity scores. To this end, the computing device 130 may be able to determine higher similarity in positive pairs and lower similarity in negative pairs. Compared to the commonly used softmax loss, which formulates ReID training as a classification task, the training objective of the similarity loss is better aligned with the testing process of the ReID task, allowing the ReID model to receive more direct supervision during its training.

[0084] Furthermore, as shown in Equation 2, the difference between the two matrices is squared to estimate the absolute value. Other variations, such as the absolute difference, can be implemented in Equation 2 to calculate the similarity loss. Additionally, the max() operation effectively suppresses the deviation between the ReID model's output and the truth. The max() operation can be replaced by other variations (e.g., the mean() operation). For example, the mean() operation can be implemented in Equation 2 to take the average deviation between all image pairs in a batch.

[0085] In the testing phase, the computing device 130 can process the feature vectors extracted by the trained ReID model and associate worker identities with the images based on the feature vectors, which should be consistent across different images and different cameras. In other words, the computing device 130 can give all images of the same worker a unique identity and distinguish different workers at the same time. Based on the extracted feature vectors, the task of identity matching is formulated as a clustering problem. Various algorithms can be implemented to solve the clustering problem. For example, the following can be employed: the method described by Wong et al. in “Enriched and discriminative convolutional neural network features for pedestrian re-identification and trajectory modeling” (published in Comput. Civ. Infrastruct. Eng. 36 (2021), available at: doi.org / 10.1111 / mice.12750), which is incorporated by reference herein in its entirety, and the improved version of the hierarchical agglomerative clustering algorithm described by Han et al. in “Data mining: concepts and techniques” (published by Elsevier, 2011, available at: doi.org / 10.1016 / C2009-0-61819-5), which is incorporated by reference herein in its entirety. Studies on ReID models show that the identity matching process largely depends on the quality of the extracted features, e.g., how well each feature vector distinguishes identities. Therefore, developing a robust ReID method is considered to be a key to facilitating Figure 2A Step 220 of the method 200 shown is critical, which is achieved by implementing a similarity loss in the ReID model as described in the present disclosure.

[0086] Figure 2C A flowchart of a method 250 of training a model for re-identification by implementing the techniques described in step 220. The method 250 can be performed by a system comprising a plurality of cameras 120, one or more computing devices 130, and other suitable devices as shown in Figure 1A / Figure 1B the network environment 100 as shown. It will be appreciated that the method 250 can be performed in any suitable environment, and any of the blocks in Figure 1A may be performed in any suitable order. Figure 2C

[0087] At step 252, the computing device 130 obtains a set of images. Each image in the set of images includes an object and is labeled with an identity associated with the object. For example, an image with an identity can be associated with a worker wearing a uniform with a number on it. The number on the uniform is the identity of the worker. The set of images can be obtained from any suitable source, such as a database, a camera, or a network.​Figure 5 The images in the similarity matrix 500 shown.

[0088] At step 254, the computing device 130 determines a similarity matrix based on the features extracted from the images using the model. The similarity matrix includes similarity scores for pairs of images in the set of images.

[0089] At step 256, the computing device 130 determines a ground truth similarity matrix based on the labels tagged on the set of images. The ground truth similarity matrix includes ground truth similarity scores for pairs of images in the set of images.

[0090] At step 258, the computing device 130 determines a comparison matrix by performing an element-wise comparison between the similarity matrix and the ground truth similarity matrix.

[0091] At step 260, the computing device 130 determines a similarity loss based on the element in the comparison matrix having the largest value.

[0092] At step 262, the computing device 130 updates the model based on the similarity loss.

[0093] The trained model can be used by the computing device 130 to perform step 220 of the method 200 as shown in Figure 2A At step 230, the computing device 130 classifies one or more objects based on the features extracted from the data.

[0094] At step 230, the computing device 130 classifies one or more objects based on the features extracted from the data. Figure 2B Block 230 in FIG. 6 shows an example result of performing step 230. As shown in block 230, each image is associated with a label such as labels 232, 234, 236, and 238 that indicates the PPE status of the detected worker in the image. Studies have shown that the necessary PPE including a helmet and a vest is required for working on site most of the time. Thus, classifying whether a worker is carrying both of these PPEs will be able to monitor the regular compliance with respect to site safety. The identification of the PPE status of the worker can be formulated as a multi-class classification problem. For example, each worker can be classified as one of the pre-defined PPE statuses.

[0095] Figure 7 A workflow 700 for PPE classification is shown that can be performed by the computing device 130. As shown in block 710, the computing device 130 can obtain data including raw images 712 and by performing steps as shown in Figure 2AThe bounding box 720 of the detected worker in the original image 712 can be generated by the computing device 130 in step 210 of the method 200 shown in FIG. 7. The image 720 corresponds to the bounding box 720. The computing device 130 can implement the CNN 730 as a classifier that takes the image 720 as input and outputs a PPE status as one of the predefined PPE statuses, as shown in block 740. In this example, four PPE classes / statuses are defined. The first class “W” indicates that the worker is not wearing both necessary PPEs. The second class “WH” indicates that the worker is wearing only a helmet. The third class “WV” indicates that the worker is wearing only a vest. The fourth class “WHV” indicates that the worker is wearing both a helmet and a vest.

[0096] Various vision-based methods for detecting the helmet and the vest worn by the worker can be employed by the computing device 130 to train the model (e.g., the CNN 730) to facilitate step 230.

[0097] Research shows that a training dataset including imbalanced samples among classes can significantly reduce the accuracy of the model with respect to PPE classification. For example, in the training dataset, the image samples labeled with the classes “WH” and “WHV” can constitute a majority (i.e., more than 90%) of the training dataset, while the image samples labeled with the classes “W” and “WV” are minority classes in the training dataset. The imbalanced samples among classes in the training dataset can cause the feature learning of the model to be biased towards the majority classes, because the model cannot utilize the features from the images of other classes due to insufficient samples. Therefore, training the model, e.g., the CNN 730 in FIG. 7, can show much lower classification accuracy, especially for the minority classes “W” and “WV”, resulting in a portion of these samples being misclassified as the majority classes. For example, an image labeled with “WV” can be classified as “WHV”. In other words, a considerable number of workers without a helmet can be undesirably overlooked, thus making the monitoring of safety compliance unreliable. Since the class imbalance in the dataset can be the root cause, a method is needed to achieve robust performance for the minority classes under limited training samples. Figure 7

[0098] In one embodiment, the computing device 130 can employ a weighted class strategy in the workflow 700 as shown in FIG. 7 to facilitate the model learning of the CNN 730 for the minority classes in the training dataset. The weighted class strategy can manipulate the weights of the classes, e.g., by controlling the extent to which each sample affects the update of the model, when the samples (i.e., image samples) in the training dataset participate in the backpropagation of the model. The general principle is to increase the weights of the samples in the minority classes so that these samples exert a greater impact on the loss calculation than the samples in the majority classes. In some cases, the weight of each class can be determined by Figure 7

[0099] ​​New class weight

[0100] where W c is the new class weight for class c, S c is the number of samples in class c. According to Equation 3, determining the weight for a class takes into account the ratio of the sample counts in all classes. Formally, given the number of samples in each class (S c ), the maximum number is first found by Then, within each class c, the new class weight is determined by dividing by the sample count of class c (S c ). Then, during model training, the new weight W c can be assigned to each sample of class c. In this way, samples can contribute to the calculation of loss according to their assigned weight. The weighted class strategy can be combined with various types of algorithms for loss calculation. For example, the computing device 130 can employ a softmax loss algorithm for loss calculation, where samples in the training dataset can be processed according to the new class weights computed by applying Equation 3.

[0101] The weighted class strategy aims to equalize the aggregate contribution of all classes when the dataset includes an imbalanced number of samples in different classes. Numerically, the new weight of the class with the maximum sample count is The new weights of other classes with fewer samples increase proportionally to the ratio of the sample counts, For example, a class with half the maximum number of samples can receive a new weight of 2.0, such that each sample has double the impact on the model relative to samples in the class with the maximum sample count. On the other hand, for each class c, the product S c x W c yields the same value of such that the aggregate weighted impact of samples in each class is equal. In this way, the problem of biased training due to imbalanced samples in different classes can be mitigated by balancing the weighted impact. Thus, a training model incorporating the weighted class strategy can be more robust when performing PPE classification, especially for minority classes with limited training samples.

[0102] Figure 2D is a flowchart of a method 270 of training a model for performing classification by implementing the techniques described in step 230. The method 270 can be performed by a system including a plurality of cameras 120, one or more computing devices 130, and other suitable devices as shown in Figure 1A / Figure 1B , which can be implemented as shown in Figure 1AThe network environment 100 shown is established. It will be recognized that the method 270 can be performed in any suitable environment, and Figure 2D Any of the blocks in the method 270 can be performed in any suitable order.

[0103] At step 272, the computing device 130 receives a set of images associated with a plurality of categories. Each image is labeled with one category.

[0104] At step 274, the computing device 130 determines a total number of images in each of the plurality of categories.

[0105] At step 276, the computing device 130 determines a first category of the plurality of categories that has a maximum number of images.

[0106] At step 278, the computing device 130 determines a weight associated with a respective category based on a ratio of the maximum number of images in the first category to a number of images in one of the plurality of categories.

[0107] At step 280, the computing device 130 updates the model each iteration based on the weights associated with the plurality of categories.

[0108] The trained model can be used by the computing device 130 to perform step 230 of the method 200 as shown. Figure 2A

[0109] At step 240, the computing device 130 determines a safety compliance of one or more objects based on the identification and classification results from steps 220 and 230. An exemplary result of performing step 240 using the computing device 130 is shown in block 240 of Figure 2B

[0110] By performing steps 220 and 230 of the method 200, the computing device 130 can obtain additional information associated with the images, such as worker IDs and PPE categories, which can be added as additional rows to the table 400 as shown. Figure 4 Figure 8 The table 800 is a table that presents information associated with a plurality of images for monitoring safety compliance at a worksite. As shown in Figure 8 The table 800 includes an image row 410, a timestamp row 420, and a camera index row 430, which rows are also included in the table 400. After performing steps 220 and 230, the computing device 130 can determine a worker ID 810 and a PPE category 820, and then add corresponding rows to the table 800. In some variations, the computing device 130 can add a label to the image indicating the PPE category 820 corresponding to the image, such as 822 and 824.

[0111] ​​​As described above, an identifier and PPE status can be output for each worker. For this purpose, the results for all workers can be aggregated to monitor on-site safety compliance, for example, by determining whether each worker is correctly wearing the necessary PPE on site. Any incidents of lost PPE can be logged and reported periodically. In some examples, safety reports can be generated periodically, such as... Figure 2B The report in box 240 shown is intended for field staff to review worker behavior on-site and design remedial measures to enhance safety awareness.

[0112] Furthermore, the determined security status can be used in various ways (e.g., such as...). Figure 2B (See the safety report shown). In one example, on-site warnings can be generated based on monitoring of the safety status. For example, when an incident of missing PPE is detected, an alarm, flashing light, or a combination thereof can be triggered in real time to warn workers of non-compliance. Different warning levels can be predefined based on the severity of the incident. In another example, warning messages can be generated based on monitoring of the safety status. Examples of warning messages include, but are not limited to, messages via messenger apps (APPs) on mobile phones (such as the WhatsApp Messenger APP), system-generated emails with screenshots capturing any incidents, etc. Warning messages can be used to notify off-site personnel or safety managers of potential risks on-site, allowing for more thorough monitoring of on-site worker behavior. In yet another example, historical data can be collected for statistical analysis of trends in detected events, for example, to generate daily or monthly summaries of each event type in each specific area or project site. Statistical analysis can provide contractors with guidance to plan long-term remedial actions, thereby changing worker behavior and mitigating on-site hazards.

[0113] In some examples, computing device 130 can run software developed to facilitate ReID and PPE classification capabilities, thereby automating the field monitoring process. By running the software, computing device 130 can enable a user interface that presents the overall workflow and functionality for field analysis in practice. Figure 9Ais a screenshot of an exemplary user interface 900 of the on-site monitoring software. The software can run on various types of computing devices, preferably with accelerated computing resources such as a graphics processing unit (GPU). First, the user can select a data folder for storing the raw input videos, and a folder for saving the generated outputs. The software can provide the user with options to define the on- and off-work hours. The on-work hours can be used to selectively process videos within certain time periods. For example, if 12:00 to 13:00 is lunch time when the on-site operation is mostly paused, the videos within this time period can be omitted. The software can also allow the user to define the time span and type of event reports to be generated. For example, a time period of 1 month can be entered so that videos within this time period are analyzed. Any events like workers without wearing PPE are recorded, where the event statistics can be summarized on a daily, weekly, or monthly basis. For the visual inspection of the safety personnel, the video frames capturing the events are extracted to a local database. To protect the privacy of the workers, their faces can be automatically blurred when saving the video frames. By clicking the “Run” button on the user interface, the video processing starts and the results are automatically saved.

[0114] The workflow can be designed for a more practical event counting, the pseudo code of which is summarized in Table 1. The computing device 130 can perform the process of event counting based on the code as shown in Table 1.

[0115] Table 1. Pseudo code of event counting.

[0116] Input:

[0117] 1. All processed bounding boxes, each bounding box labeled with a timestamp t, an identity i, a PPE status p

[0118] 2. A time threshold T b , T r

[0119] For each identity i:

[0120]

[0121] After the computing device 130 processes video frames from multiple cameras 120 by executing steps 210 to 230 of method 200, the computing device 130 can obtain a set of bounding boxes, each bounding box being labeled with a timestamp (“t”), worker identifier (“i”), and PPE status (“p”). The computing device 130 can then perform event counting for each individual worker. For example, the computing device 130 can extract the boxes associated with each worker and sort the extracted boxes in chronological order. For each worker, a state tracker (“State” in Table 1) and an event counter (“Count” in Table 1) are initialized. By processing the labeling information (t, i, and p) of each bounding box, the state tracker and event counter are updated when the criteria for identifying an event are met.

[0122] The computing device 130 can determine how to count events based on two parameters, which are (1) minimum buffer time T. b and (2) the maximum reset time T r . Figure 9B This illustrates the execution by the computing device 130 of a function based on two parameters T. b and T r An exemplary process 920 for counting controlled events. (e.g.) Figure 9B As shown, boxes 930 associated with workers are sorted along time axis 932. As shown in box 934, for example via... Figure 9A The user interface 900 shown will use parameter T. b and T r Set to a predefined value.

[0123] Parameter T b A minimum period is applied as a buffer time before counting boxes in an event begins. Minimum buffer time T b The aim is to make event logging more reasonable by inserting a delayed counting mechanism, as workers can, for example, remove their helmets briefly under safe environmental conditions to adjust their fit. Without a buffer time, penalizing workers for not wearing PPE in every frame might be too harsh for compliance monitoring. As an example of event determination, computing device 130 can count the number of consecutive frames 930, each frame associated with a “PPE-free” state identified in the corresponding frame 930, and determine the cumulative time period based on the timestamps of the counted frames 930. When the cumulative time period is greater than or equal to T... b At that time, the computing device 130 can determine the number of consecutive frames 930 and the examples of events associated with them. For example... Figure 9B As shown, computing device 130 can determine first event 940 and second event 946. Computing device 130 can determine that example 944 is not an event because the cumulative time period corresponding to example 944 is shorter than the minimum buffer time T.b In the pseudocode shown in Table 1, a timer t b is set to track the duration of a worker not wearing any necessary PPE. When the timer t b reaches a minimum buffer time (i.e., t b ≥ T b ), the status of the corresponding worker is changed to “positive” and the event count is incremented. Otherwise (i.e., t b < T b ), the status is returned to “negative”. Once the computing device 130 identifies the next box with the “no PPE” status, the counting of the boxes for the next event can begin.

[0124] Another parameter T r is used to decide whether to count two events separately if there is a time gap in between. A maximum time gap defines a reasonable duration beyond which an event is terminated. For example, if the computing device 130 does not observe the “no PPE” status for a certain period of time, the computing device 130 can reset the status tracker to the initial value of “negative”. As Figure 9B shown, the computing device 130 can maintain the “positive” status of the status tracker when counting the boxes associated with event 1 940. Then, the computing device 130 can not be able to detect the worker for a time gap. For example, the worker can leave the monitored scene (e.g., go back to an on-site office that does not require PPE), such that the PPE status is unknown for that time gap. Upon the worker’s reappearance, any missing PPE behavior should be unrelated to the previous event and thus counted as a new behavior. As such, if two boxes with the “positive” status are separated by a pre-defined time gap (i.e., t - t curr ≥ T r ), the counter for the current event is reset and any boxes with the “positive” status in future timestamps will likely be counted as another event when the minimum buffer criteria is met. Otherwise (i.e., t - t curr < T r ), the computing device 130 can continue to count the boxes with the “positive” status as being included in the current event. For example, as Figure 9B shown, event 2 946 includes a time gap that is less than the maximum reset time T r .

[0125] Figure 2E is a flowchart of a method 290 performed by the computing device 130 for monitoring site safety compliance by implementing the techniques described in step 240. It will be recognized that the method 290 can be performed in any suitable environment, and Figure 2E any of the blocks in the method 290 can be performed in any suitable order.

[0126] At step 292, the computing device 130 obtains a plurality of images. Each image is associated with a timestamp, an identification, and a safety status, e.g., as shown in Table 800 in Figure 8

[0127] At step 294, the computing device 130 determines a subset of images from the plurality of images as being associated with the event. The subset of images is associated with a time period that is greater than or equal to a minimum buffer time threshold. Further, the subset of images can include one or more time gaps, each time gap being shorter than a time gap threshold.

[0128] Further details and advantages related to the exemplary embodiments of the present disclosure are discussed in Cheng, J. P., Wong, P. K. Y., Luo, H., Wang, M., & Leung, P. H. “Vision-based monitoring of site safety compliance based on worker re-identification and personal protective equipment classification,” Automation in Construction, 139, 104312 (available at doi.org / 10.1016 / j.autcon.2022.104312), the entirety of which is incorporated herein by reference.

[0129] Note that the techniques described herein can be embodied in executable instructions stored in a computer-readable medium for use by or in connection with a processor-based instruction execution machine, system, device, or apparatus. Those skilled in the art will appreciate that the various types of computer-readable media can be included for storing data in some embodiments. As used herein, “computer-readable medium” includes one or more of any suitable media for storing the executable instructions for use by or in connection with an instruction execution machine, system, device, or apparatus, such that the machine, system, device, or apparatus can read (or fetch) the instructions from the computer-readable medium and execute the instructions for performing the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random access memories (RAM); read only memories (ROM); erasable programmable read only memories (EPROM); flash memory devices; and optical storage devices, including portable compact discs (CD), portable digital video discs (DVD), and the like.

[0130] ​It should be appreciated that the arrangement of components shown in the drawings is for illustrative purposes only and that other arrangements are possible. For example, one or more of the elements described herein can be implemented, in whole or in part, as an electronic hardware component. These elements can be implemented with software, hardware, or a combination of software and hardware. In addition, some or all of these other elements can be combined, some can be omitted altogether, and additional components can be added, while still achieving the functionality described herein. As such, the subject matter described herein can be implemented in a number of different variations and all such variations are considered to be within the scope of the claims.

[0131] To facilitate understanding of the subject matter described herein, many aspects are described in the order of acts. One skilled in the art will recognize that the various acts can be performed in a different order, or in parallel, or in different combinations than those described herein. The description herein of any act is not intended to be limiting, unless otherwise indicated or unless clearly contradicted by context. All methods described herein can be performed in any suitable order unless otherwise indicated or unless clearly contradicted by context.

[0132] The use of the terms “a” and “the” and “at least one of’ followed by a list of one or more items (for example, “one of A and B”) is to be understood as indicating one item from the list of items that is chosen or multiple items from the list of items that are chosen. As one example, “at least one of A and B” (or, equivalently, “one or more of A and B”) can be construed to mean: either A or B or both A and B. As another example, “at least one of A, B, and C” (or, equivalently, “one or more of A, B, and C”) can be construed to mean: either A or B or C or A and B or A and C or B and C or A and B and C or any permutations of A, B, and C. As a further example, “at least one of A, B, and C” (or, equivalently, “one or more of A, B, and C”) can be construed to mean: one or more instances of A or one or more instances of B or one or more instances of C. As a still further example, “at least one of A, B, and C” (or, equivalently, “one or more of A, B, and C”) can be construed to mean: at least one instance of A and at least one instance of B and at least one instance of C.

Claims

1. A system for monitoring compliance with the wearing of personal protective equipment, comprising: Multiple cameras are configured to acquire data, which includes multiple images associated with one or more objects; as well as A computing system configured to process the data to determine, based on the implementation of a first model trained for re-identification, the compliance of one or more objects associated with the plurality of images in wearing personal protective equipment. The computing system is further configured to train the first model for re-identification, wherein training the first model for re-identification includes: Obtain a first set of images, each of which includes an object and is labeled with an identifier associated with the object; Using the first model, a similarity matrix is ​​determined based on the features extracted from the images, including the similarity scores of image pairs in the first group of images; Based on the identifiers marked on the first set of images, a true similarity matrix is ​​determined, which includes the true similarity scores of the image pairs in the first set of images; The comparison matrix is ​​determined by performing element-wise comparisons between the similarity matrix and the true similarity matrix; The similarity loss is determined based on the element with the maximum value in the comparison matrix; and The first model is updated based on the similarity loss. The true similarity score of image pairs in the first group of images is determined based on a smoothing parameter P, where P is adjustable between 0 and 1; and In this study, each pair of images with the same label is assigned P as the corresponding true similarity score, and each pair of images with different labels is assigned 1-P as the corresponding true similarity score.

2. The system according to claim 1, wherein, Each pair of images in the first set of images includes a first image and a second image, and Determining the similarity matrix, which includes the similarity scores of image pairs in the first group of images, further includes: For each pair of images, determine the first feature vector of the corresponding first image and the second feature vector of the corresponding second image; and Based on the first feature vector and the second feature vector, cosine similarity is determined as the similarity score of the image pair, wherein the cosine similarity is obtained by dividing the product of the first feature vector and the second feature vector by the absolute values ​​of the first feature vector and the second feature vector.

3. The system according to claim 1, wherein, The comparison matrix is ​​further determined by performing element-wise comparisons between the similarity matrix and the true similarity matrix: Determine the absolute difference or squared difference between each similarity score in the similarity matrix and the corresponding true similarity score in the true similarity matrix, wherein the similarity score and the corresponding true similarity score are associated with the same image pair.

4. The system according to claim 1, wherein, The computing system is also configured to train a second model for classifying personal protective equipment, the training of which includes: Receive a second set of images associated with multiple categories, wherein each image in the second set of images is labeled with a category; Determine the total number of images in each of the plurality of categories; Determine the first category among the plurality of categories that has the largest number of images; A weight associated with a category is determined based on the ratio of the maximum number of images in the first category to the number of images in any of the plurality of categories; and Each iteration updates the second model based on the weights associated with the multiple categories.

5. The system according to claim 4, wherein, During the training of the second model, weights associated with a category are assigned to all images in that category; as well as In this case, the loss calculation for each iteration is based on the weights associated with the images in the second set of images.

6. The system according to claim 4, wherein, Processing the data to determine the compliance of the one or more objects associated with the plurality of images in wearing personal protective equipment also includes: By using the first model, one or more objects are identified based on features extracted from the plurality of images in the data; By using the second model, the one or more objects are classified based on the features extracted from the plurality of images in the data; and Based on the identification and classification results, the compliance of the one or more objects wearing personal protective equipment is determined.

7. The system according to claim 6, wherein, The identification and classification results include the plurality of images, and each of the plurality of images is associated with a timestamp, identifier, and security status; and The computing system is further configured to: determine a subset of images from the plurality of images associated with an event, wherein the subset of images is in a category indicating a non-safety condition, the subset of images is associated with a time period greater than or equal to a minimum buffer time threshold, and the subset of images includes one or more time gaps, each time gap being shorter than a time gap threshold.

8. The system according to claim 7, wherein, The computing system is also configured to: Generate logs including detected events; and Generate a warning corresponding to the detected event, wherein the warning includes any one of a warning message, an alarm, or a flashing light.

9. A method for monitoring compliance with the wearing of personal protective equipment, comprising: The computing system acquires data from multiple cameras, the data including multiple images associated with one or more objects; as well as The computing system processes the data to determine, based on the implementation of a first model trained for re-identification, the compliance of one or more objects associated with the plurality of images in wearing personal protective equipment. The first model is trained for re-identification based on the following steps: Obtain a first set of images, each of which includes an object and is labeled with an identifier associated with the object; Using the first model, a similarity matrix is ​​determined based on the features extracted from the images, including the similarity scores of image pairs in the first group of images; Based on the identifiers marked on the first set of images, a true similarity matrix is ​​determined, which includes the true similarity scores of the image pairs in the first set of images; The comparison matrix is ​​determined by performing element-wise comparisons between the similarity matrix and the true similarity matrix; The similarity loss is determined based on the element with the maximum value in the comparison matrix; and The first model is updated based on the similarity loss. The true similarity score of image pairs in the first group of images is determined based on a smoothing parameter P, where P is adjustable between 0 and 1; and In this study, each pair of images with the same label is assigned P as the corresponding true similarity score, and each pair of images with different labels is assigned 1-P as the corresponding true similarity score.

10. The method of claim 9, further comprising training a second model for classification, wherein, The second model trained for classifying personal protective equipment includes: Receive a second set of images associated with multiple categories, wherein each image in the second set of images is labeled with a category; Determine the total number of images in each of the plurality of categories; Determine the first category among the plurality of categories that has the largest number of images; A weight associated with a category is determined based on the ratio of the maximum number of images in the first category to the number of images in any of the plurality of categories; and Each iteration updates the second model based on the weights associated with the multiple categories.

11. The method according to claim 10, wherein, During the training of the second model, weights associated with a category are assigned to all images in that category; as well as In this case, the loss calculation for each iteration is based on the weights associated with the images in the second set of images.

12. The method according to claim 10, wherein, Processing the data to determine the compliance of the one or more objects associated with the plurality of images in wearing personal protective equipment also includes: By using the first model, one or more objects are identified based on features extracted from the plurality of images in the data; By using the second model, the one or more objects are classified based on the features extracted from the plurality of images in the data; and Based on the identification and classification results, the compliance of the one or more objects wearing personal protective equipment is determined.

13. The method according to claim 12, wherein, The identification and classification results include the plurality of images, and each of the plurality of images is associated with a timestamp, an identifier, and a security status; as well as The method further includes: determining a subset of images from the plurality of images associated with an event, wherein the subset of images is in a category indicating a non-safety condition, wherein the subset of images is associated with a time period greater than or equal to a minimum buffer time threshold, and the subset of images includes one or more time gaps, each time gap being shorter than a time gap threshold.

14. A non-transitory computer-readable medium having processor-executable instructions stored thereon for monitoring compliance with the wearing of personal protective equipment, wherein, When the processor-executable instructions are executed, they cause: Data is acquired from multiple cameras, the data including multiple images associated with one or more objects; as well as The data is processed to determine, based on the implementation of a first model trained for re-identification, the compliance of one or more objects associated with the plurality of images wearing personal protective equipment. The first model is trained for re-identification based on the following steps: Obtain a first set of images, each of which includes an object and is labeled with an identifier associated with the object; Using the first model, a similarity matrix is ​​determined based on the features extracted from the images, including the similarity scores of image pairs in the first group of images; Based on the identifiers marked on the first set of images, a true similarity matrix is ​​determined, which includes the true similarity scores of the image pairs in the first set of images; The comparison matrix is ​​determined by performing element-wise comparisons between the similarity matrix and the true similarity matrix; The similarity loss is determined based on the element with the maximum value in the comparison matrix; and The first model is updated based on the similarity loss. The true similarity score of image pairs in the first group of images is determined based on a smoothing parameter P, where P is adjustable between 0 and 1; and In this study, each pair of images with the same label is assigned P as the corresponding true similarity score, and each pair of images with different labels is assigned 1-P as the corresponding true similarity score.

15. The non-transitory computer-readable medium according to claim 14, wherein, When the processor-executable instructions are executed, they also cause: training a second model for classifying personal protective equipment, wherein training the second model for classification includes: Receive a second set of images associated with multiple categories, wherein each image in the second set of images is labeled with a category; Determine the total number of images in each of the plurality of categories; Determine the first category among the plurality of categories that has the largest number of images; A weight associated with a category is determined based on the ratio of the maximum number of images in the first category to the number of images in any of the plurality of categories; and Each iteration updates the second model based on the weights associated with the multiple categories.

16. The non-transitory computer-readable medium according to claim 15, wherein, During training, weights associated with a category are assigned to all images in that category; as well as In this case, the loss calculation for each iteration is based on the weights associated with the images in the second set of images.

17. The non-transitory computer-readable medium according to claim 15, wherein, Processing the data to determine the compliance of the one or more objects associated with the plurality of images in wearing personal protective equipment also includes: By using the first model, one or more objects are identified based on features extracted from the plurality of images in the data; By using the second model, the one or more objects are classified based on the features extracted from the plurality of images in the data; and Based on the identification and classification results, the compliance of the one or more objects wearing personal protective equipment is determined.

18. The non-transitory computer-readable medium according to claim 17, wherein, The identification and classification results include the plurality of images, and each of the plurality of images is associated with a timestamp, an identifier, and a security status; as well as When executed, the processor-executable instructions further cause the determination of a subset of images from the plurality of images to be associated with an event, wherein the subset of images is in a category indicating a non-safety state, the subset of images is associated with a time period greater than or equal to a minimum buffer time threshold, and the subset of images includes one or more time gaps, each time gap being shorter than a time gap threshold.

Images