Method for determining operation safety risk of operator and safety risk linkage system

Abstract

The invention provides a method for determining the operation safety risk of an operator and a safety risk linkage system, and the method comprises the steps: receiving the basic state information of a field operator and the equipment operation state of field operation equipment, which are transmitted by edge side equipment, the basic state information comprises identity information and behavior state information of field operating personnel; according to the basic state information of the field operation personnel, determining operation information of the field operation personnel; receiving physical sign detection information, sent by the terminal equipment, of the field operating personnel; and determining the operation safety risk of the field operation personnel according to at least one of the operation information of the field operation personnel, the physical sign detection information of the field operation personnel and the equipment operation state of the field operation equipment. According to the method, through cooperative computing of the cloud end and the edge side, the computing pressure of the cloud end is relieved, the circulation efficiency of data and states is improved, and the safety and response capability of a working site are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of operation safety monitoring, and more specifically, to a method for determining the operation safety risk of an operator, an operation site safety risk linkage system, a computer-readable storage medium, and an electronic device. Background Art

[0002] Safety management at construction sites has always relied on the supervision and urging of safety officers. In addition, workers are trained to develop safety awareness to avoid accidents. However, this traditional manual management method leads to many blind spots in supervision, and some hidden dangers cannot be discovered and rectified in time, resulting in various safety accidents.

[0003] In practical applications, on-site operation monitoring requires the calculation and storage of a large amount of real-time data, and at the same time, it must be able to meet timely communication, which places high requirements on local devices. With the advancement of technology, the cloud computing model has been proposed and has achieved rapid development. Through cloud computing technology, the data of on-site monitoring can be uploaded to the cloud, and the super-powerful and efficient computing platform of the cloud computing center can be used to centrally process the data of on-site monitoring. However, before cloud computing can centrally process data, all edge devices need to upload all data, which means that the data transmission process occupies a huge bandwidth, and during processing, the huge data required for model training and reasoning of the processing algorithm increases the pressure on the core network, making the growth of data far outstrip the growth rate of network bandwidth. The traditional cloud computing model has encountered bottlenecks in bandwidth and latency, and cannot effectively meet the real-time requirements of construction site operation monitoring. Summary of the invention

[0004] The main purpose of this application is to provide a method for determining the work safety risks of workers, a work site safety risk linkage system, a computer-readable storage medium and an electronic device, so as to at least solve the problem that the data transmission process of the traditional cloud computing model in the prior art requires a large bandwidth, and the data is large during data processing, resulting in data delays and unable to meet the real-time requirements of construction site work monitoring.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for determining the operation safety risk of an operator is provided, which is applied to a cloud device of an operation site safety risk linkage system, wherein the operation site safety risk linkage system also includes an edge side device and a terminal device, and the cloud device, the edge side device and the terminal device are communicatively connected, and the method includes: receiving basic status information of the on-site operator and the equipment operation status of the on-site operation equipment sent by the edge side device, the basic status information including the identity information and behavior status information of the on-site operator, the identity information is identification information that uniquely identifies the on-site operator, and the behavior status information at least includes the action information of the on-site operator; determining the operation operation information of the on-site operator according to the basic status information of the on-site operator, the operation operation information characterizing whether the operation operation of the on-site operator is in violation of regulations; receiving the vital sign detection information of the on-site operator sent by the terminal device, the vital sign detection information at least including the vital sign information of the on-site operator; determining the operation safety risk of the on-site operator according to at least one of the operation operation information of the on-site operator, the vital sign detection information of the on-site operator and the equipment operation status of the on-site operation equipment.

[0006] Optionally, determining the work operation information of the field worker based on the basic status information of the field worker includes: obtaining the facial data and identity data of the field worker in the basic status information, the identity data including the fingerprint data of the field worker and / or the identity card data of the field worker; performing facial recognition on the field worker based on the facial data of the field worker to obtain a facial recognition result, and matching the facial recognition result with the identity data; when the facial recognition result of the field worker successfully matches the identity data, determining the work operation information of the field worker based on the facial data of the field worker and / or the behavioral status information of the field worker.

[0007] Optionally, face recognition is performed on the on-site workers according to the facial data of the on-site workers to obtain a face recognition result, including: converting the face image to be tested collected by the edge side device into a grayscale image, and adjusting the size of the grayscale image to a preset size to obtain a first face image; convolving the first face image with the first layer of convolution filters respectively to obtain an output feature map of the first feature extraction layer as a first feature map; convolving the first feature map with the second layer of convolution filters respectively to obtain an output feature map of the second feature extraction layer as a second feature map; performing binary hash coding on the second feature map, statistically calculating local area histograms, and concatenating the statistical histograms of all local areas to obtain final output features of the first face image; inputting the final output features into the trained Linear SVM classifier for classification to obtain the face recognition result.

[0008] Optionally, the operating information of the on-site operator is determined based on the facial data of the on-site operator and / or the behavioral status information of the on-site operator, including: determining the emotional characteristics of the on-site operator based on the facial data of the on-site operator, and identifying the emotions of the on-site operator based on the emotional characteristics of the on-site operator to obtain the emotion recognition result of the on-site operator; determining the wearing status of the safety facilities of the on-site operator and the operating behavior of the on-site operator based on the behavioral status information of the on-site operator, the safety facilities being protective facilities worn by the on-site operator to reduce the operating safety risks of the on-site operator, and the operating behavior being the behavior of the on-site operator performing operating operations on the on-site operating equipment; determining whether the operating operation of the on-site operator is in violation of regulations based on at least one of the emotional recognition result of the on-site operator, the wearing status of the safety facilities of the on-site operator, and the operating behavior of the on-site operator, so as to determine the operating information of the on-site operator.

[0009] Optionally, based on the facial data of the on-site workers, the emotional characteristics of the on-site workers are determined, and the emotions of the on-site workers are identified based on the emotional characteristics of the on-site workers to obtain the emotion recognition results of the on-site workers, including: using a face detection algorithm to detect video frame pictures including the facial images of the on-site workers in the facial data, and cropping the video frame pictures to obtain the facial images of the on-site workers; inputting the facial images into a fully convolutional neural network to generate a feature map of the facial image, and determining the feature map as the facial expression characteristics of the on-site workers; performing feature matching on the facial expression characteristics with preset facial features corresponding to each emotion label to obtain multiple matching degrees, and using the emotion label corresponding to the preset facial feature with the highest matching degree as the emotion label of the facial expression characteristics to obtain the emotion recognition results of the on-site workers.

[0010] Optionally, according to the behavioral status information of the on-site workers, the wearing status of the safety facilities of the on-site workers is determined, including: obtaining a plurality of full-body original images of the on-site workers, performing data preprocessing on the full-body original images to obtain a plurality of preprocessed full-body images, wherein the data preprocessing includes at least one of the following: data enhancement processing, adaptive anchor frame processing, and adaptive image scaling processing; inputting all of the preprocessed full-body images into a target detection model to obtain the wearing status of the safety facilities of the on-site workers, wherein the target detection model is obtained based on training of the YOLOv5s network and the Res2Net network, and the wearing status of the safety facilities of the on-site workers is correct wearing or incorrect wearing.

[0011] Optionally, the working behavior of the on-site worker is determined based on the behavioral status information of the on-site worker, including: receiving initial image data of the on-site worker collected by the edge-side device; preprocessing the initial image data to obtain processed image data, and extracting image feature parameters of the processed image data; inputting the image feature parameters into a value classification model for identification to obtain the working behavior of the on-site worker, the working behavior including whether the on-site worker performs dangerous operations or does not perform the dangerous operations.

[0012] According to another aspect of the present application, a work site safety risk linkage system is provided, including: a cloud device, used to execute any one of the methods for determining the work safety risks of workers; an edge-side device, communicatively connected to the cloud device, and used to collect basic status information of on-site workers and equipment operating status of on-site work equipment; a terminal device, respectively communicatively connected to the cloud device and the edge-side device, and used to collect vital sign detection information of the on-site workers.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for determining the work safety risks of operators.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for determining the operational safety risks of an operator performing any one of the above-described operations.

[0015] Applying the technical solution of the present application, the above-mentioned method for determining the operation safety risk of the operator is applied to the cloud device of the operation site safety risk linkage system. The operation site safety risk linkage system also includes edge side devices and terminal devices. The cloud device, edge side devices and terminal devices are communicatively connected. The method first receives the basic status information of the on-site operators and the equipment operation status of the on-site equipment sent by the edge side devices. The basic status information includes the identity information and behavior status information of the on-site operators; then, the operation operation information of the on-site operators is determined based on the basic status information of the on-site operators; then, the vital sign detection information of the on-site operators sent by the terminal device is received; finally, the operation safety risk of the on-site operators is determined based on at least one of the operation operation information of the on-site operators, the vital sign detection information of the on-site operators and the equipment operation status of the on-site equipment. This method performs face detection, safety facility target detection, and equipment status monitoring through collaborative processing of the cloud and edge servers, and transmits the detection results and status monitoring data to the terminal in real time for display. At the same time, it promptly informs the operators of abnormal status (such as failure of operator identity authentication, abnormal status of on-site equipment, etc.) through the terminal, thereby minimizing safety hazards. This method reduces the computing pressure of the cloud through collaborative computing between the cloud and the edge, improves the efficiency of data and status circulation, and effectively improves the safety and responsiveness of the work site. It solves the problem that the data transmission process of the traditional cloud computing model in the prior art requires a large bandwidth, and the data is large during data processing, resulting in data delays, and cannot meet the real-time requirements of construction site operation monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for performing a method for determining a worker's work safety risk provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic flow chart of a method for determining the operation safety risk of an operator provided in accordance with an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of the structure of a device status monitoring system provided according to an embodiment of the present application is shown;

[0020] Figure 4 A schematic diagram of the structure of a work site safety risk linkage system provided according to an embodiment of the present application is shown;

[0021] Figure 5 A structural block diagram of a cloud device provided according to an embodiment of the present application is shown.

[0022] The above drawings include the following reference numerals:

[0023] 01. Cloud device; 02. Edge device; 03. Terminal device; 102. Processor; 104. Memory; 106. Transmission device; 108. Input and output device. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] As introduced in the background technology, the traditional cloud computing model has encountered bottlenecks in bandwidth and delay, and cannot effectively meet the needs of real-time monitoring of construction site operations. In order to solve the problem that the data transmission process of the traditional cloud computing model in the prior art requires a large bandwidth, and the data is large during data processing, resulting in data delays, and cannot meet the needs of real-time monitoring of construction site operations, the embodiments of the present application provide a method for determining the operation safety risks of operators, a work site safety risk linkage system, a computer-readable storage medium and an electronic device.

[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0029] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a method for determining the operation safety risk of an operator according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0030] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the safety risk of the operation of the operator in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] In this embodiment, a method for determining the work safety risks of workers running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0032] Figure 2 1 is a flow chart of a method for determining the operation safety risk of an operator according to an embodiment of the present application. The method is applied to a cloud device of a work site safety risk linkage system, the work site safety risk linkage system further includes an edge device and a terminal device, the cloud device, the edge device and the terminal device are in communication connection, such as Figure 2 As shown, the method comprises the following steps:

[0033] Step S201, receiving basic status information of the on-site workers and the equipment operation status of the on-site equipment sent by the edge side device, the basic status information including the identity information and behavior status information of the on-site workers, the identity information is identification information that uniquely identifies the on-site workers, and the behavior status information includes at least the action information of the on-site workers;

[0034] Specifically, the cloud is used to store and analyze data uploaded from the edge side of the work site, and to determine whether the on-site workers comply with safety regulations through face detection and target detection; the edge side is used to collect video, identity and status information of the workers and equipment on the work site, and combine the identity detection results with the face detection results on the cloud to determine whether the workers violate the rules, analyze the workers' behavior based on the video data, and determine whether the on-site equipment is faulty based on the status information of the equipment. The edge side is deployed with edge servers to provide computing resources and storage resources.

[0035] The above-mentioned edge side is deployed with video surveillance equipment, identity authentication equipment and equipment status monitoring system; the above-mentioned video surveillance equipment is used to record the video of on-site workers in real time, and the data is used to analyze the behavior of the workers; the above-mentioned identity authentication equipment is used to authenticate the workers, and is combined with the face detection results in the cloud to verify the uniqueness of the workers' identity; the above-mentioned equipment status monitoring system is used to monitor the operating status of the equipment at the work site, and promptly issue an alarm signal when the equipment is abnormal.

[0036] Among them, the identity information of the on-site workers can be used to identify the fingerprint data of the on-site workers through the fingerprint recognition device. The above-mentioned identity card authentication device is used to identify the identity card carried by the workers, and combine the fingerprint recognition results, identity card recognition results and cloud face detection results to verify the uniqueness of the workers' identity.

[0037] Step S202, determining the operation information of the on-site operator according to the basic status information of the on-site operator, wherein the operation information indicates whether the operation of the on-site operator is in violation of regulations;

[0038] Specifically, the operation information of the on-site operators is whether the on-site operators operate the on-site operation equipment in a standardized manner.

[0039] Wherein, determining the operation information of the on-site workers according to the basic status information of the on-site workers comprises the following steps:

[0040] Step S301, acquiring the face data and identity data of the on-site worker in the basic status information, wherein the identity data includes the fingerprint data of the on-site worker and / or the identity card data of the on-site worker;

[0041] Step S302, performing facial recognition on the on-site worker according to the facial data of the on-site worker to obtain a facial recognition result, and matching the facial recognition result with the identity data;

[0042] Among them, performing face recognition on the above-mentioned on-site workers according to the above-mentioned face data of the above-mentioned on-site workers to obtain face recognition results includes the following steps:

[0043] Step S3021, converting the face image to be tested acquired by the edge side device into a grayscale image, and adjusting the size of the grayscale image to a preset size to obtain a first face image;

[0044] Step S3022, convolving the first face image with the first layer of convolution filters respectively, to obtain an output feature map of the first feature extraction layer as a first feature map;

[0045] Step S3023, convolving the first feature map with the second layer of convolutional filters respectively, to obtain an output feature map of the second feature extraction layer as a second feature map;

[0046] Step S3024, performing binary hash coding on the second feature map, calculating local area histograms, and concatenating the statistical histograms of all local areas to obtain the final output feature of the first face image;

[0047] Step S3025, input the above-mentioned final output features into the trained Linear SVM classifier for classification to obtain the above-mentioned face recognition result.

[0048] Specifically, the cloud uniquely verifies the identity of the operator based on the facial image collected by the edge-side device.

[0049] Among them, in the above cloud, the method of performing face detection includes the following steps:

[0050] Step S1: convert the face image B to be tested collected from the edge side into a grayscale image, and adjust the image size to the same size p×q;

[0051] Step S2: Convolutional filter Convolution is performed to obtain the feature maps of the first layer.

[0052] Step S3: The feature map As the original input of the second feature extraction layer, and the second layer of convolutional filters Convolve to get the output feature map of the second layer

[0053] Step S4: Feature map Perform binary hash coding, calculate the local area histogram, and concatenate the statistical histograms of all local areas as the final extracted features of face image B.

[0054] Step S5: Output the final features of the image The data is sent to the trained Linear SVM classifier for classification to obtain the classification result, i.e. the face recognition result.

[0055] First, the face detection model is constructed, including the following steps:

[0056] 1. Convert the training set samples into grayscale images and adjust the image size to the same size p×q;

[0057] 2. Send the training samples to the first feature extraction layer of the GB(2D)2PCANet model in sequence to obtain the Gabor feature image of the first feature extraction layer; specifically, let represents the face image training set, where N is the number of samples in the training set, Represents a face sample image; each sample A i The first feature extraction layer is fed sequentially, and a 2D Gabor filter combining 5 scales and 8 directions is first performed. The final Gabor feature image is obtained by downsampling, which is recorded as in s represents the number of pixels after downsampling of the feature image, and t represents the number of 2D Gabor filters;

[0058] 3. For each Gabor feature image B i , scan and extract m×n image blocks of size l1×l2, perform a mean removal operation on the image blocks, and obtain in Indicates B i The jth de-meaning image block in ; after all Gabor feature images are processed in the same way, the sample matrix is ​​obtained:

[0059]

[0060] For the convenience of description, all image blocks in I are represented by consecutive serial numbers and recorded as

[0061] 4. Using the feature projection vector analysis method based on (2D) 2PCA, the optimal projection axis of the sample matrix I is extracted from both the row and column directions as the convolution filter in the first layer feature extraction stage. N1 represents the number of convolutional filters in the first layer;

[0062] The feature projection vector analysis method specifically includes the following steps:

[0063] 1) Assume that the number of convolutional filters in the first layer is N1 (taking the AR face database as an example, N1=3); for each image block in the sample matrix I From the direction of the row, find the optimal projection axis The row covariance matrix is ​​defined as follows:

[0064] in, is the average value of all training samples;

[0065] The row covariance matrix G row Perform KL decomposition, By G row The eigenvectors corresponding to the first N1 largest eigenvalues ​​of

[0066] 2) In the same direction, find the optimal projection axis The column covariance matrix is ​​defined as follows: in, By the column covariance matrix G col The eigenvectors corresponding to the first N1 largest eigenvalues ​​of

[0067] 3) Calculate N1 convolution filters: W n =Y n X n T ,n=1,2,…,N1.

[0068] 5. The convolution filter learned in step 4 Original face images from the training set Convolve separately to get N×N1 feature maps in

[0069] 6. Each training sample A obtained in step 5 i The corresponding feature map As the input of the second feature extraction layer, the convolution filters of the second layer (face distinguishing features) are learned in sequence using the same feature learning method as steps 2 to 4. N2 is the number of convolutional filters in the second layer; and convolutional filters are used The feature map obtained in step 5 Convolve separately to obtain N×N1×N2 second-layer feature maps in

[0070] 7. For each training sample A obtained in step 6 i The corresponding second layer feature map Binary hash coding to obtain binary feature map Specifically, we first use the binary hash function H(·) to convert Binarization, where H(·) is 1 when the input is greater than 0, and H(·) is 0 when the input is less than or equal to 0; then all the inputs from the second layer (The first layer consists of training samples A i The N2 binary feature maps obtained by the secondary convolution are taken as a group, and the binary vectors composed of the binary numbers at the same pixel position of these N2 binary feature maps are converted into decimal numbers to obtain an integer value output map. where i∈[1,N], n∈[1,N1]; so a single sample A i Finally, N1 binary feature maps are generated All samples Finally, we get N×N1 binary feature maps

[0071] 8. For each binary feature map Take blocks of size [b1 b2] in the form of a sliding window, with an overlap ratio of α, and calculate the statistical histogram of each block, denoted as Then all the samples A i Generated N1 binary feature maps The statistical histograms of the local areas are spliced ​​together to obtain the face image A i The final output features are:

[0072] 9. All samples obtained in step 8 Output characteristics The data is sent to the Linear SVM classifier for training, and the optimal Linear SVM face detection model based on GB(2D)2PCANet is obtained.

[0073] Step S303, when the face recognition result of the on-site worker successfully matches the identity data, determine the operation information of the on-site worker according to the face data of the on-site worker and / or the behavior status information of the on-site worker.

[0074] Specifically, the fingerprint recognition results, identity card recognition results and cloud face detection results are combined to verify the uniqueness of the operator's identity.

[0075] Among them, by deploying edge servers, video surveillance equipment, identity authentication equipment and various equipment status monitoring systems on the edge side, video collection and identity authentication of on-site workers are realized, and the operating status of on-site equipment is monitored. Among them, the collected video data is uploaded to the cloud for face detection and target recognition, and is used for personnel behavior recognition on the edge side; and the identity authentication equipment includes a fingerprint recognition device and an identity card authentication device. The fingerprint recognition device is used to identify the fingerprint data of on-site workers, and the identity card authentication device is used to identify the identity card carried by the worker, and the fingerprint recognition results, the identity card recognition results and the cloud face detection results are combined, so as to realize the uniqueness of the identity of the worker when the worker enters the site at the beginning of the operation, and avoid impersonation between workers.

[0076] Step S203, receiving the physical sign detection information of the on-site worker sent by the terminal device, wherein the physical sign detection information at least includes the vital sign information of the on-site worker;

[0077] Among them, the terminal is used to receive alarm information sent from the cloud or edge side, and at the same time receive detection results from the cloud and detection results from the edge side. In addition, the above-mentioned terminal also monitors the location and status of the operating personnel.

[0078] Step S204, determining the operational safety risk of the on-site operator based on at least one of the operational information of the on-site operator, the vital sign detection information of the on-site operator, and the equipment operating status of the on-site operation equipment.

[0079] Wherein, determining the operation information of the on-site operator according to the facial data of the on-site operator and / or the behavior status information of the on-site operator includes the following steps:

[0080] Step S401, determining the emotional characteristics of the on-site workers according to the facial data of the on-site workers, and identifying the emotions of the on-site workers according to the emotional characteristics of the on-site workers, to obtain the emotion recognition results of the on-site workers;

[0081] The method of determining the emotional characteristics of the on-site workers according to the facial data of the on-site workers, and identifying the emotions of the on-site workers according to the emotional characteristics of the on-site workers to obtain the emotion recognition results of the on-site workers includes the following steps:

[0082] Step S4011, using a face detection algorithm to detect the video frame images including the face images of the on-site workers in the face data, and cropping the video frame images to obtain the face images of the on-site workers;

[0083] Step S4012, inputting the facial image into a fully convolutional neural network to generate a feature map of the facial image, and determining the feature map as the facial expression features of the on-site operator;

[0084] Step S4013, feature matching is performed on the above-mentioned facial expression features with the preset facial features corresponding to each emotion label to obtain multiple matching degrees, and the above-mentioned emotion label corresponding to the above-mentioned preset facial feature with the highest matching degree is used as the emotion label of the above-mentioned facial expression feature to obtain the emotion recognition result of the above-mentioned on-site operator.

[0085] Specifically, the emotion feature is matched with the pre-extracted facial expression feature, and the emotion label corresponding to the facial expression feature with the highest matching degree is selected as the recognized emotion, thereby realizing the recognition of the operator's emotion.

[0086] The above-mentioned analysis of operator behavior based on video data includes:

[0087] 1) Use the face detection algorithm to detect the face of the video frame image, and crop the detected face to obtain a face image;

[0088] 2) Input the face image in the first frame as the template input, and then input the face image in the subsequent frame as the input to be detected; the two inputs are respectively subjected to feature mapping through the weight-sharing full convolutional neural network operation; the template input is convolved with the convolution kernel to obtain the template feature, and the input to be detected is convolved with the convolution kernel to obtain the feature to be detected;

[0089] 3) Divide the template features into k parts in order on the convolution channel, and regard them as k convolution kernels; perform convolution processing on the k convolution kernels and the features to be detected to generate a feature map with k convolution channels, and use the generated feature map as the facial expression feature;

[0090] 4) Matching the emotion feature with the pre-extracted facial expression feature, and selecting the emotion label corresponding to the facial expression feature with the highest matching degree as the recognized emotion, thereby realizing the recognition of the operator's emotion;

[0091] 5) Preprocessing the image in the video frame, extracting the image feature parameters in the preprocessed image, and sending them into the recurrent neural network to obtain the mapping between the image feature parameters and the high-dimensional vector;

[0092] 6) Based on the high-dimensional vector of the video frame, a classifier model is established, a mapping from the high-dimensional vector to the final irregular behavior category is established, and the classifier model is trained to identify the operator's satisfaction and continuation through the trained classifier model;

[0093] 7) Conduct a comprehensive analysis based on the emotion recognition results and behavior recognition results to determine the behavior of the operator and thus judge whether the operator has performed any dangerous operations.

[0094] Step S402, determining the wearing status of the safety equipment of the on-site workers and the operation behavior of the on-site workers according to the behavior status information of the on-site workers, wherein the safety equipment refers to the protective equipment worn by the on-site workers to reduce the operation safety risk of the on-site workers, and the operation behavior refers to the operation performed by the on-site workers on the on-site operation equipment;

[0095] Wherein, determining the wearing status of the safety equipment of the above-mentioned on-site workers according to the behavior status information of the above-mentioned on-site workers includes the following steps:

[0096] Step S501, obtaining a plurality of full-body original images of the above-mentioned on-site workers, performing data preprocessing on the above-mentioned full-body original images to obtain a plurality of preprocessed full-body images, wherein the above-mentioned data preprocessing includes at least one of the following: data enhancement processing, adaptive anchor frame processing, and adaptive image scaling processing;

[0097] Step S502, all the above-mentioned pre-processed full-body images are input into the target detection model to obtain the wearing status of the safety facilities of the above-mentioned on-site workers. The above-mentioned target detection model is obtained based on the training of the YOLOv5s network and the Res2Net network. The wearing status of the safety facilities of the above-mentioned on-site workers is correct or incorrect.

[0098] Specifically, the full-body images of the workers collected on the edge side are detected based on the neural network model to determine whether the workers are wearing various safety facilities as required. It can accurately identify whether the safety equipment of the workers on site is worn correctly, among which the safety facilities include safety helmets, seat belts, etc.

[0099] By deploying models based on deep learning algorithms such as face detection models and target detection models in the cloud, the image data collected on the edge is processed to identify the faces of on-site workers and the implementation of safety measures. The implementation of safety measures includes wearing safety ropes, safety belts, helmets, smart bracelets, and breathing sensors.

[0100] The target detection is used to detect the wearing of safety equipment by on-site workers. The target detection includes the following steps:

[0101] 1) Obtain multiple full-body original images of on-site workers, perform preprocessing such as data enhancement, adaptive anchor frame, and adaptive image scaling on each image, and obtain multiple preprocessed images to form a model construction data set; use LabelImg to label the full-body images, and divide all labeled images into training sets and test sets;

[0102] 2) Based on the network structure of YOLOv5s, the Res2Net network is introduced, and all C3 modules in YOLOv5s are replaced with C3_Res2Block modules; the convolutional attention module CBAM is introduced and added to the previous layer of SPPF in the YOLOv5s network backbone; finally, the loss function CIOU used by YOLOv5s is replaced with the bounding box loss function EIOU to obtain an improved object detection model;

[0103] Among them, the YOLOv5 model is improved and Res2Net is introduced. All C3 modules in YOLOv5s are replaced with C3_Res2Block modules. Res2Net extracts multi-scale features in a finer granularity by convolution grouping. After passing through 1×1 convolution, the input features are evenly divided into s feature subsets and x i Represents, where i∈{1,2,...,s}. Each feature subset x i All have the same spatial size. Except for x1, each xi There are corresponding 3×3 convolutions, using K i to represent it, and its output is represented as y i .y i The representation is as follows:

[0104] In the Res2Net module, a new control parameter "scale" is used, represented by z. A larger s can make the model have a larger receptive field. Assume that the number of channels of input and output features is I c and O c , for the residual block, its parameter amount is 3×3×I c ×O c , that is, 9×I c ×O c , and for the Res2Net module, its parameter volume is (s-1)×9×(I c / s)(O c / s). Obviously, the Res2Net module has fewer parameters.

[0105] In Backbone, a lightweight attention module CBAM is added to the upper layer of SPPF. CBAM multiplies the output features of the channel attention module and the spatial attention module element by element to obtain the final attention enhancement feature. Input feature F∈R C ×H×W , followed by a channel attention module one-dimensional convolution M c ∈R C*1*1 , multiply the convolution result by the original image, take the CAM output result as input, and perform the two-dimensional convolution M of the spatial attention module s ∈R 1*H*W , and then multiply the output result by the original image, as shown in the following formula:

[0106]

[0107] This enhanced feature will be used as input to subsequent network layers to suppress noise and irrelevant information while retaining key information.

[0108] In order to solve the limitations of CIOU, the loss function in YOLOv5 is changed to the improved bounding box loss function EIOU. The EIOU loss function is composed of the overlap loss L IoU , center distance loss L dis and width-height loss L asp Composition, which is expressed as:

[0109]

[0110] Among them, c w and ch Respectively represent the width and height of the minimum bounding box covering the predicted box and the real box; w and w gt are the widths of the predicted box and the true box respectively; h and h gt are the lengths of the predicted box and the true box respectively; c is the diagonal distance of the minimum closure area that can contain both the predicted bounding box and the true bounding box. IoU is the intersection over union ratio of the predicted bounding box and the true bounding box. After the above steps, the improved model Trash-YOLO is obtained.

[0111] 3) The Trash-YOLO model in step 2 is trained according to the labeled data set in step 1. The training environment is Python-3.8.0torch-1.13.0+cu116 CUDA:0 (NVIDIAGeForce RTX 3060Laptop GPU, 6144MiB). The number of training rounds is set to 100, and the batch_size is set to 8. Finally, the trained model is obtained, and then the safety facilities worn by the operators are detected through the trained model.

[0112] The above bounding box loss function EIOU includes the overlap loss L IoU , center distance loss L dis and width-height loss L asp , as shown below:

[0113]

[0114] In the formula, c w and c h Respectively represent the width and height of the minimum bounding box covering the predicted box and the real box; w and w gt point

[0115] Indicates the width of the prediction box and the real box; h and h gt They represent the length of the predicted box and the true box respectively; c represents the diagonal distance of the minimum closure area that can contain both the predicted bounding box and the true bounding box; IoU represents the intersection over union ratio of the predicted bounding box to the true bounding box.

[0116] Wherein, determining the operation behavior of the above-mentioned on-site operation personnel according to the behavior status information of the above-mentioned on-site operation personnel includes the following steps:

[0117] Step S601, receiving the initial image data of the on-site worker acquired by the edge device;

[0118] Step S602, preprocessing the initial image data to obtain processed image data, and extracting image feature parameters of the processed image data;

[0119] Step S603: input the image feature parameters into a value classification model for identification to obtain the work behavior of the on-site workers. The work behavior includes whether the on-site workers have performed dangerous operations or not.

[0120] Specifically, a comprehensive analysis is conducted based on the emotion recognition results and the behavior recognition results to determine the behavior of the operator and thus judge whether the operator has performed any dangerous operations.

[0121] Step S403, determining whether the operation of the above-mentioned on-site workers is in violation of regulations based on at least one of the emotion recognition results of the above-mentioned on-site workers, the wearing of safety equipment of the above-mentioned on-site workers and the operation behavior of the above-mentioned on-site workers, so as to determine the above-mentioned operation information of the above-mentioned on-site workers.

[0122] Specifically, the cloud and the edge can communicate bidirectionally. The edge sends the collected video data, monitored equipment information, personnel status information and other data to the cloud for storage, and the cloud sends the detection results such as face detection and target detection to the edge for combined verification. The cloud can send various stored data to the terminal and display them through the terminal to facilitate the management of on-site personnel. The edge can send its real-time status monitoring data and alarm information in abnormal situations to the terminal to facilitate emergency response by on-site personnel.

[0123] In this embodiment, a smart terminal is used to communicate with the cloud and edge sides respectively, and receive linkage alarm signals sent from the cloud and edge sides and equipment status monitoring data from the edge side; in addition, the smart terminal is also used to monitor the location information and status information of on-site workers to identify safety risks of the workers.

[0124] Among them, smart terminals are devices with communication capabilities, vital signs detection capabilities or data processing capabilities, such as mobile phones, walkie-talkies, tablet computers, host computers, computers, smart bracelets and other devices.

[0125] Among them, vital signs include the pulse and breathing of the workers. The pulse can be monitored by a smart bracelet, and the breathing can be monitored by wearing a flexible respiratory sensor. When the vital sign information of a worker is abnormal, an alarm will be issued through the smart bracelet, and the alarm information will be sent to the edge side, which will be distributed to the on-site management personnel to initiate rescue measures.

[0126] The method for determining the above-mentioned operation safety risks of the operators of the present application is applied to the cloud device of the operation site safety risk linkage system. The operation site safety risk linkage system also includes edge side devices and terminal devices. The cloud device, edge side devices and terminal devices are communicatively connected. The method first receives the basic status information of the on-site operators and the equipment operation status of the on-site equipment sent by the edge side devices. The basic status information includes the identity information and behavior status information of the on-site operators; then, the operation operation information of the on-site operators is determined according to the basic status information of the on-site operators; then, the vital sign detection information of the on-site operators sent by the terminal device is received; finally, the operation safety risks of the on-site operators are determined according to at least one of the operation operation information of the on-site operators, the vital sign detection information of the on-site operators and the equipment operation status of the on-site equipment. This method performs face detection, safety facility target detection, and equipment status monitoring through collaborative processing of the cloud and edge servers, and transmits the detection results and status monitoring data to the terminal in real time for display. At the same time, it promptly informs the operators of abnormal status (such as failure of operator identity authentication, abnormal status of on-site equipment, etc.) through the terminal, thereby minimizing safety hazards. This method reduces the computing pressure of the cloud through collaborative computing between the cloud and the edge, improves the efficiency of data and status circulation, and effectively improves the safety and responsiveness of the work site. It solves the problem that the data transmission process of the traditional cloud computing model in the prior art requires a large bandwidth, and the data is large during data processing, resulting in data delays, and cannot meet the real-time requirements of construction site operation monitoring.

[0127] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for determining the operational safety risk of operators of the present application will be described in detail below in conjunction with specific embodiments.

[0128] This embodiment relates to a specific method for determining the operation safety risk of an operator, such as Figure 3 As shown, the following steps are included:

[0129] In this embodiment, human behavior recognition is achieved in the following ways:

[0130] S1. Record the monitoring video of the operator in real time through the video monitoring equipment, and transmit the video data to the edge server through data wiring or wireless network.

[0131] S2. Extract facial expression features of operators and perform emotion analysis on operators through emotion recognition model, specifically:

[0132] S21, using a face detection algorithm to detect faces in the video frame image, and cropping the detected faces to obtain a face image;

[0133] S22, inputting the face image in the first frame as a template input, and then inputting the face images in subsequent frames as inputs to be detected;

[0134] S23, the two inputs are respectively subjected to feature mapping through a weight-sharing full convolutional neural network operation; the template input is convolved using a convolution kernel to obtain a template feature, and the input to be detected is convolved using a convolution kernel to obtain a feature to be detected;

[0135] S24, dividing the template feature into k parts in order on the convolution channel, and treating it as k convolution kernels; performing convolution processing on the k convolution kernels and the feature to be detected to generate a feature map with k convolution channels, and using the generated feature map as the facial expression feature;

[0136] S25. Match the emotion feature with the pre-extracted facial expression feature, and select the emotion label corresponding to the facial expression feature with the highest matching degree as the recognized emotion, so as to realize the recognition of the emotion of the operator.

[0137] S3, collect images from the video data, and identify the behavior of the operator through the trained classifier model, specifically:

[0138] S31, pre-processing the image in the video frame, segmenting the area of ​​the operator, and capturing or tracking the target;

[0139] S32, extracting image feature parameters from the preprocessed image, and sending them into a recurrent neural network to obtain a mapping between the image feature parameters and the high-dimensional vector; specifically, using an openpose recognition algorithm and a target tracking algorithm to extract human image feature parameters from the image frame, so as to extract more accurate image features;

[0140] S33. Based on the high-dimensional vector of the video frame, a classifier model is established, a mapping from the high-dimensional vector to the final irregular behavior category is established, and the classifier model is trained.

[0141] S4. Conduct a comprehensive analysis based on the emotion recognition results and behavior recognition results to determine the operator's behavior and thus judge whether the operator has performed any dangerous operations.

[0142] In this embodiment, a work site safety risk linkage system is provided, such as Figure 3The system includes: a cloud device 01, which is used to execute any one of the above-mentioned methods for determining the work safety risks of the operators; an edge-side device 02, which is communicated with the above-mentioned cloud device 01, and the above-mentioned edge-side device 02 is used to collect basic status information of on-site operators and equipment operation status of on-site operation equipment; a terminal device 03, which is respectively communicated with the above-mentioned cloud device 01 and the above-mentioned edge-side device 02, and the above-mentioned terminal device 03 is used to collect the vital sign detection information of the above-mentioned on-site operators.

[0143] The system consists of three parts: cloud, edge and terminal. The cloud is used to store and analyze the data uploaded by the edge of the work site, and judge whether the on-site workers comply with safety regulations through face detection and target detection; the edge is used to collect images, identities and status information of the workers and equipment on the work site, and combine the identity detection results with the face detection results on the cloud to judge whether the workers violate the rules, and judge whether the on-site equipment has faults based on the status information of the equipment; the terminal is used to receive alarm information sent by the cloud or edge, and receive detection results from the cloud and the edge at the same time. This system can improve the safety and responsiveness of the work site.

[0144] The above system is divided into three parts: cloud, edge and terminal. It takes the edge intelligent terminal (edge) as the core, integrates platform (cloud) operation plans, work tickets, personnel files and other business information, and integrates intelligent terminal products (end) to realize multi-terminal decision-making and linkage alarm for safety risks at the work site.

[0145] Among them, the cloud side stores and analyzes the data of the work site through the cloud platform. The cloud platform can use data analysis and algorithms to analyze and model various data of the work site in real time and identify potential safety risks. Safety supervision equipment is deployed near the work site on the edge side to process and analyze the data generated on site in real time. The safety supervision equipment can respond to changes on site in a timely manner and perform some immediate control and intervention operations, such as shutting down certain equipment and activating alarms, through local calculations and decisions. The terminal includes various terminal devices at the work site, which can monitor the status of personnel and can be linked with the cloud platform and safety supervision equipment. When the cloud platform or safety supervision equipment detects a safety risk, it can promptly notify the on-site personnel through linkage alarms to trigger corresponding emergency response measures. For example, send an alarm notification to the supervisor's mobile phone or walkie-talkie, or start an on-site voice notification.

[0146] The above-mentioned edge side is deployed with video surveillance equipment, identity authentication equipment and equipment status monitoring system; the above-mentioned video surveillance equipment is used to record the video of on-site workers in real time, and the data is used to analyze the behavior of the workers; the above-mentioned identity authentication equipment is used to authenticate the workers, and is combined with the face detection results in the cloud to verify the uniqueness of the workers' identity; the above-mentioned equipment status monitoring system is used to monitor the operating status of the equipment at the work site, and promptly issue an alarm signal when the equipment is abnormal.

[0147] The identity authentication device includes a fingerprint recognition device and an identity card authentication device; the above-mentioned fingerprint recognition device is used to identify the fingerprint data of the on-site workers, and the above-mentioned identity card authentication device is used to identify the identity card carried by the workers, and the fingerprint recognition results, identity card recognition results and cloud face detection results are combined to verify the uniqueness of the worker's identity.

[0148] like Figure 4 As shown, the equipment status monitoring system includes: a data acquisition module, which is used to collect equipment operation status data; a data processing module, which is used to process the data collected by the above data acquisition module; a status evaluation module, which is used to evaluate the health status of the field equipment based on the data processed by the above data processing module, and adopts fuzzy neural network or support vector machine algorithm to perform status identification and evaluation; a fault prediction module, which is used to monitor various parameters that can cause faults and establish threshold values, and monitor parameter value changes at any time; an alarm protection module, which can respond quickly and take protective measures when the field equipment is abnormal;

[0149] The above-mentioned data acquisition module includes a data conversion submodule, a data transmission submodule and a data storage submodule; the above-mentioned data conversion submodule converts the original data in the data acquisition module into different data formats and exchanges them with subsequent data; the above-mentioned data transmission submodule is used to transmit the collected data to other modules or the cloud, terminal; the above-mentioned data storage submodule is used to store the collected data in a local or cloud storage device for subsequent processing and analysis.

[0150] The data acquisition module includes various types of sensors, such as temperature sensors, pressure sensors, current sensors, vibration sensors, and humidity sensors, to collect various parameters of the operating status of various equipment at the work site. These sensors are connected to the data acquisition module by wire or wireless means to achieve real-time transmission and storage of data. In addition, high-precision measurement and advanced signal processing technology can also be introduced to improve the accuracy and stability of data acquisition and ensure the integrity and security of data.

[0151] The above-mentioned data processing module is used to process the data collected by the data acquisition module; this module can perform more advanced data processing and analysis algorithms, such as wavelet transform, Fourier transform, principal component analysis, cluster analysis, etc., to extract more useful features and information from the original data. In addition, data mining techniques such as association rule mining and decision tree classification can also be introduced to discover hidden patterns and associations in the data. At the same time, data preprocessing steps such as data cleaning and outlier processing can be considered to improve the accuracy and reliability of data analysis.

[0152] The above-mentioned status assessment module is used to assess the health status of the equipment according to the data processed by the data processing module, and adopts fuzzy neural network or support vector machine algorithm for status recognition and assessment; this module can adopt a variety of machine learning algorithms to assess the health status of the equipment, such as support vector machine, random forest, deep learning, etc. At the same time, unsupervised learning algorithms, such as cluster analysis, can be introduced to group and classify the operating status of the equipment. In addition, a variety of evaluation indicators, such as statistical indicators, prediction error indicators, etc., can also be used to comprehensively evaluate the health status of the equipment.

[0153] The above-mentioned fault prediction module is used to monitor various parameters that can cause faults and establish threshold values, and monitor parameter value changes at any time; a variety of prediction methods are used for fault warning, such as time series prediction, regression analysis, anomaly detection, etc. It is possible to consider introducing deep learning models, such as recurrent neural networks (RNN) or convolutional neural networks (CNN), to efficiently predict time series data. In addition, reinforcement learning algorithms can also be introduced to achieve intelligent control and optimization of equipment operating status. It is possible to consider introducing artificial intelligence technologies, such as genetic algorithms, particle swarm optimization algorithms, etc., to achieve intelligent decision-making and optimization of fault warnings.

[0154] The above alarm protection module can quickly respond and take protective measures when abnormalities occur in the equipment at the operation site; it introduces intelligent control theory to achieve rapid fault location and automatic isolation. At the same time, the alarm protection module can also remotely cut off the power supply and start the backup equipment through the Internet or a private network to ensure the safe and stable operation of the equipment.

[0155] Among them, the data transmission module, status assessment module, fault prediction module and alarm protection module are all connected to the above-mentioned terminals through the network, and transmit data to the terminals in real time for display.

[0156] The embodiment of the present application also provides a cloud device. It should be noted that the cloud device of the operator in the embodiment of the present application can be used to execute the method for determining the operational safety risk of the operator provided in the embodiment of the present application. The cloud device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the cloud devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0157] The following is an introduction to the cloud device provided in the embodiments of the present application.

[0158] Figure 5 Schematic diagram of a cloud device according to an embodiment of the present application. The cloud device is a cloud device of a work site safety risk linkage system, the work site safety risk linkage system further includes an edge device and a terminal device, and the cloud device, the edge device and the terminal device are in communication connection. Figure 5 As shown, the cloud device includes a first receiving unit 10, a first determining unit 20, a second receiving unit 30 and a second determining unit 40. The first receiving unit 10 is used to receive the basic status information of the on-site workers and the equipment operation status of the on-site equipment sent by the edge side device. The basic status information includes the identity information and behavior status information of the on-site workers. The identity information is the identification information that uniquely identifies the on-site workers, and the behavior status information at least includes the action information of the on-site workers. The first determining unit 20 is used to determine the operation information of the on-site workers according to the basic status information of the on-site workers, and the operation information represents whether the operation of the on-site workers is in violation of regulations. The second receiving unit 30 is used to receive the vital sign detection information of the on-site workers sent by the terminal device, and the vital sign detection information at least includes the vital sign information of the on-site workers. The second determining unit 40 is used to determine the operation safety risk of the on-site workers according to at least one of the operation information of the on-site workers, the vital sign detection information of the on-site workers and the equipment operation status of the on-site equipment.

[0159] In some instances, the first determination unit includes a first acquisition module, a first processing module and a first determination module, the first acquisition module is used to obtain the facial data and identity data of the above-mentioned on-site workers in the above-mentioned basic status information, the above-mentioned identity data includes the fingerprint data of the above-mentioned on-site workers and / or the identity card data of the above-mentioned on-site workers; the first processing module is used to perform facial recognition on the above-mentioned on-site workers according to the above-mentioned facial data of the above-mentioned on-site workers, obtain facial recognition results, and match the above-mentioned facial recognition results with the above-mentioned identity data; the first determination module is used to determine the work operation information of the above-mentioned on-site workers according to the above-mentioned facial data of the above-mentioned on-site workers and the above-mentioned identity data when the above-mentioned facial recognition results of the above-mentioned on-site workers successfully match the above-mentioned identity data according to the above-mentioned on-site workers' facial data and / or the above-mentioned on-site workers' behavioral status information.

[0160] In some instances, the first processing module includes a second processing module, a first convolution module, a second convolution module splicing module and a classification module. The second processing module is used to convert the face image to be tested collected by the above-mentioned edge side device into a grayscale image, and adjust the size of the above-mentioned grayscale image to a preset size to obtain a first face image; the first convolution module is used to convolve the above-mentioned first face image with the first layer of convolution filters respectively, and obtain the output feature map of the first feature extraction layer as the first feature map; the second convolution module is used to convolve the above-mentioned first feature map with the second layer of convolution filters respectively, and obtain the output feature map of the second feature extraction layer as the second feature map; the splicing module is used to perform binary hash encoding on the above-mentioned second feature map, statistically calculate the local area histogram, and splice the statistical histograms of all local areas to obtain the final output features of the above-mentioned first face image; the classification module is used to input the above-mentioned final output features into the trained Linear SVM classifier for classification to obtain the above-mentioned face recognition result.

[0161] In some instances, the first determination module includes a second determination module, a third determination module and a fourth determination module, the second determination module is used to determine the emotional characteristics of the above-mentioned on-site workers based on the facial data of the above-mentioned on-site workers, and identify the emotions of the above-mentioned on-site workers based on the emotional characteristics of the above-mentioned on-site workers to obtain the emotion recognition results of the above-mentioned on-site workers; the third determination module is used to determine the wearing status of the safety facilities of the above-mentioned on-site workers and the working behavior of the above-mentioned on-site workers based on the behavioral status information of the above-mentioned on-site workers, the above-mentioned safety facilities are protective facilities worn by the above-mentioned on-site workers to reduce the operational safety risks of the above-mentioned on-site workers, and the above-mentioned working behavior is the behavior of the above-mentioned on-site workers performing working operations on the above-mentioned on-site working equipment; the fourth determination module is used to determine whether the working operation of the above-mentioned on-site workers is in violation of regulations based on at least one of the three: the emotional recognition results of the above-mentioned on-site workers, the wearing status of the safety facilities of the above-mentioned on-site workers and the working behavior of the above-mentioned on-site workers, so as to determine the above-mentioned working operation information of the above-mentioned on-site workers.

[0162] In some instances, the second determination module includes a first processing submodule, a second processing submodule and a third processing submodule. The first processing submodule is used to detect the video frame images of the facial images of the above-mentioned on-site workers in the above-mentioned facial data using a face detection algorithm, and to crop the above-mentioned video frame images to obtain the facial images of the above-mentioned on-site workers; the second processing submodule is used to input the above-mentioned facial images into a fully convolutional neural network, generate a feature map of the above-mentioned facial images, and determine the above-mentioned feature map as the facial expression features of the above-mentioned on-site workers; the third processing submodule is used to perform feature matching on the above-mentioned facial expression features with preset facial features corresponding to each emotion label to obtain multiple matching degrees, and use the above-mentioned emotion label corresponding to the above-mentioned preset facial feature with the highest matching degree as the emotion label of the above-mentioned facial expression features to obtain the emotion recognition results of the above-mentioned on-site workers.

[0163] In some instances, the third determination module includes an acquisition submodule and a fourth processing submodule, the acquisition submodule is used to acquire multiple full-body original images of the above-mentioned on-site workers, perform data preprocessing on the above-mentioned full-body original images, and obtain multiple preprocessed full-body images, and the above-mentioned data preprocessing includes at least one of the following: data enhancement processing, adaptive anchor frame processing, and adaptive image scaling processing; the fourth processing submodule is used to input all of the above-mentioned preprocessed full-body images into the target detection model to obtain the wearing status of the safety facilities of the above-mentioned on-site workers, the above-mentioned target detection model is obtained based on the training of the YOLOv5s network and the Res2Net network, and the wearing status of the safety facilities of the above-mentioned on-site workers is correct wearing or incorrect wearing.

[0164] In some instances, the third determination module includes a receiving module, a fifth processing sub-module and an identification module, the receiving module is used to receive the initial image data of the above-mentioned on-site workers collected by the above-mentioned edge-side device, the fifth processing sub-module is used to pre-process the above-mentioned initial image data to obtain processed image data, and extract image feature parameters of the above-mentioned processed image data; the identification module is used to input the above-mentioned image feature parameters into a value classification model for identification to obtain the working behavior of the above-mentioned on-site workers, and the above-mentioned working behavior includes the presence of dangerous operations by the above-mentioned on-site workers or the absence of the above-mentioned dangerous operations.

[0165] The cloud device includes a processor and a memory, and the first determination unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0166] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the data transmission process of the traditional cloud computing model in the prior art requires a large bandwidth, and the data processing is large, resulting in data delay, and cannot meet the real-time requirements of construction site operation monitoring.

[0167] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0168] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for determining the operation safety risk of the operator.

[0169] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the program executes the method for determining the operation safety risk of the operator when running.

[0170] An embodiment of the present invention provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the steps of a method for determining the operation safety risk of an operator are implemented. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0171] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initiating the steps of a method for determining a work safety risk of at least a worker.

[0172] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0173] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0174] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0177] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0178] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0179] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0180] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0181] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0182] 1) The method for determining the operational safety risks of the above-mentioned operators of the present application is applied to the cloud device of the operation site safety risk linkage system. The operation site safety risk linkage system also includes edge side devices and terminal devices. The cloud device, edge side devices and terminal devices are communicatively connected. The method first receives the basic status information of the on-site operators and the equipment operation status of the on-site equipment sent by the edge side devices. The basic status information includes the identity information and behavior status information of the on-site operators; then, the operation operation information of the on-site operators is determined based on the basic status information of the on-site operators; then, the vital sign detection information of the on-site operators sent by the terminal device is received; finally, the operational safety risks of the on-site operators are determined based on at least one of the operation operation information of the on-site operators, the vital sign detection information of the on-site operators and the equipment operation status of the on-site equipment. This method performs face detection, safety facility target detection, and equipment status monitoring through collaborative processing of the cloud and edge servers, and transmits the detection results and status monitoring data to the terminal in real time for display. At the same time, it promptly informs the operators of abnormal status (such as failure of operator identity authentication, abnormal status of on-site equipment, etc.) through the terminal, thereby minimizing safety hazards. This method reduces the computing pressure of the cloud through collaborative computing between the cloud and the edge, improves the efficiency of data and status circulation, and effectively improves the safety and responsiveness of the work site. It solves the problem that the data transmission process of the traditional cloud computing model in the prior art requires a large bandwidth, and the data is large during data processing, resulting in data delays, and cannot meet the real-time requirements of construction site operation monitoring.

[0183] 2) The system consists of three parts: cloud, edge and terminal. The cloud is used to store and analyze the data uploaded by the edge of the work site, and judge whether the on-site workers comply with safety regulations through face detection and target detection; the edge is used to collect images, identities and status information of the workers and equipment on the work site, and combine the identity detection results with the face detection results on the cloud to judge whether the workers violate the rules, and judge whether there is a fault in the on-site equipment based on the status information of the equipment; the terminal is used to receive alarm information sent by the cloud or edge, and receive detection results from the cloud and the edge at the same time. This system can improve the safety and responsiveness of the work site.

[0184] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining the operation safety risk of an operator, characterized in that: A cloud device applied to a work site safety risk linkage system, wherein the work site safety risk linkage system further comprises an edge device and a terminal device, wherein the cloud device, the edge device and the terminal device are communicatively connected, and the method comprises: Receive basic status information of the on-site workers and the equipment operation status of the on-site equipment sent by the edge side device, the basic status information including the identity information and behavior status information of the on-site workers, the identity information is identification information that uniquely identifies the on-site workers, and the behavior status information includes at least the action information of the on-site workers; Determining, according to the basic status information of the on-site operator, operation information of the on-site operator, wherein the operation information indicates whether the operation of the on-site operator is in violation of regulations; receiving the physical sign detection information of the on-site worker sent by the terminal device, wherein the physical sign detection information at least includes the vital sign information of the on-site worker; The operation safety risk of the on-site operator is determined based on at least one of the operation information of the on-site operator, the vital sign detection information of the on-site operator, and the equipment operation status of the on-site operation equipment.

2. The method according to claim 1, characterized in that Determining the operation information of the on-site operator according to the basic status information of the on-site operator includes: Acquire facial data and identity data of the on-site worker in the basic status information, wherein the identity data includes fingerprint data of the on-site worker and / or identity card data of the on-site worker; Performing facial recognition on the on-site worker according to the facial data of the on-site worker to obtain a facial recognition result, and matching the facial recognition result with the identity data; When the face recognition result of the on-site worker successfully matches the identity data, the work operation information of the on-site worker is determined according to the face data of the on-site worker and / or the behavior status information of the on-site worker.

3. The method according to claim 2, characterized in that Performing face recognition on the on-site worker according to the face data of the on-site worker to obtain a face recognition result includes: Converting the face image to be tested acquired by the edge device into a grayscale image, and adjusting the size of the grayscale image to a preset size to obtain a first face image; Convolving the first face image with the first layer of convolutional filters respectively, to obtain an output feature map of the first feature extraction layer as a first feature map; Convolving the first feature map with the second layer of convolutional filters respectively, to obtain an output feature map of the second feature extraction layer as a second feature map; Performing binary hash coding on the second feature map, calculating local area histograms, and concatenating the statistical histograms of all local areas to obtain a final output feature of the first face image; The final output features are input into the trained Linear SVM classifier for classification to obtain the face recognition result.

4. The method according to claim 2, characterized in that: Determining the operation information of the on-site operator according to the facial data of the on-site operator and / or the behavior status information of the on-site operator includes: Determining the emotional characteristics of the on-site workers according to the facial data of the on-site workers, and identifying the emotions of the on-site workers according to the emotional characteristics of the on-site workers to obtain an emotion recognition result of the on-site workers; Determine the wearing status of the safety equipment of the on-site workers and the working behavior of the on-site workers according to the behavior status information of the on-site workers, wherein the safety equipment refers to the protective equipment worn by the on-site workers to reduce the working safety risks of the on-site workers, and the working behavior refers to the behavior of the on-site workers in performing working operations on the on-site working equipment; Determine whether the on-site worker's operating operation is in violation of regulations based on at least one of the emotion recognition result of the on-site worker, the wearing status of the safety equipment of the on-site worker, and the operating behavior of the on-site worker, so as to determine the operating operation information of the on-site worker.

5. The method according to claim 4, characterized in that Determining the emotional characteristics of the on-site workers according to the facial data of the on-site workers, and recognizing the emotions of the on-site workers according to the emotional characteristics of the on-site workers to obtain the emotion recognition results of the on-site workers, including: Detecting the video frame images including the face images of the on-site workers in the face data by using a face detection algorithm, and cropping the video frame images to obtain the face images of the on-site workers; Inputting the facial image into a fully convolutional neural network to generate a feature map of the facial image, and determining the feature map as the facial expression features of the on-site operator; The facial expression feature is feature matched with the preset facial features corresponding to each emotion label to obtain multiple matching degrees, and the emotion label corresponding to the preset facial feature with the highest matching degree is used as the emotion label of the facial expression feature to obtain the emotion recognition result of the on-site worker.

6. The method according to claim 4, characterized in that Determining the wearing status of the safety equipment of the on-site workers according to the behavior status information of the on-site workers includes: Acquire a plurality of full-body original images of the on-site workers, perform data preprocessing on the full-body original images to obtain a plurality of preprocessed full-body images, wherein the data preprocessing includes at least one of the following: data enhancement processing, adaptive anchor frame processing, and adaptive image scaling processing; All the pre-processed full-body images are input into the target detection model to obtain the wearing status of the safety facilities of the on-site workers. The target detection model is trained based on the YOLOv5s network and the Res2Net network. The wearing status of the safety facilities of the on-site workers is correct or incorrect.

7. The method according to claim 4, characterized in that Determining the operation behavior of the on-site operator according to the behavior status information of the on-site operator includes: Receiving initial image data of the on-site worker collected by the edge-side device; Preprocessing the initial image data to obtain processed image data, and extracting image feature parameters of the processed image data; The image feature parameters are input into a value classification model for identification to obtain the working behavior of the on-site operator, where the working behavior includes whether the on-site operator performs dangerous operations or does not perform the dangerous operations.

8. A work site safety risk linkage system, characterized in that: include: A cloud device, used to execute the method for determining the operation safety risk of an operator as described in any one of claims 1 to 7; An edge-side device is connected to the cloud-side device for communication, and is used to collect basic status information of on-site workers and equipment operation status of on-site equipment; The terminal device is respectively connected to the cloud device and the edge device for communication, and the terminal device is used to collect the vital sign detection information of the on-site workers.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for determining the operation safety risk of the operator as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for determining the work safety risk of a worker as described in any one of claims 1 to 7.

Images