Artificial intelligence-based violation behavior alerting method, system, medium, and device

Abstract

The application provides a kind of based on artificial intelligence's illegal behavior warning method, system, medium and equipment, method includes: obtaining image to be identified, including construction site and construction personnel;The background environment of image to be identified is identified, the dangerous area of construction site is determined;Image to be identified is handled, and the target image containing target construction personnel is obtained, and the target image is input into the preset behavior recognition model, and the behavior category of target construction personnel output after the behavior recognition model is identified is acquired;Based on dangerous area and the behavior category of target construction personnel, the illegal personnel of implementing illegal behavior is determined, and illegal behavior prompt information is generated to carry out illegal prompt.Such a way can consider the complex environment of construction site, determine the dangerous area from it, monitor the behavior category of construction personnel in dangerous area, so as to be applicable to the complex environment of construction site, realize the effective supervision of construction personnel.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence recognition, in particular to a method and system for warning illegal behavior based on artificial intelligence, a medium and equipment. BACKGROUND

[0002] Safety problems of construction sites have always been concerned about, and safe operation is the premise for the smooth progress of site construction, and safety supervision is the guarantee of safe operation. In order to realize effective supervision of illegal behavior of site operators, the existing supervision methods generally adopt manual supervision, man-machine combined supervision (manual + intelligent supervision) or intelligent supervision.

[0003] However, in the existing supervision methods, the behavior monitoring of site construction personnel is not accurate enough, resulting in low monitoring accuracy, and the behavior monitoring of construction personnel lacks effective definition, which is difficult to apply to the complex environment of construction sites, resulting in that the pure intelligent supervision is not effective enough and is difficult to apply to complex construction site supervision scenarios. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a method and system for warning illegal behavior based on artificial intelligence, a medium and equipment, so as to be applicable to the complex environment of construction sites and realize effective supervision of construction personnel.

[0005] In order to achieve the above purpose, the embodiments of the present application are realized by the following ways:

[0006] In a first aspect, the embodiments of the present application provide a method for warning illegal behavior based on artificial intelligence, comprising: acquiring a to-be-recognized image, wherein the to-be-recognized image contains a construction site and a construction personnel; performing background environment recognition on the to-be-recognized image to determine a dangerous area of the construction site; judging whether there is a target construction personnel located in the dangerous area in the to-be-recognized image; if there is, processing the to-be-recognized image to obtain a target image containing the target construction personnel, inputting the target image into a preset behavior recognition model, acquiring a behavior category of the target construction personnel output by the behavior recognition model after recognition, and determining whether there is a person who implements illegal behavior based on the dangerous area and the behavior category of the target construction personnel, and generating illegal behavior prompt information to prompt illegal behavior when there is a person who implements illegal behavior.

[0007] In the embodiments of the present application, the background environment recognition is performed on the to-be-recognized image to determine the dangerous area of the construction site, it is judged whether the target construction personnel located in the dangerous area exists in the to-be-recognized image, the target image containing the target construction personnel is obtained by processing the to-be-recognized image, the behavior category of the target construction personnel is recognized by inputting the target image into the preset behavior recognition model, and it is judged whether the target construction personnel violates the rules by combining the dangerous area and the behavior category of the target construction personnel. In this way, the complex environment of the construction site can be considered to determine the dangerous area, and the behavior category of the construction personnel in the dangerous area is monitored, so that the complex environment of the construction site can be applied, and effective supervision of the construction personnel can be realized.

[0008] In combination with the first aspect, in a first possible implementation manner of the first aspect, the background environment recognition is performed on the to-be-recognized image to determine the dangerous area of the construction site, including: image segmentation is performed on the to-be-recognized image to recognize the edge environment in the to-be-recognized image, wherein the edge environment includes a hole edge, a roof periphery, a well edge, a foundation pit edge, a unloading platform edge, and a high-rise window edge; and a corresponding dangerous area is determined based on each edge environment.

[0009] In this implementation manner, the image segmentation is performed on the to-be-recognized image to recognize the edge environment (for example, the hole edge, the roof periphery, the well edge, the foundation pit edge, the unloading platform edge, the high-rise window edge, etc.) in the to-be-recognized image; and then the corresponding dangerous area is determined based on each edge environment (to demarcate the range of a dangerous area by the edge environment). In this way, the complex construction environment of the construction site can be considered to recognize different types of edge environment and further demarcate the dangerous area. Since the types of dangerous behaviors are different due to different construction environments, different behaviors in different construction environments can be considered to facilitate the effectiveness of the rule violation behavior recognition.

[0010] In combination with the first aspect, in a second possible implementation manner of the first aspect, the behavior recognition model includes a joint sequence capture module and a behavior category classification module, the target image is input into the preset behavior recognition model, the behavior category of the target construction personnel output by the behavior recognition model after recognition is obtained, including: the target image is input into the joint sequence capture module, the target joint sequence corresponding to the target image is calculated through the joint sequence capture module; the target joint sequence is input into the behavior category classification module, the target joint sequence is classified and recognized through the behavior category classification module, and the behavior category of the target construction personnel is output after being determined.

[0011] In a third possible implementation form of the first aspect, in conjunction with the second possible implementation form of the first aspect, the joint sequence capturing module calculates the target joint sequence corresponding to the target image in the following manner: the target joint sequence corresponding to the target image is calculated by using the following formula:

[0012]

[0013] wherein f out is an output of the joint sequence capturing module, k v is a kernel size of a spatial dimension, here k v is 3, W k is a weight tensor, f in is an input parameter corresponding to the target image, and is a CxTxN-dimensional tensor, wherein N represents a number of nodes, C represents a number of channels, and T represents a time length, X k is an N*N-order adjacency matrix, representing a physical structure feature of human joints, Y k is also an N*N-order adjacency matrix, used for autonomously learning features of different layers, Z k is a data dependency graph, used for determining a closeness degree between different joint nodes, and a∈(0, 1) is a weight of Z k , represents an element product between two matrices, M k is an N*N-order attention mask graph, initialized as a matrix full of 1s.

[0014] In a fourth possible implementation form of the first aspect, in conjunction with the third possible implementation form of the first aspect, before calculating the target joint sequence corresponding to the target image, the method further comprises:

[0015] The similarity between two nodes is calculated by using the following formula:

[0016]

[0017] wherein v i and v j are joint nodes corresponding to the target image, N is a total number of joint nodes corresponding to the target image, f(v i ,v j ) is a similarity between node v i and node v j , θ(v i ) is an embedding function, corresponding to a feature map of node v i , θ(v i ) T is a transposed form of θ(v i ), and φ(v j ) is an embedding function, corresponding to a feature map of node vj characteristic map, δ is a constant, and is used to stabilize the function θ(v i ) T φ(v j ) and τ ∈ 0, f in ] is a hyperparameter, which is used to increase the sensitivity of the function θ(v i ) T φ(v j ) to low-probability candidates;

[0018] Based on the embedding functions θ(v i ) and φ(v j ), the data dependency graph Z k is calculated using the Log-Softmax Temperatue classification function:

[0019]

[0020] where f in is an input parameter, which is a CxTxN-dimensional tensor, f in T is the transpose of f in , and W φk are the parameter matrices corresponding to the embedding functions θ(v i ) T and φ(v j ), respectively.

[0021] In a fifth possible implementation manner of the first aspect, in combination with the second possible implementation manner of the first aspect, the behavior category classification module is provided with a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, and an output layer connected in sequence, and the behavior category classification module classifies and identifies the target joint sequence in the following manner: the first convolutional layer is used to convolve the target joint sequence, and the first pooling layer is used to pool; then the second convolutional layer is used to convolve, and the second pooling layer is used to pool; then the third convolutional layer is used to convolve, and the third pooling layer is used to pool, and a fully connected layer is used to fully connect to obtain a processed vector sequence; the vector sequence is input to the output layer, and a Softmax classifier in the output layer is used to classify and identify the vector sequence.

[0022] In a sixth possible implementation manner of the first aspect, in the first possible implementation manner of the first aspect, the behavior category includes running, climbing, crossing, leaning, throwing, and normal, and the determining whether there is a rule-breaking personnel implementing a rule-breaking behavior based on the dangerous area and the behavior category of the target construction personnel includes: if the edge environment of the dangerous area is a hole edge or a well edge, and the behavior category of the target construction personnel is running, crossing, or throwing, it is determined that the target construction personnel has a rule-breaking behavior; if the edge environment of the dangerous area is a foundation pit edge, and the behavior category of the target construction personnel is running or leaning, it is determined that the target construction personnel has a rule-breaking behavior; if the edge environment of the dangerous area is an unloading platform edge or a roof periphery, and the behavior category of the target construction personnel is climbing or leaning, it is determined that the target construction personnel has a rule-breaking behavior; if the edge environment of the dangerous area is a high-rise window edge, and the behavior category of the target construction personnel is climbing, leaning, or throwing, it is determined that the target construction personnel has a rule-breaking behavior; and the target construction personnel implementing the rule-breaking behavior is determined as the rule-breaking personnel.

[0023] In this implementation manner, the corresponding relationship between the dangerous area of different edge environments and different construction behavior categories can be considered, so that the monitoring of the rule-breaking behavior of the construction personnel in the construction site can be more accurately and effectively implemented.

[0024] In a second aspect, an embodiment of the present application provides an alarm system for rule-breaking behavior based on artificial intelligence, including: an image acquisition unit configured to acquire a to-be-identified image, wherein the to-be-identified image contains a construction site and a construction personnel; an environment identification unit configured to identify a background environment of the to-be-identified image, and determine a dangerous area of the construction site; a judgment unit configured to judge whether there is a target construction personnel located in the dangerous area in the to-be-identified image; a behavior identification unit configured to, when there is a target construction personnel located in the dangerous area in the to-be-identified image, process the to-be-identified image to obtain a target image containing the target construction personnel, input the target image into a preset behavior identification model, and acquire a behavior category of the target construction personnel output by the behavior identification model after identification; and a rule-breaking identification unit configured to determine whether there is a rule-breaking personnel implementing a rule-breaking behavior based on the dangerous area and the behavior category of the target construction personnel, and generate a rule-breaking behavior prompt information to perform a rule-breaking prompt when there is a rule-breaking personnel.

[0025] In a third aspect, an embodiment of the present application provides a storage medium including a stored program, wherein the program controls a device where the storage medium is located to execute the alarm method for rule-breaking behavior based on artificial intelligence of the first aspect or any one of the possible implementation manners of the first aspect when the program is running.

[0026] In a fourth aspect, an electronic device is provided, which comprises a memory and a processor, the memory is configured to store information including program instructions, and the processor is configured to control execution of the program instructions, and the program instructions are loaded and executed by the processor to implement the method for warning illegal behavior based on artificial intelligence according to the first aspect or any possible implementation manner of the first aspect.

[0027] In order to make the above objectives, characteristics and advantages of the present application more apparent, more comprehensible, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to, and the detailed description is as follows. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0029] Figure 1 The flow chart of the method for warning illegal behavior based on artificial intelligence provided by the embodiments of the present application.

[0030] Figure 2 The schematic diagram of the behavior recognition model provided by the embodiments of the present application.

[0031] Figure 3 The schematic diagram of the running principle of the joint sequence capture module.

[0032] Figure 4 The schematic diagram of the behavior category classification module.

[0033] Figure 5 The structural block diagram of the system for warning illegal behavior based on artificial intelligence provided by the embodiments of the present application.

[0034] Figure 6 The structural block diagram of the electronic device provided by the embodiments of the present application.

[0035] Icon: 10- system for warning illegal behavior based on artificial intelligence; 11- image acquisition unit; 12- environment recognition unit; 13- judgment unit; 14- behavior recognition unit; 15- illegal behavior recognition unit; 20- electronic device; 21- memory; 22- communication module; 23- bus; 24- processor. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described below in combination with the drawings in the embodiments of the present application.

[0037] In order to realize the identification of the illegal behavior of the construction personnel in the construction site, the electronic device can run the illegal behavior alarm method based on artificial intelligence.

[0038] Please refer to Figure 1 , Figure 1 A flowchart of an illegal behavior alarm method based on artificial intelligence provided by the embodiment of the application. The illegal behavior alarm method based on artificial intelligence can include steps S10, S20, S30, S40 and S50.

[0039] In this embodiment, the electronic device can first perform step S10.

[0040] Step S10: Obtain an image to be identified, wherein the image to be identified contains a construction site and a construction personnel.

[0041] In this embodiment, the construction site is provided with a plurality of monitoring devices for monitoring the situation of the construction site in real time. The construction site contains various construction site environments, such as construction sites with openings, buildings, wellheads, foundation pits, unloading platforms and other various construction environments. Different construction environments usually have different safety hazards, such as environments involving openings, wellheads, foundation pits, etc., which usually have serious hazards such as falling and throwing, while environments such as buildings have serious hazards such as high falling and throwing.

[0042] Each monitoring device of the construction site can monitor the construction site (usually a part of the construction site) in real time, and the monitoring video can be used as the original data of the monitoring. In order to realize the supervision of the construction site and detect the illegal behavior of the construction personnel, the electronic device can obtain an image to be identified, which can be an image in a monitoring video (for example, a plurality of images to be identified are obtained by frame extraction processing of the monitoring video), and the image to be identified contains a construction site and a construction personnel. Of course, there are images without construction personnel in the monitoring video, but the images without construction personnel are not within the scope of discussion of the present scheme, so they are not described here. For the images without construction personnel, they can be not identified as images to be identified.

[0043] After obtaining the image to be identified, the electronic device can perform step S20.

[0044] Step S20: Perform background environment identification on the image to be identified to determine the dangerous area of the construction site.

[0045] In this embodiment, the electronic device can perform background environment identification on the image to be identified to determine the dangerous area of the construction site.

[0046] Exemplarily, the electronic device can perform image segmentation on the to-be-identified image to identify the edge environment in the to-be-identified image, wherein the edge environment includes a hole edge, a roof periphery, a well edge, a foundation pit edge, a discharge platform edge, and a high-rise window edge.

[0047] For example, the edge environment can be identified by using an image segmentation algorithm, and whether there is a field worker in the environment can also be detected. The image segmentation algorithm can specifically be a YOLOv4-based image segmentation algorithm. First, the performance indicators of the model are defined as follows:

[0048] TP: correctly classified positive samples, that is, the intersection between the predicted value and the true value.

[0049] TN: correctly classified negative samples, that is, the intersection between the complement of the predicted value and the true value and the image space.

[0050] FP: incorrectly classified positive samples, that is, the difference between the predicted value and TP.

[0051] FN: incorrectly classified negative samples, that is, the difference between the true value and TN.

[0052] Then, the intersection over union (IoU) is redefined for the calculation of the loss function:

[0053]

[0054] Therefore, the model loss function can be designed as follows:

[0055]

[0056]

[0057] wherein Distance_C is the diagonal distance between the predicted frame and the true frame, Distance_2 is the Euclidean distance between the predicted frame and the true frame, is the aspect ratio of the true frame, is the aspect ratio of the predicted frame.

[0058] The model is trained by using a training set (containing multiple training images of a set size), and after the model is trained, verification and testing are performed. When the indicators are met, the image segmentation model is obtained, which is used to detect the edge environment in the to-be-identified image.

[0059] After the edge environment in the to-be-identified image is identified, the electronic device can determine a corresponding dangerous area based on each edge environment, for example, to demarcate the range of a dangerous area with the edge environment. The way of demarcating the dangerous area can be different for different edge environments, for example, for the edge of the hole, the edge of the well, and the edge of the foundation pit, a dangerous area can be demarcated at a distance of one meter (of course, one meter here is only an exemplary introduction, and for the range of the image, pixels are usually counted, because the shooting position of the monitoring device is fixed, and there is no change, the actual distance represented by a pixel point is relatively fixed). For the edge of the unloading platform, the edge of the high-rise window, and the periphery of the roof, a dangerous area can be demarcated at a distance of 2 meters, which is not limited here, and can be adjusted based on actual needs.

[0060] The to-be-identified image is segmented, the edge environment in the to-be-identified image is identified, and then the corresponding dangerous area is determined, so that the complex construction environment of the construction site can be considered, different types of edge environments are identified, and the dangerous area is further demarcated. Because the types of dangerous behaviors are different in different construction environments, different behaviors in different construction environments can be considered, so as to facilitate improving the effectiveness of the illegal behavior identification.

[0061] After the dangerous area of the construction site is determined, the electronic device can perform step S30.

[0062] Step S30: determining whether a target construction worker located in the dangerous area exists in the to-be-identified image.

[0063] In this embodiment, the electronic device can perform construction worker detection on the to-be-identified image to detect the position of the construction worker. In order to facilitate accurate judgment of whether the construction worker is located in the dangerous area, in this embodiment, the center coordinates of the nearest foot (palm) of the construction worker to the center point of the dangerous area are used to represent the position of the construction worker relative to the dangerous area, so as to determine whether the construction worker is located in the dangerous area. For the relationship between the same construction worker and multiple dangerous areas, this method can be used for judgment one by one.

[0064] If the target construction worker located in the dangerous area exists in the to-be-identified image, the electronic device can perform step S40.

[0065] Step S40: processing the to-be-identified image to obtain a target image containing the target construction worker, inputting the target image into a preset behavior recognition model, and obtaining a behavior category of the target construction worker output by the behavior recognition model after recognition.

[0066] First, the electronic device can process the image to be recognized to obtain a target image containing the target construction worker. Since this solution utilizes a behavior recognition model to identify behavior categories, the image to be recognized needs to be processed into a target image of a set size (containing the target construction worker) to facilitate processing by the behavior recognition model. Here, the target image can be cropped to a set size using the centroid coordinates of the target construction worker (e.g., the center of the target construction worker's torso) as the image center, thus obtaining the target image.

[0067] To facilitate understanding of this solution, the behavior recognition model will be introduced first. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of the behavior recognition model provided in the embodiments of this application.

[0068] In this embodiment, the behavior recognition model mainly includes two parts: a joint sequence capture module and a behavior category classification module.

[0069] The joint sequence capture module is primarily used to calculate the target joint sequence corresponding to the target image. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram illustrating the operating principle of the joint sequence capture module.

[0070] For the target image input to the joint sequence capture module, the joint sequence capture module can perform sampling (e.g., sampling human joints using a pose capture algorithm). Each target image samples N joints, and a set of T consecutive target images is considered as a target image set (here, the T consecutive target images are not limited to the T consecutive images within the surveillance video, but rather the T target images corresponding to the T images to be identified, arranged chronologically after frame extraction). Each image has C channels, thus obtaining the input parameter f corresponding to a target image set. in Input parameter f in The input parameters for the target image (the target image set it belongs to) are a C×T×N dimensional tensor, where N represents the number of nodes, C represents the number of channels, and T represents the time length (i.e. the number of frames).

[0071] It should be noted that for a single target construction worker, there are T target images corresponding to one input parameter f. in For cases where there are multiple target construction workers in the target image, an input parameter f can be obtained for each target construction worker using these T target images. in Therefore, the number of target construction workers determines the number of input parameters f. in For each input parameter f in It can be run independently using a behavior recognition model, so the behavior recognition of different target construction workers does not affect each other.

[0072] For input parameter f in , the output at position x can be computed by a convolutional neural network based on the joint graph:

[0073] Define a sampling function p:

[0074] B(v ti )→V, (4)

[0075] where B(v ti ) = {v tj | d(v ti , v tj )≤ D} is the neighborhood set of node v ti , v ti represents the i-th joint node at the t-th frame, v tj represents the j-th joint node at the t-th frame, V is the set of joint sequence feature matrixes, V = {v ti | t = 1,..., T; i = 1,..., N}, N is the number of joint nodes (i.e. corresponding to the selected N human body joints, for example, 14 joint nodes are selected, or more or fewer joint nodes are selected), (v ti , v tj ) is the shortest length from v ti to v tj , and the sampling function is:

[0076] d(v ti , v tj ) = v tj , (5)

[0077] Divide the neighborhood set B(v ti ) of joint node v ti into K labeled subsets, and define a mapping l ti : B(v ti )→ {0,..., K}, which maps the nodes in the neighborhood to the subset labels. Define a weight function w: B(v ti )→ R c , R c is the node space, and the weight function w(v ti , v tj ) is:

[0078] w(v ti , v tj ) = w'(l ti (v tj )), (6)

[0079] where w'(l ti (v tj )) represents w(lti (v tj ) is the derivative of f

[0080] The neighborhood is defined as three subsets, the first subset is the root node itself; the second subset is the centripetal node group (closer to the joint center of gravity relative to the root node itself); the third subset is the centrifugal node group (further away from the joint center of gravity relative to the root node itself). The average coordinates of all joint nodes in a single frame skeleton are regarded as the joint center of gravity thereof.

[0081] Then r i can be defined as the average distance from the center of gravity to v ti in the training set, and r j is the distance from v tj to the center of gravity. Then l ti (v tj ) is defined as:

[0082]

[0083] Then the convolution feature map output can be defined as:

[0084]

[0085] Where f out (v ti ) is the output, Z ti (v tj ) = | {v tk | l ti (v tk ) = l ti (v tj )} |, and is a normalization term, taking its reciprocal, used to average the contribution of different subsets.

[0086] The scheme is improved on this basis. According to the definition of the graph convolution formula in ST-GCN, formula (8) can be rewritten as:

[0087]

[0088] Where f out is the output of the joint sequence capture module, k v is the kernel size of the spatial dimension, here k v takes the value of 3, W k is the weight tensor, f in is the input parameter corresponding to the target image, which is a CxTxN tensor, N represents the number of nodes, C represents the number of channels, T represents the length of time, X k is an N*N order adjacency matrix, representing the physical structure characteristics of human joints, and Y kis also an N×N adjacency matrix, used for learning features of different layers autonomously, Z k is a data dependency graph, used for determining the tightness of the connection between different joints, and a∈(0, 1) is the weight of Z k , used for adjusting the weight of the feature dependency graph to adapt to the tightness of the connection between the joints of the target construction personnel in different environments, represents the element product between two matrices, M k is an N×N attention mask graph, initialized as a matrix full of 1s.

[0089] Here, the weight a of Z k , since in different construction environments, different dangerous areas, and illegal behavior is different, and the construction personnel's action in the construction environment may be different, for example, for the construction personnel working at the edge of the high building, wearing anti-falling devices, such as anti-falling lock protection devices based on speed difference automatic controllers, the construction personnel's action has obvious characteristics, which can be reflected in the tightness of the connection between the joints. By using the weight a, adaptive adjustment can be made. Exemplarily, the present scheme can set corresponding weights a for different categories of dangerous areas to adapt to the action characteristics of the target construction personnel in different categories.

[0090] For the N×N adjacency matrix X k representing the physical structure characteristics of the human joints, there is:

[0091]

[0092] wherein, Λ k is a diagonal matrix, is an N×N matrix with non-zero diagonal.

[0093] In order to calculate the data dependency graph Z k , a normalized embedded Gaussian formula can be defined to calculate the similarity between two nodes:

[0094]

[0095] wherein, v i , v j are the target image corresponding joints, N is the total number of target image corresponding joints, f(v i , v j ) is the similarity between node v i and node v j , θ(v i ) is an embedded function corresponding to the feature map of node v i , θ(v i ) T i ​The transpose of φ(v) j ) is an embedded function, corresponding to node v j The feature map, where δ is a constant, is used to stabilize the function θ(v i ) T φ(v j The value of ), τ∈(0,f in ] is a hyperparameter used to increase the function θ(v) i ) T φ(v j Sensitivity to low-probability candidates.

[0096] Based on the embedded function θ(v) i ) and φ(v j The data dependency graph Z can be calculated using the Log-SoftmaxTemperature classification function. k :

[0097]

[0098] Among them, f in The input parameter is a C×T×N dimensional tensor, f in T f in transpose, and W φk These are the embedded functions θ(v) i ) T and φ(v) j The parameter matrix corresponding to ).

[0099] The above is an introduction to the joint sequence capture module, which can be combined with... Figure 3 Refer to both for easier understanding of the output f of the convolutional feature map. out The calculation process (which can be understood as a target joint sequence) is as follows: Based on this, the target image (essentially a set of T target images) is input into the joint sequence capture module, and the target joint sequence corresponding to the target image can be calculated by the joint sequence capture module.

[0100] After obtaining the target joint sequence corresponding to the target image, the target joint sequence can be input into the behavior category classification module. The behavior category classification module will classify and identify the target joint sequence, determine the behavior category of the target construction worker, and then output the result.

[0101] Please see Figure 4 , Figure 4 This is a schematic diagram of the behavior category classification module. In this embodiment, the behavior category classification module may include multiple levels L1 to L12:

[0102] L1 is the input layer, used to receive the input target joint sequence.

[0103] L2-L3 is a first convolutional layer, configured to perform convolution on the target joint sequence: the target joint sequence can be three-dimensionally convolved by using a convolution kernel (the number of channels of the convolution kernel is consistent with the number of channels of the target joint sequence). In order to increase the feature map obtained by the convolution operation, 2-3 convolution kernels can be further configured to perform three-dimensional convolution.

[0104] L4 is a first pooling layer, configured to perform pooling on the first feature map output by the first convolutional layer.

[0105] L5-L6 is a second convolutional layer, configured to perform three-dimensional convolution on the pooled first feature map.

[0106] L7 is a second pooling layer, configured to perform pooling on the second feature map output by the second convolutional layer.

[0107] L8-L9 is a third convolutional layer, configured to perform three-dimensional convolution on the pooled second feature map.

[0108] L10 is a third pooling layer, configured to perform pooling on the third feature map output by the third convolutional layer.

[0109] L11 is a fully connected layer, and the one-dimensional vector obtained by flattening the pooled third feature map can be input to the fully connected layer to perform full connection, so as to obtain a vector sequence (for example, a vector sequence with 128 dimensions or 256 dimensions).

[0110] L12 is an output layer, and a Softmax classifier is built in the output layer, which can classify the vector sequence, so as to realize behavior category classification and identification of the target construction personnel based on the target joint sequence.

[0111] It should be noted that if there is no target construction personnel in the dangerous area, the illegal behavior monitoring can not be performed.

[0112] After the behavior category of the target construction personnel is determined, the electronic device can perform step S50.

[0113] Step S50: based on the dangerous area and the behavior category of the target construction personnel, it is determined whether there is a person who implements illegal behavior, and illegal behavior prompt information is generated to prompt the illegal behavior when there is a person who implements illegal behavior.

[0114] In this embodiment, the behavior category can include sprinting, climbing, crossing, leaning, throwing, and normal (i.e., behaviors other than sprinting, climbing, crossing, leaning, and throwing). The edge environment includes a hole edge, a roof periphery, a well edge, a foundation pit edge, a material unloading platform edge, and a high-rise window edge.

[0115] If the edge environment of the dangerous area is a hole edge or a well edge, and the behavior category of the target construction worker is sprinting, crossing or throwing, it can be determined that the target construction worker has a violation behavior.

[0116] If the edge environment of the dangerous area is a foundation pit edge, and the behavior category of the target construction worker is sprinting or leaning, it can be determined that the target construction worker has a violation behavior.

[0117] If the edge environment of the dangerous area is a discharge platform edge or a roof periphery, and the behavior category of the target construction worker is climbing or leaning, it can be determined that the target construction worker has a violation behavior.

[0118] If the edge environment of the dangerous area is a high-rise window edge, and the behavior category of the target construction worker is climbing, leaning or throwing, it can be determined that the target construction worker has a violation behavior.

[0119] After that, the electronic device can determine the target construction worker who has a violation behavior as a violation worker.

[0120] In this way, the corresponding relationship between the dangerous area of different edge environments and different construction behavior categories can be considered, so that the monitoring of the violation behavior of the construction workers on the construction site can be more accurate and more effective.

[0121] After the violation worker is determined, the electronic device can generate a violation behavior prompt information for violation prompting. For example, a sound and light alarm can be issued, or a promotion information can be issued, or even an identity recognition can be performed on the construction workers by using a smart device (such as a bracelet, a wearable smart device, a terminal, etc.) carried by the construction workers in combination with the construction workers in the image, for example, by using the positioning information of the smart device in combination with the positioning in the image to be recognized, the identity matching between the construction workers in the image and the construction workers on the construction site is realized, so that the violation prompt information is further sent to the violation worker, and accurate violation prompting is performed.

[0122] Based on the same inventive concept, the application also provides a violation behavior alarm system based on artificial intelligence, please refer to Figure 5 The violation behavior alarm system 10 based on artificial intelligence can include:

[0123] An image acquisition unit 11 is configured to acquire an image to be recognized, wherein the image to be recognized includes a construction site and construction workers.

[0124] An environment identification unit 12 is configured to identify the background environment of the image to be recognized, and determine a dangerous area of the construction site.

[0125] A judgment unit 13 is configured to judge whether a target construction worker located in the dangerous area exists in the image to be recognized.

[0126] The behavior recognition unit 14 is configured to, when the target construction personnel located in the dangerous area exists in the image to be recognized, process the image to be recognized to obtain a target image containing the target construction personnel, input the target image into a preset behavior recognition model, and obtain a behavior category of the target construction personnel output by the behavior recognition model.

[0127] The violation identification unit 15 is configured to determine whether a violation personnel exists based on the dangerous area and the behavior category of the target construction personnel, and generate a violation prompt information to prompt the violation when the violation personnel exists.

[0128] In this embodiment, the environment recognition unit 12 is specifically configured to perform image segmentation on the image to be recognized to identify a marginal environment in the image to be recognized, wherein the marginal environment includes a hole edge, a roof periphery, a well edge, a foundation pit edge, a unloading platform edge, and a high-rise window edge; and determine a corresponding dangerous area based on each of the marginal environments.

[0129] In this embodiment, the behavior recognition model includes a joint sequence capturing module and a behavior category classification module, and the behavior recognition unit 14 is specifically configured to input the target image into the joint sequence capturing module, calculate a target joint sequence corresponding to the target image through the joint sequence capturing module, input the target joint sequence into the behavior category classification module, and output the behavior category of the target construction personnel through the behavior category classification module after classifying and identifying the target joint sequence.

[0130] In this embodiment, the behavior recognition unit 14 is specifically configured to calculate the target joint sequence corresponding to the target image by using the following formula:

[0131]

[0132] wherein f out is an output of the joint sequence capturing module, k v is a kernel size of a spatial dimension, and k v is 3, W k is a weight tensor, f in is an input parameter corresponding to the target image, and is a C×T×N dimensional tensor, N represents a number of nodes, C represents a number of channels, T represents a time length, X k is an N×N order adjacency matrix, representing a physical structure feature of a human joint, Y k is also an N×N order adjacency matrix, used for autonomously learning features of different layers, Z k is a data dependency graph, used for determining a closeness between different joint nodes, and a∈(0, 1) is Zk The weight, M represents the element-wise product of two matrices. k The attention mask is of order N×N and is initialized as a matrix of all 1s.

[0133] In this embodiment, the behavior recognition unit 14 is specifically used to: calculate the similarity between two nodes using the following formula:

[0134]

[0135] Among them, v i v j All are joints corresponding to the target image, N is the total number of joints corresponding to the target image, f(v i ,v j ) is node v i With node v j The similarity between them, θ(v) i ) is an embedded function, corresponding to node v i Feature map, θ(v i ) T For θ(v) i The transpose of φ(v) j ) is an embedded function, corresponding to node v j The feature map, where δ is a constant, is used to stabilize the function θ(v i ) T φ(v j The value of ), τ∈(0,f in ] is a hyperparameter used to increase the function θ(v) i ) T φ(v j Sensitivity to low-probability candidates;

[0136] Based on the embedded function θ(v) i ) and φ(v j The data dependency graph Z is calculated using the Log-SoftmaxTemperature classification function. k :

[0137]

[0138] Among them, f in The input parameter is a C×T×N dimensional tensor, f in T f in transpose, and W φk These are the embedded functions θ(v) i ) T and φ(v) jcorresponding parameter matrix.

[0139] In the embodiment, the behavior category classification module is provided with a first convolution layer, a first pooling layer, a second convolution layer, a second pooling layer, a third convolution layer, a third pooling layer, a full connection layer and an output layer connected in sequence, and the behavior recognition unit 14 is specifically configured to: perform convolution on the target joint sequence by using the first convolution layer, and perform pooling by using the first pooling layer; then perform convolution by using the second convolution layer, and perform pooling by using the second pooling layer; then perform convolution by using the third convolution layer, and perform pooling by using the third pooling layer, and then perform full connection by using the full connection layer to obtain a processed vector sequence; and input the vector sequence to the output layer, and perform classification recognition on the vector sequence by using a Softmax classifier in the output layer.

[0140] In the embodiment, the behavior categories include sprinting, climbing, crossing, leaning, throwing and normal, and the illegal behavior identification unit 15 is specifically configured to: if the edge environment of the dangerous area is a hole edge or a well edge, and the behavior category of the target construction personnel is sprinting, crossing or throwing, it is determined that the target construction personnel has an illegal behavior; if the edge environment of the dangerous area is a foundation pit edge, and the behavior category of the target construction personnel is sprinting or leaning, it is determined that the target construction personnel has an illegal behavior; if the edge environment of the dangerous area is a material unloading platform edge or a roof periphery, and the behavior category of the target construction personnel is climbing or leaning, it is determined that the target construction personnel has an illegal behavior; if the edge environment of the dangerous area is a high-rise window edge, and the behavior category of the target construction personnel is climbing, leaning or throwing, it is determined that the target construction personnel has an illegal behavior; and the target construction personnel who has an illegal behavior is determined as an illegal person.

[0141] Please refer to Figure 6 , Figure 6 A structural block diagram of an electronic device 20 provided in the embodiment is shown in FIG. 1.

[0142] For example, the electronic device 20 can include a communication module 22 connected with the outside through a network, one or more processors 24 for executing program instructions, a bus 23 and different forms of memories 21, such as a disk, a ROM or a RAM, or any combination thereof. The memories 21, the communication module 22 and the processors 24 can be connected through the bus 23.

[0143] For example, the memories 21 store programs. The processors 24 can call and run the programs from the memories 21, so that the method for alarming illegal behavior based on artificial intelligence can be implemented by running the programs.

[0144] The embodiment of the present application further provides a storage medium, which comprises a stored program, wherein the device where the storage medium is located executes the method for warning illegal behavior based on artificial intelligence when the program runs.

[0145] To sum up, the embodiment of the present application provides a method, system, medium and device for warning illegal behavior based on artificial intelligence, which identifies the background environment by using the to-be-identified image, determines the dangerous area of the construction site, judges whether the target construction personnel in the dangerous area exist in the to-be-identified image, processes the to-be-identified image to obtain a target image containing the target construction personnel, inputs the target image into a preset behavior recognition model, and identifies the behavior category of the target construction personnel by using the behavior recognition model, so as to judge whether the target construction personnel violates the rules in combination with the dangerous area and the behavior category of the target construction personnel. In this way, the complex environment of the construction site can be considered to determine the dangerous area, and the behavior category of the construction personnel in the dangerous area is monitored, so that the complex environment of the construction site can be applied, and the construction personnel can be effectively supervised.

[0146] In the embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other manners. The above described system embodiments are only schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0147] In this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0148] The above only describes the embodiments of the present application, and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for issuing traffic violation alerts based on artificial intelligence, characterized in that, The method comprises the following steps: acquiring an image to be recognized, wherein the image to be recognized contains a construction site and a construction worker; performing background environment recognition on the image to be recognized to determine a dangerous area of the construction site; judging whether a target construction worker located in the dangerous area exists in the image to be recognized; if so, processing the image to be recognized to obtain a target image containing the target construction worker, inputting the target image into a preset behavior recognition model, and acquiring a behavior category of the target construction worker output by the behavior recognition model after recognition; based on the dangerous area and the behavior category of the target construction worker, determining whether a rule-violating worker exists, and generating a rule-violating behavior prompt information to prompt the rule violation when the rule-violating worker exists; the behavior recognition model comprises a joint sequence capturing module and a behavior category classification module, the target image is input into the behavior recognition model, the behavior category of the target construction worker output by the behavior recognition model after recognition is acquired, and the method comprises the following steps: the target image is input into the joint sequence capturing module, and the target joint sequence corresponding to the target image is calculated by the joint sequence capturing module; the target joint sequence is input into the behavior category classification module, the target joint sequence is classified and recognized by the behavior category classification module, and the behavior category of the target construction worker is output after being determined; the joint sequence capturing module calculates the target joint sequence corresponding to the target image in the following manner: the target joint sequence corresponding to the target image is calculated by using the following formula: where f out is the output of the joint sequence capture module, k v is the kernel size of the spatial dimension, here k v is 3, W k is the weight tensor, f in is the input parameter corresponding to the target image, which is a C × T × N tensor, N represents the number of nodes, C represents the number of channels, T represents the length of time, X k is an N × N adjacency matrix, representing the physical structure characteristics of human joints, Y k is also an N × N adjacency matrix, used for autonomous learning of features at different layers, Z k is a data dependency graph, used to determine the closeness of the relationship between different joint nodes, and α ∈ (0, 1) is the weight of Z k , represents the element product between two matrices, M k is an N × N attention mask graph, initialized as a matrix of all 1s; before calculating the target joint sequence corresponding to the target image, the method further comprises the following steps: the similarity between two nodes is calculated by using the following formula: Among them, v i v j All are joints corresponding to the target image, N is the total number of joints corresponding to the target image, f(v i ,v j ) is node v i With node v j The similarity between them, θ(v) i ) is an embedded function, corresponding to node v i Feature map, θ(v i ) T For θ(v) i The transpose of φ(v) j ) is an embedded function, corresponding to node v j The feature map, where δ is a constant, is used to stabilize the function θ(v i ) T φ(v j The value of ), τ∈(0,f in ] is a hyperparameter used to increase the function θ(v) i ) T φ(v j Sensitivity to low-probability candidates; Based on the embedding functions θ(v i ) and φ(v j ), the data dependency graph Z k is computed using the Log-Softmax Temperatue classification function. where f in is an input parameter, is a CxTxN tensor, f in t is the transpose of f in , and W φk are the parameter matrices corresponding to the embedding functions θ(v i ) T and θ(v j ), respectively. 2.The AI-based violation warning method of claim 1, wherein the background environment recognition is performed on the image to be recognized to determine the dangerous area of the construction site, which comprises the following steps: the image to be recognized is segmented, and a temporary edge environment in the image to be recognized is recognized, wherein the temporary edge environment comprises a hole edge, a roof periphery, a well edge, a foundation pit edge, a unloading platform edge, and a high-rise window edge; based on each temporary edge environment, a corresponding dangerous area is determined. 3.The AI-based violation warning method of claim 1, wherein the behavior category classification module is provided with a first convolution layer, a first pooling layer, a second convolution layer, a second pooling layer, a third convolution layer, a third pooling layer, a full connection layer, and an output layer connected in sequence, and the behavior category classification module classifies and recognizes the target joint sequence in the following manner: the target joint sequence is convolved by using the first convolution layer, and is pooled by using the first pooling layer; then the target joint sequence is convolved by using the second convolution layer, and is pooled by using the second pooling layer; then the target joint sequence is convolved by using the third convolution layer, and is pooled by using the third pooling layer, and then the target joint sequence is fully connected by using the full connection layer to obtain a processed vector sequence; the vector sequence is input into the output layer, and the vector sequence is classified and recognized by using a Softmax classifier in the output layer. 4.The AI-based violation warning method of claim 2, wherein The behavior categories include sprinting, climbing, crossing, leaning, throwing, and normal, and whether there is a rule-breaking person implementing a rule-breaking behavior is determined based on the dangerous area and the behavior category of the target construction worker, including: If the edge environment of the dangerous area is a hole edge or a well edge, and the behavior category of the target construction worker is sprinting, crossing, or throwing, it is determined that the target construction worker has a rule-breaking behavior; If the edge environment of the dangerous area is a foundation pit edge, and the behavior category of the target construction worker is sprinting or leaning, it is determined that the target construction worker has a rule-breaking behavior; If the edge environment of the dangerous area is a material unloading platform edge or a roof periphery, and the behavior category of the target construction worker is climbing or leaning, it is determined that the target construction worker has a rule-breaking behavior; If the edge environment of the dangerous area is a high-rise window edge, and the behavior category of the target construction worker is climbing, leaning, or throwing, it is determined that the target construction worker has a rule-breaking behavior; The target construction worker implementing a rule-breaking behavior is determined to be a rule-breaking person.

5. An AI-based violation alert system, characterized by, It includes: An image acquisition unit is configured to acquire a to-be-identified image, wherein the to-be-identified image contains a construction site and a construction worker; An environment recognition unit is configured to perform background environment recognition on the to-be-identified image to determine a dangerous area of the construction site; A judgment unit is configured to judge whether there is a target construction worker located in the dangerous area in the to-be-identified image; A behavior recognition unit is configured to, when there is a target construction worker located in the dangerous area in the to-be-identified image, process the to-be-identified image to obtain a target image containing the target construction worker, input the target image into a preset behavior recognition model, and acquire a behavior category of the target construction worker output by the behavior recognition model after recognition; A rule-breaking recognition unit is configured to determine whether there is a rule-breaking person implementing a rule-breaking behavior based on the dangerous area and the behavior category of the target construction worker, and generate a rule-breaking behavior prompt information for rule-breaking prompting when there is a rule-breaking person; The behavior recognition model includes a joint sequence capture module and a behavior category classification module, and the behavior recognition unit is specifically configured to: input the target image into the joint sequence capture module, calculate a target joint sequence corresponding to the target image through the joint sequence capture module, input the target joint sequence into the behavior category classification module, and classify and recognize the target joint sequence through the behavior category classification module to determine and output the behavior category of the target construction worker; The behavior recognition unit is specifically configured to calculate the target joint sequence corresponding to the target image by using the following formula: where f out is the output of the joint sequence capture module, k v is the kernel size of the spatial dimension, here k v is 3, W k is the weight tensor, f in is the input parameter corresponding to the target image, which is a CxTxN dimensional tensor, N represents the number of nodes, C represents the number of channels, T represents the length of time, X k is an N*N order adjacency matrix, representing the physical structure characteristics of human joints, Y k is also an N*N order adjacency matrix, used for autonomous learning of features of different layers, Z k is a data dependency graph, used to determine the close degree of contact between different joint nodes, and α∈(0,1) is the weight of Z k , represents the element product between two matrices, M k is an N*N order attention mask graph, initialized as a matrix full of 1s; The behavior recognition unit is specifically configured to calculate the similarity between two nodes by using the following formula: Among them, v i v j All are joints corresponding to the target image, N is the total number of joints corresponding to the target image, f(v i ,v j ) is node v i With node v j The similarity between them, θ(v) i ) is an embedded function, corresponding to node v i Feature map, θ(v i ) T For θ(v) i The transpose of φ(v) j ) is an embedded function, corresponding to node v j The feature map, where δ is a constant, is used to stabilize the function θ(v i ) T φ(v j The value of ), τ∈(0,f in ] is a hyperparameter used to increase the function θ(v) i ) T φ(v j Sensitivity to low-probability candidates; Based on the embedding functions θ(v i ) and φ(v j ), the data dependency graph Z k is computed using the Log-Softmax Temperatue classification function. where f in is an input parameter, is a C x T x N tensor, f in T is the transpose of f in , and W φk are the parameter matrices corresponding to the embedding functions θ(v i ) T and φ(v j ), respectively.

6. A storage medium, characterized by The storage medium includes a stored program, wherein the program controls a device where the storage medium is located to execute the rule-breaking behavior warning method based on artificial intelligence in any one of claims 1 to 4 when the program is running.

7. An electronic device, comprising: An artificial intelligence-based violation behavior warning method according to any one of claims 1 to 4, including a memory for storing information including program instructions, and a processor for controlling execution of the program instructions, the program instructions being loaded and executed by the processor.

Images