Method, electronic device and medium for detecting unsafe behavior of operators in dynamic trajectory

Abstract

The embodiments of the present disclosure disclose a method, electronic device and medium for detecting unsafe behaviors of workers in dynamic trajectories. A specific implementation of the method includes: obtaining initial user behavior video information; converting and processing the initial user behavior video information; performing image preprocessing on each initial user behavior image information in the initial user behavior image information set; inputting each user behavior image information in the user behavior image information set into a pre-trained user behavior prediction model; inputting each user behavior prediction image information in the user behavior prediction image information set into a pre-trained user behavior anomaly recognition model; sending each user behavior anomaly result in the user behavior anomaly result set that meets the preset anomaly condition to the alarm terminal for alarm processing. This implementation can improve the safety of substation operations.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the computer field, and in particular to a method, electronic device and medium for detecting unsafe behaviors of dynamic trajectories of operators. Background Art

[0002] Identifying abnormal user behaviors that occur at the substation operation site (for example, workers smoking in non-smoking areas, users reaching out to touch live equipment, etc.) and issuing alarms for abnormal user behaviors can improve the safety of substation operations. Currently, when performing abnormal user behavior detection, the usual method is to identify abnormal user behaviors in user behavior videos through manual recognition.

[0003] However, in practice, it is found that when the above method is used to detect abnormal user behavior, the following technical problems often occur:

[0004] First, it is difficult to identify abnormal user behaviors in each frame of user behavior images in user behavior videos through manual recognition, which makes it difficult to issue alarms for some abnormal user behaviors, thereby reducing the safety of substation operations.

[0005] In the process of adopting technical solutions to solve the above-mentioned technical problem one, the following problems often occur: Since there is a lot of information in the user behavior video, directly identifying abnormal user behavior in the user behavior video through the abnormal behavior recognition model will lead to a high complexity of the abnormal behavior model, resulting in the need to consume a large amount of computing resources to train the abnormal behavior model, resulting in a waste of computing resources. In response to the above problems, the conventional solution is generally: first, the watershed algorithm is used to perform image segmentation processing on each frame of the user behavior image in the user behavior video to obtain user behavior prediction image information. Then, the convolutional neural network model is used to identify the abnormal user behavior results corresponding to the user behavior prediction image information. However, the above-mentioned conventional solution still has the following technical problem two: the accuracy of segmenting the user behavior image through the watershed algorithm is low, resulting in a low accuracy of the abnormal user behavior results identified through the user behavior prediction image information with low accuracy, making it difficult to identify some abnormal user behaviors, and making it difficult to issue alarms for some abnormal user behaviors;

[0006] Third, the accuracy of abnormal user behavior identified by the convolutional neural network model is low, which makes it difficult to identify some abnormal user behaviors and difficult to issue alarms for some abnormal user behaviors.

[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0008] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0009] Some embodiments of the present disclosure propose methods, devices, electronic devices and computer-readable media for detecting unsafe behaviors of operators in dynamic trajectories to solve one or more of the technical problems mentioned in the above background technology section.

[0010] In a first aspect, some embodiments of the present disclosure provide a method for detecting unsafe behaviors of workers in dynamic trajectories, the method comprising: obtaining initial user behavior video information; converting the initial user behavior video information to generate an initial user behavior image information set; performing image preprocessing on each initial user behavior image information in the initial user behavior image information set to generate user behavior image information and obtain a user behavior image information set; inputting each user behavior image information in the user behavior image information set into a pre-trained user behavior prediction model to generate user behavior prediction image information and obtain a user behavior prediction image information set; inputting each user behavior prediction image information in the user behavior prediction image information set into a pre-trained user behavior anomaly recognition model to generate user behavior anomaly results and obtain a user behavior anomaly result set; and sending each user behavior anomaly result in the user behavior anomaly result set that meets preset anomaly conditions to an alarm terminal for alarm processing.

[0011] In a second aspect, some embodiments of the present disclosure provide a device for detecting unsafe behaviors of workers in dynamic trajectories, the device comprising: an acquisition unit configured to acquire initial user behavior video information; a conversion unit configured to perform conversion processing on the initial user behavior video information to generate an initial user behavior image information set; a preprocessing unit configured to perform image preprocessing on each initial user behavior image information in the initial user behavior image information set to generate user behavior image information and obtain a user behavior image information set; a first input unit configured to input each user behavior image information in the user behavior image information set into a pre-trained user behavior prediction model to generate user behavior prediction image information and obtain a user behavior prediction image information set; a second input unit configured to input each user behavior prediction image information in the user behavior prediction image information set into a pre-trained user behavior anomaly recognition model to generate a user behavior anomaly result and obtain a user behavior anomaly result set; a sending unit configured to send each user behavior anomaly result in the user behavior anomaly result set that meets a preset anomaly condition to an alarm terminal for alarm processing.

[0012] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0013] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0014] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the unsafe behavior detection method of the dynamic trajectory of the operator in some embodiments of the present disclosure, the safety of the substation operation can be improved. Specifically, the reason for reducing the safety of the substation operation is that it is difficult to identify the abnormal user behavior in each frame of the user behavior image in the user behavior video through manual recognition, which makes it difficult to warn some abnormal user behaviors. Based on this, the unsafe behavior detection method of the dynamic trajectory of the operator in some embodiments of the present disclosure first obtains the initial user behavior video information. Thus, the initial user behavior video information captured by the camera can be obtained. Secondly, the initial user behavior video information is converted to generate an initial user behavior image information set. Thus, the initial user behavior image information representing each frame of the image can be obtained. Then, each initial user behavior image information in the initial user behavior image information set is subjected to image preprocessing to generate user behavior image information and obtain a user behavior image information set. Thus, the initial user behavior image can be subjected to grayscale processing, interpolation processing, and noise reduction processing to eliminate redundant information in the initial user behavior image and improve the quality of the initial user behavior image. Next, each user behavior image information in the above user behavior image information set is input into a pre-trained user behavior prediction model to generate user behavior prediction image information and obtain a user behavior prediction image information set. Thus, the user behavior prediction image information that more accurately represents the user behavior in the user behavior image information can be identified through the pre-trained user behavior prediction model. Then, each user behavior prediction image information in the above user behavior prediction image information set is input into a pre-trained user behavior abnormality recognition model to generate a user behavior abnormality result and obtain a user behavior abnormality result set. Thus, the user behavior abnormality result that represents whether the user behavior is abnormal in the more accurate user behavior prediction image information can be identified through the pre-trained user behavior abnormality recognition model and the more accurate user behavior prediction image information. Finally, each user behavior abnormality result that meets the preset abnormality condition in the above user behavior abnormality result set is sent to the alarm terminal for alarm processing. Thus, the user behavior abnormality result that represents the abnormality of the user behavior can be alarmed through the more accurate user behavior abnormality result. Thus, the safety of substation operation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0016] Figure 1is a flow chart of some embodiments of the method for detecting unsafe behaviors of workers according to dynamic trajectories of the present disclosure;

[0017] Figure 2 It is a schematic diagram of the structure of some embodiments of the device for detecting unsafe behavior of workers in dynamic trajectory according to the present disclosure;

[0018] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] Figure 1 1 is a flowchart of some embodiments of the method for detecting unsafe behaviors of workers in dynamic trajectories according to the present disclosure. 100 of some embodiments of the method for detecting unsafe behaviors of workers in dynamic trajectories according to the present disclosure is shown. The method for detecting unsafe behaviors of workers in dynamic trajectories comprises the following steps:

[0026] Step 101, obtaining initial user behavior video information.

[0027] In some embodiments, the execution body of the method for detecting unsafe behaviors of operators in dynamic trajectories may obtain the initial user behavior video information from the camera by means of a wired connection or a wireless connection. The above-mentioned initial user behavior video information may represent: a video representing user behaviors (e.g., smoking, climbing over a fence, touching live equipment, etc.) within a preset time period. The initial user behavior video information may include: an initial user behavior frame image information sequence. Here, the arrangement order of the initial user behavior frame image information in the initial user behavior frame image information sequence may be a time order. The initial user frame image information in the initial user behavior frame image information sequence may represent: a frame of image included in the video represented by the initial user behavior video information. The camera may be a camera that captures user behaviors in substation operations. For example, the preset time period may be a time period from ten minutes before the current time to the current time. The user may refer to an operator.

[0028] It should be noted that the above wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0029] Step 102: convert the initial user behavior video information to generate an initial user behavior image information set.

[0030] In some embodiments, the execution subject may convert the initial user behavior video information to generate an initial user behavior image information set. In practice, the execution subject may convert the initial user behavior video information using a preset conversion algorithm to generate an initial user behavior image information set. For example, the preset conversion algorithm may be: determining each initial user behavior frame image information in the initial user behavior frame image information sequence included in the initial user behavior video information as initial user behavior image information, and obtaining an initial user behavior image information set.

[0031] Step 103 , performing image preprocessing on each initial user behavior image information in the initial user behavior image information set to generate user behavior image information, thereby obtaining a user behavior image information set.

[0032] In some embodiments, the execution entity may perform image preprocessing on each initial user behavior image information in the initial user behavior image information set to generate user behavior image information and obtain the user behavior image information set.

[0033] In practice, the execution subject may perform image preprocessing on each initial user behavior image information in the initial user behavior image information set through the following steps to generate user behavior image information:

[0034] The first step is to grayscale the initial user behavior image information to generate user grayscale image information. In practice, the execution subject may grayscale the initial user behavior image information using a preset grayscale algorithm to generate user grayscale image information. For example, the preset grayscale algorithm may be: component method, maximum value method, average value method, weighted average method.

[0035] The second step is to interpolate the user grayscale image information to generate user interpolation image information. In practice, the execution subject can interpolate the user grayscale image information by using a preset interpolation algorithm to generate user interpolation image information. For example, the preset interpolation algorithm can be: nearest neighbor interpolation algorithm, bilinear interpolation algorithm, bicubic interpolation algorithm.

[0036] The third step is to perform noise reduction processing on the user interpolation image information to generate user behavior image information. In practice, the execution subject can perform noise reduction processing on the user interpolation image information by using a preset noise reduction algorithm to generate user behavior image information. For example, the preset noise reduction algorithm can be: non-local mean algorithm, median filter algorithm, Gaussian low-pass filter algorithm.

[0037] Therefore, firstly, the initial user behavior image can be grayed out by using a preset graying algorithm to convert the color image into a gray image to improve the operation speed. Then, the user gray image can be interpolated by using a preset interpolation algorithm to restore the lost information in the image. After that, the user interpolation image information can be denoised by using a preset denoising algorithm to reduce the noise in the image and improve the image quality.

[0038] Step 104 , input each user behavior image information in the user behavior image information set into a pre-trained user behavior prediction model to generate user behavior prediction image information, thereby obtaining a user behavior prediction image information set.

[0039] In some embodiments, the execution subject may input each user behavior image information in the user behavior image information set into a pre-trained user behavior prediction model to generate user behavior prediction image information and obtain a user behavior prediction image information set. The pre-trained user behavior prediction model may be a neural network model that takes user behavior image information as input and takes user behavior prediction image information as output. The user behavior prediction image information in the user behavior prediction image information set may represent: an image of a specific location in the user behavior image information that represents the user behavior.

[0040] Optionally, the pre-trained user behavior prediction model may be trained through the following steps:

[0041] The first step is to obtain the first training sample set.

[0042] In some embodiments, the execution subject may obtain the first training sample set from the terminal device via a wired connection or a wireless connection. The first training sample in the first training sample set may include: sample user behavior image information and sample user behavior prediction image information. Here, the sample user behavior prediction image information may be a sample label corresponding to the sample user behavior image information.

[0043] The second step is to determine the initial user behavior prediction model.

[0044] In some embodiments, the execution subject may determine an initial user behavior prediction model. The initial user behavior prediction model may be a neural network model that takes sample user behavior image information as input and initial user behavior prediction image information as output. Here, the initial user behavior prediction model may include but is not limited to: an initial feature extraction model and an initial screening model.

[0045] Among them, the initial feature extraction model can be a model that takes sample user behavior image information as input and takes initial user behavior image feature information as output. The initial feature extraction model may include: an initial convolution model sequence. The initial convolution model in the initial convolution model sequence may include: an initial convolution sub-model and an initial activation model. Here, the input of the initial convolution sub-model included in the initial convolution model in the initial convolution model sequence may be: the output of the initial activation model included in the previous initial convolution model. The input of the initial convolution sub-model included in the first initial convolution model in the initial convolution model sequence may be: sample user behavior image information. The input of the initial activation model of the initial convolution model in the initial convolution model sequence may be: the output of the initial convolution sub-model included in the initial convolution model. For example, the number of initial convolution models in the initial convolution model sequence may be 22.

[0046] For example, the initial convolution sub-model may include but is not limited to at least one of the following: a 1×1 convolution kernel, a 3×3 convolution kernel. The initial activation model may include the following formula:

[0047] .

[0048] in, Represents the initial activation model. Represents the input of the initial activation model. Indicates the preset activation parameter (e.g. 0.1).

[0049] The initial screening model may be a model that takes the initial user behavior image feature information as input and takes the initial user behavior prediction image information as output. The initial screening model is used to screen the initial user behavior image feature information through a preset screening algorithm to generate the initial user behavior prediction image information. For example, the preset screening algorithm may be an NMS (Non Maximum Suppression) algorithm.

[0050] The third step is to select the first training sample from the first training sample set.

[0051] In some embodiments, the execution entity may select the first training sample from the first training sample set. In practice, the execution entity may randomly select the first training sample from the first training sample set.

[0052] In the fourth step, the sample user behavior image information included in the selected first training sample is input into the initial feature extraction model to obtain the initial user behavior image feature information.

[0053] In some embodiments, the execution entity may input the sample user behavior image information included in the selected first training sample into an initial feature extraction model to obtain initial user behavior image feature information.

[0054] The fifth step is to input the initial user behavior image feature information into the initial screening model to obtain the initial user behavior prediction image information.

[0055] In some embodiments, the execution entity may input the initial user behavior image feature information into the initial screening model to obtain the initial user behavior prediction image information.

[0056] In the sixth step, based on a preset loss function, a prediction difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample is determined.

[0057] In some embodiments, based on a preset loss function, the execution subject may determine a predicted difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample. The preset loss function may include: a first preset loss function, a second preset loss function, a third preset loss function, a fourth preset loss function, and a fifth preset loss function.

[0058] In practice, based on the preset loss function, the execution subject may determine the prediction difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample through the following sub-steps:

[0059] The first sub-step is to determine the first difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample based on the first preset loss function. The first preset loss function may be:

[0060] .

[0061] in, Indicates the first difference value. Indicates a preset first balancing parameter (eg 5). Indicates the number of rows of all grid cells (e.g. 7). Here, the number of columns of grid cells is the same as the number of rows. Represents the number of all predicted boxes (e.g. 2). Indicates the grid cell number. Indicates the ordinal number of the prediction box. Indicated in The first grid cell Is the prediction box responsible for predicting the initial user behavior prediction image information? When it is 1, it means The first grid cell The prediction box is responsible for predicting the initial user behavior prediction image information. When it is 0, it means The first grid cell The prediction box is not responsible for predicting the initial user behavior prediction image information. ). Indicated in The horizontal coordinate of the sample user behavior prediction image information in each grid cell. Indicated in The horizontal coordinate of the initial user behavior prediction image information predicted in each grid cell. Indicated in The vertical coordinate of the sample user behavior prediction image information in each grid cell. Indicated in The vertical coordinate of the initial user behavior prediction image information predicted in each grid cell.

[0062] The second sub-step is to determine the second difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample based on the second preset loss function. The second preset loss function may be:

[0063] .

[0064] in, Represents the second difference value. Indicated in The width of the sample user behavior prediction image information in grid cells. Indicated in The width of the initial user behavior prediction image information predicted in grid cells. Indicated in The height of the sample user behavior prediction image information in each grid cell. Indicated in The height of the initial user behavior prediction image information predicted in each grid cell.

[0065] The third sub-step is to determine the third difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample based on the third preset loss function. The third preset loss function may be:

[0066] .

[0067] in, Represents the third difference value. Indicated in The confidence level corresponding to the sample user behavior prediction image information in each grid cell. Indicated in The confidence level corresponding to the initial user behavior prediction image information predicted in each grid cell.

[0068] The fourth sub-step is to determine a fourth difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample based on the fourth preset loss function. The fourth preset loss function may be:

[0069] .

[0070] in, Represents the fourth difference value. Indicates a preset second balance parameter (eg 0.5). Indicated in The first grid cell Is the prediction box not responsible for predicting the initial user behavior prediction image information? When it is 1, it means The first grid cell The prediction box is not responsible for predicting the initial user behavior prediction image information. When it is 0, it means The first grid cell The prediction box is responsible for predicting the initial user behavior prediction image information. ).

[0071] The fifth sub-step is to determine the fifth difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample based on the fifth preset loss function. The fifth preset loss function may be:

[0072] .

[0073] in, Represents the fifth difference value. Indicates whether the initial user behavior prediction image information appears in the In the grid cells ( When it is 1, it means that the initial user behavior prediction image information appears in the in a grid unit. When it is 0, it means that the initial user behavior prediction image information does not appear in the grid cells. ). Represents the category of the initial user behavior prediction image information. Represents various categories of initial user behavior prediction image information. Indicates The corresponding appearance of sample user behavior prediction image information in grid cells The probability of predicting image information based on user behavior of class samples. Indicates The initial user behavior prediction image information corresponding to the appearance of The probability of class initial user behavior predicting image information.

[0074] In a sixth sub-step, the sum of the first difference value, the second difference value, the third difference value, the fourth difference value and the fifth difference value is determined as the predicted difference value.

[0075] In a seventh step, in response to determining that the predicted difference value is greater than or equal to the preset difference value, the network parameters of the initial user behavior prediction model are adjusted.

[0076] In some embodiments, in response to determining that the predicted difference value is greater than or equal to the preset difference value, the execution subject may adjust the network parameters of the initial user behavior prediction model. For example, the predicted difference value and the preset difference value may be differentiated. On this basis, the network parameters of the initial user behavior prediction model are adjusted using methods such as back propagation and gradient descent. There is no limitation on the setting of the preset difference value, for example, the preset difference value may be 0.1.

[0077] Optionally, in response to determining that the predicted difference value is less than the preset difference value, the initial user behavior prediction model is determined as the trained user behavior prediction model.

[0078] In some embodiments, the execution entity may determine the initial user behavior prediction model as the trained user behavior prediction model in response to determining that the predicted difference value is less than the preset difference value.

[0079] The relevant content of step 104, as an inventive point of an embodiment of the present disclosure, solves the technical problem 2 mentioned in the background technology, "making it difficult to perform alarm processing on some abnormal user behaviors". Among them, the factors that make it difficult to perform alarm processing on some abnormal user behaviors are often as follows: the accuracy of segmenting the user behavior image by the watershed algorithm is low, resulting in the low accuracy of the abnormal user behavior results identified by the user behavior prediction image information with low accuracy, making it difficult to identify some abnormal user behaviors. If the above factors are solved, the effect of performing alarm processing on some abnormal user behaviors can be achieved. In order to achieve this effect, the present disclosure can replace the watershed algorithm to identify the user behavior prediction image information by the user behavior prediction model, and the user behavior prediction model includes an initial feature extraction model, an initial screening model, and the user behavior prediction model also takes into account five loss functions. Therefore, the horizontal coordinate, vertical coordinate, width, height, confidence and category loss of the user behavior prediction image information can be taken into account by five loss functions. Therefore, a relatively accurate user behavior prediction model can be trained by including the initial feature extraction model and the initial screening model and five loss functions. Therefore, more accurate user behavior prediction image information can be predicted based on a more accurate user behavior prediction model, so that more accurate abnormal user behavior can be identified through the more accurate user behavior prediction image information. Furthermore, alarm processing can be performed on some abnormal user behaviors.

[0080] Step 105 , input each user behavior prediction image information in the user behavior prediction image information set into a pre-trained user behavior anomaly recognition model to generate a user behavior anomaly result, thereby obtaining a user behavior anomaly result set.

[0081] In some embodiments, the above-mentioned execution entity may input each user behavior prediction image information in the above-mentioned user behavior prediction image information set into a pre-trained user behavior anomaly recognition model to generate a user behavior anomaly result and obtain a user behavior anomaly result set. Among them, the pre-trained user behavior anomaly recognition model may be a neural network model that takes user behavior prediction image information as input and takes user behavior anomaly results as output. Here, the user behavior anomaly results in the user behavior anomaly result set may characterize the presence of unsafe behaviors in the user behavior (for example, smoking, climbing over fences, touching live equipment, etc.). The user behavior anomaly results in the user behavior anomaly result set may also characterize the absence of unsafe behaviors in the user behavior.

[0082] Optionally, the pre-trained user behavior abnormality recognition model may be trained through the following steps:

[0083] The first step is to obtain the second training sample set.

[0084] In some embodiments, the execution subject may obtain the second training sample set from the terminal device via a wired connection or a wireless connection. The second training samples in the second training sample set may include: sample user behavior prediction image information and sample user behavior abnormality results.

[0085] The second step is to determine the initial user behavior anomaly recognition model.

[0086] In some embodiments, the execution subject may determine an initial user behavior anomaly recognition model, wherein the initial user behavior anomaly recognition model may include: an initial aggregation model, an initial perception model, an initial enhancement model, and an initial recognition convolution model.

[0087] Here, the initial aggregation model may be a neural network model that takes sample user behavior prediction image information as input and takes initial user image aggregation information as output. For example, the initial aggregation model may be a TCA (Temporal Context Aggregation) model.

[0088] The initial perception model may be a neural network model that takes initial user image aggregation information as input and takes initial user image perception information as output. For example, the initial perception model may be an MLP (Multilayer Perceptron) model.

[0089] The initial enhancement model may be a neural network model that takes initial user image perception information as input and takes initial user image enhancement information as output. For example, the initial enhancement model may be a PEL (Presupposition Evidence Logic, enhanced prompt) model.

[0090] The initial recognition convolution model may be a neural network model that takes the initial user image enhancement information as input and takes the initial user behavior abnormality result as output. For example, the initial recognition convolution model may be a CC (causal convolution) model.

[0091] The third step is to select the second training sample from the second training sample set.

[0092] In some embodiments, the execution entity may select the second training sample from the second training sample set. In practice, the execution entity may randomly select the second training sample from the second training sample set.

[0093] In the fourth step, the sample user behavior prediction image information included in the selected second training sample is input into the initial aggregation model to obtain initial user image aggregation information.

[0094] In some embodiments, the execution entity may input the sample user behavior prediction image information included in the selected second training sample into the initial aggregation model to obtain initial user image aggregation information.

[0095] The fifth step is to input the initial user image aggregation information into the initial perception model to obtain the initial user image perception information.

[0096] In some embodiments, the execution entity may input the initial user image aggregation information into the initial perception model to obtain the initial user image perception information.

[0097] In the sixth step, the initial user image perception information is input into the initial enhancement model to obtain the initial user image enhancement information.

[0098] In some embodiments, the execution entity may input the initial user image perception information into the initial enhancement model to obtain the initial user image enhancement information.

[0099] In the seventh step, the initial user image enhancement information is input into the initial recognition convolutional model to obtain the initial user behavior abnormality results.

[0100] In some embodiments, the above-mentioned execution entity can input the initial user image enhancement information into the initial recognition convolution model to obtain the initial user behavior abnormality result.

[0101] In the eighth step, based on a preset recognition loss function, a recognition difference value between the initial user behavior abnormal result and the sample user behavior abnormal result included in the selected second training sample is determined.

[0102] In some embodiments, based on a preset recognition loss function, the execution subject may determine the recognition difference value between the initial user behavior abnormality result and the sample user behavior abnormality result included in the selected second training sample. The preset recognition loss function may be, but is not limited to, mean square error loss function (MSE), hinge loss function (SVM), cross entropy loss function (CrossEntropy), 0-1 loss function, absolute value loss function, log loss function, square loss function, exponential loss function, etc.

[0103] In a ninth step, in response to determining that the recognition difference value is greater than or equal to a preset recognition difference value, adjusting network parameters of the initial user behavior abnormality recognition model.

[0104] In some embodiments, the execution subject may adjust the network parameters of the initial user behavior abnormality recognition model in response to determining that the recognition difference value is greater than or equal to the preset recognition difference value. For example, the recognition difference value and the preset recognition difference value may be differentiated. On this basis, the network parameters of the initial user behavior prediction model are adjusted using methods such as back propagation and gradient descent. There is no limitation on the setting of the preset recognition difference value, for example, the preset recognition difference value may be 0.1.

[0105] Optionally, in response to determining that the recognition difference value is less than the preset recognition difference value, the initial user behavior abnormality recognition model is determined as the trained user behavior abnormality recognition model.

[0106] In some embodiments, the execution entity may determine the initial user behavior abnormality recognition model as the trained user behavior abnormality recognition model in response to determining that the recognition difference value is less than the preset recognition difference value.

[0107] The relevant content of step 105, as an inventive point of an embodiment of the present disclosure, solves the technical problem three mentioned in the background technology, "making it difficult to perform alarm processing on some abnormal user behaviors". Among them, the factors that make it difficult to perform alarm processing on some abnormal user behaviors are often as follows: the accuracy of the abnormal user behavior results identified by the convolutional neural network model is low, which makes it difficult to identify some abnormal user behaviors. If the above factors are solved, the effect of being able to perform alarm processing on some abnormal user behaviors can be achieved. In order to achieve this effect, first, through the initial aggregation model, the initial user image aggregation information considering the time and context relationship can be more accurately identified. Then, through the initial perception model, the feature information in the initial user image aggregation information can be more accurately identified. Then, through the initial enhancement model, more accurate initial user image enhancement information considering the user behavior logic can be obtained. After that, through the initial recognition convolution model, more accurate initial user behavior abnormal results considering the time sequence can be obtained. Therefore, a more accurate user behavior abnormal result can be obtained by training a user behavior abnormality recognition model including an initial aggregation model, an initial perception model, an initial enhancement model, and an initial recognition convolution model. Thus, the abnormal user behavior can be more accurately identified through the more accurate user behavior abnormal results. Therefore, alarm processing can be performed on some abnormal user behaviors.

[0108] Step 106 , sending each user behavior abnormal result that meets the preset abnormal condition in the user behavior abnormal result set to the alarm terminal for alarm processing.

[0109] In some embodiments, the execution subject may send each abnormal user behavior result in the abnormal user behavior result set that meets the preset abnormal condition to the alarm terminal for alarm processing. The preset abnormal condition may be: the abnormal user behavior result indicates that the user behavior is unsafe. The alarm processing may be to display warning text or control the speaker to emit a prompt sound to warn the monitoring terminal that the user behavior is abnormal.

[0110] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an abnormal user behavior detection device. These abnormal user behavior detection device embodiments are Figure 1 Corresponding to the method embodiments shown, the abnormal user behavior detection device can be specifically applied to various electronic devices.

[0111] like Figure 2 As shown, the abnormal user behavior detection device 200 of some embodiments includes: an acquisition unit 201, a conversion unit 202, a pre-processing unit 203, a first input unit 204, a second input unit 205 and a sending unit 206. Among them, the acquisition unit 201 is configured to acquire initial user behavior video information; the conversion unit 202 is configured to perform conversion processing on the above-mentioned initial user behavior video information to generate an initial user behavior image information set; the preprocessing unit 203 is configured to perform image preprocessing on each initial user behavior image information in the above-mentioned initial user behavior image information set to generate user behavior image information and obtain a user behavior image information set; the first input unit 204 is configured to input each user behavior image information in the above-mentioned user behavior image information set into a pre-trained user behavior prediction model to generate user behavior prediction image information and obtain a user behavior prediction image information set; the second input unit 205 is configured to input each user behavior prediction image information in the above-mentioned user behavior prediction image information set into a pre-trained user behavior anomaly recognition model to generate a user behavior anomaly result and obtain a user behavior anomaly result set; the sending unit 206 is configured to send each user behavior anomaly result in the above-mentioned user behavior anomaly result set that meets the preset anomaly condition to the alarm terminal for alarm processing.

[0112] It is understandable that the units described in the abnormal user behavior detection device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the abnormal user behavior detection device 200 and the units included therein, and will not be described in detail here.

[0113] Reference below Figure 3, which shows a schematic diagram of the structure of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0114] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0115] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0116] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0117] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0118] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0119] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: performs image detection on the target page image to generate page image detection information, wherein the above-mentioned page image detection information includes: page detection image; determines the image similarity between the above-mentioned page detection image and each basic component image in the pre-established basic component image library to obtain an image similarity group; determines the basic component corresponding to the maximum image similarity in the above-mentioned image similarity group as the target basic component; stores the above-mentioned target basic component and the corresponding position coordinate information in the form of a mapping object in a temporary memory; in response to detecting a user's click operation on the selection box mode on the user interaction page, monitors the mouse on the above-mentioned user interaction page; A frame selection operation is performed on the target page image on the surface to determine the circled range corresponding to the frame selection operation; a frame selection tool is used to capture the circled image corresponding to the above circled range, and the above circled image is stored in the above temporary memory; the similarity between the above circled image and each component image in the pre-established component image resource library is determined to obtain a similarity group; based on the above similarity group, a target component image is determined; the above circled range is compared with the above position coordinate information; in response to determining that the above circled range contains the coordinate range corresponding to the above position coordinate information, all the coordinate information and the corresponding component information covered in the above circled range are determined; based on the above target component image, all the above coordinate information and the corresponding component information, a front-end page code template text corresponding to the above target page image is generated.

[0120] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising: an acquisition unit, a conversion unit, a preprocessing unit, a first input unit, a second input unit and a sending unit. Among them, the names of these units do not constitute a limitation on the units themselves in certain circumstances, for example, the sending unit may also be described as "a unit that sends each user behavior abnormality result in the above-mentioned user behavior abnormality result set that meets the preset abnormality condition to the alarm terminal for alarm processing".

[0123] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0124] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for detecting unsafe behaviors of operators in dynamic trajectories, comprising: Obtain initial user behavior video information; Converting the initial user behavior video information to generate an initial user behavior image information set; Performing image preprocessing on each initial user behavior image information in the initial user behavior image information set to generate user behavior image information, thereby obtaining a user behavior image information set; Inputting each user behavior image information in the user behavior image information set into a pre-trained user behavior prediction model to generate user behavior prediction image information, thereby obtaining a user behavior prediction image information set; Inputting each user behavior prediction image information in the user behavior prediction image information set into a pre-trained user behavior anomaly recognition model to generate a user behavior anomaly result, thereby obtaining a user behavior anomaly result set; Sending each user behavior abnormal result that meets the preset abnormal condition in the user behavior abnormal result set to the alarm terminal for alarm processing; The pre-trained user behavior prediction model is obtained by training through the following steps: Acquire a first training sample set, wherein the first training sample in the first training sample set includes: sample user behavior image information and sample user behavior prediction image information; Determine an initial user behavior prediction model, wherein the initial user behavior prediction model includes: an initial feature extraction model and an initial screening model, wherein the initial feature extraction model includes: an initial convolution model sequence, and the initial convolution model in the initial convolution model sequence includes: an initial subconvolution model and an initial activation model; Selecting a first training sample from the first training sample set; Inputting the sample user behavior image information included in the selected first training sample into the initial feature extraction model to obtain initial user behavior image feature information; Inputting the initial user behavior image feature information into the initial screening model to obtain the initial user behavior prediction image information; Determine a prediction difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample based on a preset loss function, wherein the preset loss function includes: a first preset loss function, a second preset loss function, a third preset loss function, a fourth preset loss function, and a fifth preset loss function; In response to determining that the predicted difference value is greater than or equal to a preset difference value, adjusting a network parameter of the initial user behavior prediction model; Wherein, determining the prediction difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample based on the preset loss function includes: Based on the first preset loss function, determining a first difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample; Based on the second preset loss function, determining a second difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample; Based on the third preset loss function, determining a third difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample; Based on the fourth preset loss function, determining a fourth difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample; Based on the fifth preset loss function, determining a fifth difference value between the initial user behavior prediction image information and the sample user behavior prediction image information included in the selected first training sample; Determine the sum of the first difference value, the second difference value, the third difference value, the fourth difference value and the fifth difference value as the predicted difference value; The pre-trained user behavior abnormality recognition model is obtained by training through the following steps: Acquire a second training sample set, wherein the second training samples in the second training sample set include: sample user behavior prediction image information and sample user behavior abnormality results; Determine an initial user behavior anomaly recognition model, wherein the initial user behavior anomaly recognition model includes: an initial aggregation model, an initial perception model, an initial enhancement model, and an initial recognition convolution model; Selecting a second training sample from the second training sample set; Inputting sample user behavior prediction image information included in the selected second training sample into the initial aggregation model to obtain initial user image aggregation information; Inputting the initial user image aggregation information into the initial perception model to obtain the initial user image perception information; Inputting the initial user image perception information into the initial enhancement model to obtain the initial user image enhancement information; Input the initial user image enhancement information into the initial recognition convolution model to obtain the initial user behavior abnormality result; Based on a preset recognition loss function, determining a recognition difference value between an initial user behavior abnormal result and a sample user behavior abnormal result included in the selected second training sample; In response to determining that the recognition difference value is greater than or equal to the preset recognition difference value, a network parameter of the initial user behavior abnormality recognition model is adjusted.

2. The method for detecting unsafe behaviors of operators in dynamic trajectories according to claim 1, wherein: The performing image preprocessing on each initial user behavior image information in the initial user behavior image information set to generate user behavior image information includes: Performing grayscale processing on the initial user behavior image information to generate user grayscale image information; Performing interpolation processing on the user grayscale image information to generate user interpolation image information; The user interpolation image information is subjected to noise reduction processing to generate user behavior image information.

3. The method for detecting unsafe behaviors of operators in dynamic trajectories according to claim 1, wherein: The method further comprises: In response to determining that the predicted difference value is less than the preset difference value, the initial user behavior prediction model is determined as the trained user behavior prediction model.

4. A device for detecting unsafe behaviors of dynamic trajectories of operators, applied to the method according to claim 1, comprising: an acquisition unit, configured to acquire initial user behavior video information; a conversion unit, configured to perform conversion processing on the initial user behavior video information to generate an initial user behavior image information set; a preprocessing unit configured to perform image preprocessing on each initial user behavior image information in the initial user behavior image information set to generate user behavior image information and obtain the user behavior image information set; A first input unit is configured to input each user behavior image information in the user behavior image information set into a pre-trained user behavior prediction model to generate user behavior prediction image information and obtain a user behavior prediction image information set; A second input unit is configured to input each user behavior prediction image information in the user behavior prediction image information set into a pre-trained user behavior anomaly recognition model to generate a user behavior anomaly result, thereby obtaining a user behavior anomaly result set; The sending unit is configured to send each abnormal user behavior result in the abnormal user behavior result set that meets the preset abnormal condition to the alarm terminal for alarm processing.

5. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

6. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images