A network model training method and device for detecting smoking behavior

Abstract

The present application provides a network model training method and device for detecting smoking behavior. The network model training method includes: establishing an initial network model; obtaining a training data set about smoking behavior, and annotating the training data set to obtain annotation information about the training data set; inputting the training data set and its annotation information into a first-stage network model to generate a detection frame for each image in the training data set; inputting each image in the training data set and its detection frame into the second-stage network model to generate a judgment result about smoking behavior in each image in the training data set; training the first-stage network model and the second-stage network model based on the judgment result to obtain a preset network model for detecting smoking behavior. In the above manner, the network model training method of the present application can obtain a network model for detecting smoking behavior, which solves the limitations of traditional methods and has good accuracy.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a network model training method and device for detecting smoking behavior. Background Art

[0002] Smoking is one of the top ten prohibitions on construction sites. If it is not controlled, it will increase the possibility of fire, resulting in major property losses and casualties. At present, most construction sites use smoke detectors and manual monitoring to control smoking, but smoke sensors are limited by space. In outdoor scenes, that is, when the space is large, the accuracy of smoke sensor detection will decrease, and manual monitoring methods waste manpower and cannot achieve real-time detection. Therefore, accurately and real-time detection of smoking behavior at construction sites is a difficult problem that needs to be solved urgently.

[0003] The traditional method of detecting unsafe behaviors of workers is based on manual detection, which is completely dependent on manpower. The accuracy of detection depends on the concentration of the detector. As the coverage area of ​​construction sites becomes larger and larger, it becomes increasingly difficult to rely on manpower to detect whether workers on the construction site have unsafe behaviors. Summary of the invention

[0004] The present application provides a network model training method and device for detecting smoking behavior.

[0005] The present application provides a network model training method for detecting smoking behavior, the network model training method comprising:

[0006] Establishing an initial network model, wherein the initial network model includes a first-stage network model and a second-stage network model;

[0007] Acquiring a training data set about the smoking behavior, and annotating the training data set to obtain annotated information about the training data set;

[0008] Inputting the training data set and its annotation information into the first-stage network model to generate a detection frame for each image in the training data set;

[0009] Inputting each image and its detection frame in the training data set into the two-stage network model to generate a judgment result on smoking behavior in each image in the training data set;

[0010] The first-stage network model and the second-stage network model are trained based on the judgment result to obtain a preset network model for detecting smoking behavior.

[0011] Wherein, the step of obtaining a training data set about the smoking behavior includes:

[0012] Under different external conditions, images of smoking behaviors in preset scenes are collected to obtain smoking behavior images;

[0013] Under different external conditions, images of non-smoking behaviors in preset scenes are collected to obtain images of non-smoking behaviors;

[0014] The training data set is established based on the smoking behavior images and the non-smoking behavior images.

[0015] The step of labeling the training data set includes:

[0016] Annotating a worker detection frame and a worker skeleton point in each image in the training data set;

[0017] The worker skeleton points include basic skeleton points and hand skeleton points.

[0018] The step of inputting the training data set and its annotation information into the first-stage network model includes:

[0019] Normalizing all images in the training data set;

[0020] The normalized image and its annotation information are input into the one-stage network model.

[0021] The loss function of the first-stage network model includes a detection box classification loss function and a detection box coordinate loss function;

[0022] Among them, the detection frame classification loss function is used to learn the relationship between the confidence of person classification in the predicted detection frame and the confidence of person classification in the labeled detection frame, and the detection frame coordinate loss function is used to learn the relationship between the coordinate position of the predicted detection frame and the coordinate position of the labeled detection frame.

[0023] Before inputting each image and its detection box in the training data set into the two-stage network model, the network model training method further includes:

[0024] Using a preset algorithm to screen the detection frame in the image;

[0025] The filtered detection frame is fused with the corresponding image.

[0026] The step of generating a judgment result on smoking behavior in each image in the training data set includes:

[0027] Get pre-labeled real smoking actions;

[0028] Inputting each image and its detection frame in the training data set into the two-stage network model to obtain a preset smoking action in the detection frame of each image;

[0029] Calculating the posture distance between the actual smoking action and the preset smoking action;

[0030] When the gesture distance is less than a preset threshold, it is confirmed that smoking behavior exists within the detection frame range of the image.

[0031] The step of calculating the posture distance between the actual smoking action and the preset smoking action includes:

[0032] Obtaining the number of joints that match the actual smoking action with the preset smoking action;

[0033] Calculating the spatial distance between the same joint in the actual smoking action and the preset smoking action;

[0034] The posture distance between the actual smoking action and the preset smoking action is calculated based on the number of matched joints and the spatial distance of the same joints.

[0035] Wherein, the network model training method further includes:

[0036] Obtain real-time monitoring images;

[0037] Inputting the real-time monitoring image into the preset network model to obtain a judgment result output by the preset network model;

[0038] Based on the judgment result, it is confirmed whether there is smoking behavior in the real-time monitoring image.

[0039] The present application also provides a terminal device, the terminal device comprising a memory and a processor, wherein the memory is coupled to the processor;

[0040] Among them, the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned network model training method.

[0041] The present application also provides a computer storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the above-mentioned network model training method.

[0042] The beneficial effects of the present application are: the terminal device establishes an initial network model; obtains a training data set about smoking behavior, and annotates the training data set to obtain annotation information about the training data set; inputs the training data set and its annotation information into the first-stage network model to generate a detection frame for each image in the training data set; inputs each image in the training data set and its detection frame into the second-stage network model to generate a judgment result about smoking behavior in each image in the training data set; trains the first-stage network model and the second-stage network model based on the judgment result to obtain a preset network model for detecting smoking behavior. In the above manner, the network model training method of the present application can obtain a network model for detecting smoking behavior, which solves the limitations of traditional methods and has good accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0044] Figure 1 It is a flowchart of an embodiment of a network model training method for detecting smoking behavior provided by the present application;

[0045] Figure 2 It is a schematic diagram of the framework of an embodiment of the overall network model provided by the present application;

[0046] Figure 3 It is a structural diagram of an embodiment of a terminal device provided by the present application;

[0047] Figure 4 It is a structural diagram of an embodiment of a terminal device provided by the present application;

[0048] Figure 5 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] In order to solve the above technical problems, the present application discloses a two-stage detection method for unsafe behaviors of construction site workers based on deep learning, which is applicable to the field of worker behavior detection in construction sites in major cities. The method trains the first-stage deep neural network with pre-processed unsafe behavior image data of construction site workers. The first-stage network is responsible for generating worker detection frames, and the detection frames are used as training data for the second-stage deep neural network. The second-stage network is responsible for detecting the behavior of workers. The method is based on a deep learning framework, has good autonomous learning ability, fault tolerance and generalization ability for various external factors that are difficult to quantify, and has good scalability and high prediction accuracy. In view of the gradual popularization of construction sites, taking workers smoking as an example, the construction sites gradually and comprehensively ban smoking and other issues, the method can realize the automatic monitoring and alarm of unsafe behaviors of construction site workers, reduce the management cost of construction sites, and improve the safety and reliability of construction.

[0051] Please refer to Figure 1 , Figure 1 It is a flow chart of an embodiment of a network model training method for detecting smoking behavior provided in the present application.

[0052] The network model training method of the present application is applied to a terminal device, wherein the terminal device of the present application can be a server, or a system in which a server and a mobile terminal cooperate with each other. Accordingly, the various parts of the terminal device, such as various units, subunits, modules, and submodules, can all be set in the server, or can be set in the server and the mobile terminal respectively.

[0053] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the network model training method of the embodiment of the present application can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0054] Specifically, if Figure 1 As shown, the network model training method of the embodiment of the present application specifically includes the following steps:

[0055] Step S11: Establishing an initial network model, wherein the initial network model includes a first-stage network model and a second-stage network model.

[0056] In the embodiment of the present application, the terminal device establishes Figure 2The initial network model shown, wherein the initial network model includes a one-stage network model and a two-stage network model, the one-stage network model is responsible for generating a worker detection frame, and the two-stage network model is responsible for detecting the worker's behavior.

[0057] Specifically, the first-stage network model can be set based on faster-rcnn, and the second-stage network model can be set based on RMPE: Regional Multi-Person Pose Estimation. In other embodiments, other feasible network models can also be used to build the initial network model.

[0058] Step S12: Obtain a training data set about smoking behavior, and annotate the training data set to obtain annotation information about the training data set.

[0059] In an embodiment of the present application, a terminal device collects images of workers at a construction site and establishes a training data set. The staff uses annotation tools to annotate the specific positions of the workers in the image in the training data set, that is, annotate the worker detection frame, annotate the basic skeleton points of the workers, and annotate the hand skeleton points in detail. Among them, the basic skeleton points include trunk skeleton points, hand skeleton points, leg skeleton points, foot skeleton points, etc.; the hand skeleton points include shoulder skeleton points, elbow skeleton points, wrist skeleton points, palm skeleton points and five finger skeleton points, a total of 9 hand skeleton points. Furthermore, the staff also needs to reward and annotate each image in the training data set to mark whether the behavior of the worker in the image is smoking behavior.

[0060] Specifically, when staff collect images for a training data set of workers' smoking behavior in a construction site scenario, they need to include images of smoking behavior of different scales, as well as non-smoking behavior, under different lighting conditions, to enrich the content of the training data set.

[0061] Step S13: Input the training data set and its annotation information into the first-stage network model to generate a detection box for each image in the training data set.

[0062] In the embodiments of the present application, Figure 2 As shown in FIG. 1 , the terminal device processes all images in the training data set through the convolution layer to obtain the feature map of each image. Then, the terminal device normalizes the image feature map, and simultaneously inputs the normalized image feature map and the manually annotated annotation information into the one-stage network model to obtain the predicted detection box generated by the one-stage network model on the image feature map.

[0063] Specifically, the terminal device generates several candidate regions on the image feature map through a candidate region network, and then selects the region of interest regarding the smoking behavior through a classifier to generate a predicted detection box.

[0064] Furthermore, the loss function for the one-stage network model setting in the embodiment of the present application is defined as follows:

[0065]

[0066] Among them, i is the predicted detection box, that is, the index of the anchor box generated by the one-stage network model. i is the confidence that a worker exists in the i-th predicted detection box, the true label Indicates that when there is a worker in the predicted detection box, the true label Indicates that there is no worker in the predicted detection box and it is only the background. is the true position coordinate of the i-th annotation prediction box, t i is the position coordinate of the i-th predicted detection box, including [t x , t y , t w , t h ]Four parameters. N cls is the adjustment parameter of the detection box classification loss function, N reg and λ are the adjustment functions of the detection box coordinate loss function. It can be seen that the loss function of the one-stage network model is specifically composed of two parts: the detection box classification loss function and the detection box coordinate loss function.

[0067] In the detection box regression, [t x , t y , t w , t h ]The four parameters are defined as follows:

[0068] t x =(xx a ) / w a t y =(yy a ) / h a t w =log(w / w a )t h =log(h / h a )

[0069] Among them, [x, y, w, h] are the center coordinates, width and height of the predicted detection box respectively, [x a ,y a , w a ,h a ] are the center coordinates, width and height of the labeled detection box, respectively, [t x , t y , t w , t h ]The four parameters are used to characterize the offset between the predicted detection box and the labeled detection box.

[0070] Before inputting the predicted detection box into the second-stage network model, the terminal device can also filter the predicted detection box generated by the first-stage network model using the NMS (non-maximum suppression) algorithm and then fuse it with the image feature map, and input the fusion result into the second-stage network model. The second-stage network model mainly re-selects the detection box generated by the first-stage network model through the posture analysis method to generate the final result.

[0071] The definition of the NMS algorithm in the embodiment of the present application is as follows:

[0072]

[0073]

[0074] Specifically, the terminal device inputs an image feature map and all its predicted detection boxes B = {b1…b N}, the confidence level S corresponding to each predicted detection box is S = {s1...s N}. Loop through the following steps until all predicted detection frames are traversed: Get the predicted detection frame with the highest confidence in B, and then calculate the intersection and union ratio of the predicted detection frame with the highest confidence and other predicted detection frames. When the intersection and union ratio is greater than the preset threshold N t When , the predicted detection box and its confidence are deleted; when the intersection-over-union ratio is less than or equal to the preset threshold N t , retain the predicted detection box and its confidence.

[0075] After traversing all the predicted detection frames based on the NMS algorithm, the retained predicted detection frames are fused with the image feature map, and the fusion results are input into the two-stage network model, where the two-stage network model mainly reselects the predicted detection frames generated by the first-stage network model through the posture analysis method to generate the final result.

[0076] Step S14: Input each image and its detection frame in the training data set into the two-stage network model to generate a judgment result on smoking behavior in each image in the training data set.

[0077] In the embodiment of the present application, the terminal device defines a smoking action with m joints in a two-stage network model. in, are the j-th position coordinates of the ith joint and its confidence level respectively.

[0078] The second-stage network model identifies the worker smoking action in the predicted detection frame according to the predicted detection frame generated in step S13. Assume that the smoking action P i The detection box is Then a soft matching function can be defined to calculate the smoking action P i and smoking action P j The number of joints that match between the two poses is:

[0079]

[0080] Among them, σ1 is the hyperparameter of the soft matching function.

[0081] Furthermore, define the smoking action P i and smoking action P j The formula for the spatial distance between the same joints in two postures is:

[0082]

[0083] Among them, σ2 is the hyperparameter of the spatial distance formula.

[0084] Finally, the attitude distance function is obtained:

[0085] d pose (P i , P j |Λ)=K Sim (P i , P j |σ1)+λH Sim (P i , P j |σ2)

[0086] It can be seen that the two-stage network model is responsible for generating the worker smoking action P j , and calculate the worker's smoking action P j With the marked smoking action P i When the distance is less than a preset threshold, it can be determined that the action is a smoking action.

[0087] Step S15: training the first-stage network model and the second-stage network model based on the judgment result to obtain a preset network model for detecting smoking behavior.

[0088] In the embodiment of the present application, the terminal device can train the one-stage network model and the two-stage network model based on the loss functions of the one-stage network model and the two-stage network model respectively, so as to Figure 2 The overall network model is trained, and finally a preset network model for detecting smoking behavior is obtained.

[0089] Furthermore, after the preset network model for detecting smoking behavior is trained, the terminal device can obtain a real-time monitoring image; input the real-time monitoring image into the preset network model to obtain a judgment result output by the preset network model; and confirm whether there is smoking behavior in the real-time monitoring image based on the judgment result.

[0090] In the embodiment of the present application, the terminal device establishes an initial network model; obtains a training data set about smoking behavior, and annotates the training data set to obtain annotation information about the training data set; inputs the training data set and its annotation information into the first-stage network model to generate a detection frame for each image in the training data set; inputs each image in the training data set and its detection frame into the second-stage network model to generate a judgment result about smoking behavior in each image in the training data set; and trains the first-stage network model and the second-stage network model based on the judgment result to obtain a preset network model for detecting smoking behavior. In the above manner, the network model training method of the present application can obtain a network model for detecting smoking behavior, which solves the limitations of traditional methods and has good accuracy.

[0091] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0092] In order to implement the network model training method of the above embodiment, this application also proposes a terminal device, please refer to Figure 3 , Figure 3 It is a structural diagram of an embodiment of a terminal device provided in this application.

[0093] The terminal device 300 of the embodiment of the present application includes a model building module 31, a data acquisition module 32, a behavior judgment module 33 and a network training module 34; wherein,

[0094] The model building module 31 is used to build an initial network model, wherein the initial network model includes a first-stage network model and a second-stage network model.

[0095] The data acquisition module 32 is used to acquire a training data set related to the smoking behavior, and to annotate the training data set to obtain annotation information about the training data set.

[0096] The behavior judgment module 33 is used to input the training data set and its annotation information into the first-stage network model to generate a detection frame for each image in the training data set; input each image in the training data set and its detection frame into the second-stage network model to generate a judgment result on smoking behavior in each image in the training data set.

[0097] The network training module 34 is used to train the first-stage network model and the second-stage network model based on the judgment result to obtain a preset network model for detecting smoking behavior.

[0098] In order to implement the network model training method of the above embodiment, this application also proposes another terminal device, please refer to Figure 4 , Figure 4 It is a structural diagram of another embodiment of the terminal device provided by the present application.

[0099] The terminal device 400 of the embodiment of the present application includes a memory 41 and a processor 42, wherein the memory 41 and the processor 42 are coupled.

[0100] The memory 41 is used to store program data, and the processor 42 is used to execute the program data to implement the network model training method described in the above embodiment.

[0101] In this embodiment, the processor 42 may also be referred to as a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip having the ability to process signals. The processor 42 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 42 may also be any conventional processor, etc.

[0102] The present application also provides a computer storage medium, such as Figure 5 As shown, the computer storage medium 500 is used to store program data 51. When the program data 51 is executed by the processor, it is used to implement the network model training method described in the above embodiment.

[0103] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the network model training method as described in the embodiment of the present application. The computer program product can be a software installation package.

[0104] The network model training method described in the above embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0105] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A network model training method for detecting smoking behavior, characterized in that: The network model training method comprises: Establishing an initial network model, wherein the initial network model includes a first-stage network model and a second-stage network model; Acquiring a training data set about the smoking behavior, and annotating the training data set to obtain annotated information about the training data set; Inputting the training data set and its annotation information into the first-stage network model to generate a detection frame for each image in the training data set; Inputting each image and its detection frame in the training data set into the two-stage network model to generate a judgment result on smoking behavior in each image in the training data set; Training the first-stage network model and the second-stage network model based on the judgment result to obtain a preset network model for detecting smoking behavior; Generating a judgment result on smoking behavior in each image in the training data set includes: Get pre-labeled real smoking actions; Inputting each image and its detection frame in the training data set into the two-stage network model to obtain a preset smoking action in the detection frame of each image; Calculating the posture distance between the actual smoking action and the preset smoking action; When the posture distance is less than a preset threshold, confirming that there is smoking behavior within the detection frame range of the image; The step of calculating the posture distance between the actual smoking action and the preset smoking action includes: Obtaining the number of joints that match the actual smoking action with the preset smoking action; Calculating the spatial distance between the same joint in the actual smoking action and the preset smoking action; Calculating the posture distance between the actual smoking action and the preset smoking action based on the matched joint numbers and the spatial distance of the same joints; Among them, the number of joints K Sim (P i ,P j |σ1) is expressed as the following formula, Wherein, the spatial distance H Sim (P i ,P j |σ2) is expressed as the following formula, Among them, the posture distance d pose (P i ,P j |Λ) is expressed as the following formula, d pose (P i ,P j |Λ)=K Sim (P i ,P j |σ1)+λH Sim (P i ,P j |σ2) Among them, σ1 is the hyperparameter of the soft matching function, It represents a real smoking action with m joints, are the qth position coordinates and their confidence of the ith joint, It represents a preset smoking action with m joints, are the qth position coordinates of the jth joint and their confidence, σ2 is the hyperparameter of the spatial distance formula, λ is the adjustment function, Detection box representing the real smoking action.

2. The network model training method according to claim 1, characterized in that: The obtaining of a training data set about the smoking behavior comprises: Under different external conditions, images of smoking behaviors in preset scenes are collected to obtain smoking behavior images; Under different external conditions, images of non-smoking behaviors in preset scenes are collected to obtain images of non-smoking behaviors; The training data set is established based on the smoking behavior images and the non-smoking behavior images.

3. The network model training method according to claim 1, characterized in that: The labeling of the training data set includes: Annotating a worker detection frame and a worker skeleton point in each image in the training data set; Among them, the worker skeleton points include basic skeleton points, the basic skeleton points include trunk skeleton points, hand skeleton points, leg skeleton points, and foot skeleton points, and the hand skeleton points include shoulder skeleton points, elbow skeleton points, wrist skeleton points, palm skeleton points and five finger skeleton points.

4. The network model training method according to claim 3, characterized in that: The step of inputting the training data set and its annotation information into the first-stage network model includes: Normalizing all images in the training data set; The normalized image and its annotation information are input into the one-stage network model.

5. The network model training method according to claim 1 or 4, characterized in that: The loss function of the one-stage network model includes a detection box classification loss function and a detection box coordinate loss function; Among them, the detection frame classification loss function is used to learn the relationship between the confidence of person classification in the predicted detection frame and the confidence of person classification in the labeled detection frame, and the detection frame coordinate loss function is used to learn the relationship between the coordinate position of the predicted detection frame and the coordinate position of the labeled detection frame.

6. The network model training method according to claim 1, characterized in that: Before inputting each image and its detection box in the training data set into the two-stage network model, the network model training method further includes: Using a preset algorithm to screen the detection frame in the image; The filtered detection frame is fused with the corresponding image.

7. The network model training method according to claim 1, characterized in that: The network model training method also includes: Obtain real-time monitoring images; Inputting the real-time monitoring image into the preset network model to obtain a judgment result output by the preset network model; Based on the judgment result, it is confirmed whether there is smoking behavior in the real-time monitoring image.

8. A terminal device, characterized in that: The terminal device includes a memory and a processor, wherein the memory is coupled to the processor; Wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the network model training method described in any one of claims 1-7.

9. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by the processor, it is used to implement the network model training method described in any one of claims 1-7.

Images