Personnel smoking behavior detection method based on space-time double-flow information fusion

Abstract

The invention belongs to the technical field of computer vision application, and particularly discloses a person smoking behavior detection method based on space-time double-flow information fusion, which comprises the following steps: S1, obtaining person smoking behavior data and constructing a data set; s2, generating an adjacent frame optical flow image; s3, constructing a spatio-temporal information fusion module; s4, constructing a smoking behavior detection module; s5, training the constructed smoking behavior detection network based on space-time double-flow information fusion by using the data set; and S6, performing smoking behavior detection by using the trained optimal network weight. The smoking behavior detection method has the beneficial effects that the time information and the space information of the video are combined, the smoking behavior of a person is detected according to the space information of a single image frame, and the characteristics of actions, forms and the like contained in the time information are also considered; the accuracy of smoking behavior detection can be remarkably improved, and false alarms and missing alarms are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision application, and specifically discloses a method for detecting smoking behavior of a person based on spatiotemporal dual-stream information fusion. Background Art

[0002] In production sites such as renewable energy power generation stations, safety management is a key factor in ensuring production efficiency and personnel safety.

[0003] Because these places usually involve dangerous factors such as high temperature, high pressure, flammable and explosive materials, any unsafe behavior, especially smoking, may cause serious safety accidents, resulting in casualties and property losses. Therefore, timely identification and control of people's smoking behavior is crucial to preventing accidents and ensuring production safety.

[0004] With the development of technology, video image smoking behavior detection technology based on computer vision and deep learning utilizes the powerful image processing capabilities of computer vision and the advanced feature extraction of deep learning to achieve rapid identification and positioning of smoking behavior.

[0005] However, the actual production environment is complex, and is affected by the camera shooting angle, image resolution, and the small target area of ​​cigarettes, which are difficult to distinguish from the monitoring images. The existing technology has the following shortcomings in detecting the smoking behavior of people in special environments such as power plants: the existing smoking behavior detection method only relies on a single frame image, and detects the position of cigarettes and hands through spatial information to determine whether there is smoking behavior. It does not consider the time information of the smoking behavior, and is easily misjudged due to the small cigarette target and hand position interference in the single frame image.

[0006] Based on the above problems, the present invention provides a method for detecting smoking behavior of people based on spatiotemporal dual-stream information fusion. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to provide a method for detecting smoking behavior of people based on spatiotemporal dual-stream information fusion, so as to solve the problem that smoking behavior detection in the background technology is prone to omissions and false alarms.

[0008] Technical solution: The present invention provides a method for detecting smoking behavior of people based on spatiotemporal dual-stream information fusion, comprising the following steps: S1: acquiring smoking behavior data of people and constructing a data set; S2: generating optical flow images of adjacent frames; S3: constructing a spatiotemporal information fusion module, processing optical flow images and visible light images, and obtaining spatiotemporal information fusion features; S4: constructing a smoking behavior detection module; S5: using the data set to train the constructed smoking behavior detection network based on spatiotemporal dual-stream information fusion; S6: using the optimal network weights after training to perform smoking behavior detection.

[0009] In the technical solution, the process of acquiring smoking behavior data and constructing a data set in S1 includes: (1) collecting surveillance videos of people smoking in the scene; (2) splitting the video data into visible light images frame by frame; (3) annotating the image data, using rectangular boxes to annotate the smoking behavior of people in the image, for training the smoking behavior detection model. The visible light images and the corresponding annotation files together constitute the smoking behavior detection data set.

[0010] In the technical solution, the adjacent frame optical flow image generation in S2 is based on two temporally continuous frames of image I T ,I T+1 As a pair of inputs, the optical flow estimation network based on deep learning is used to estimate the optical flow field between two frames of images and generate an optical flow image O. T+1 Optical flow is a method that uses the correspondence between the previous frame and the current frame in an image sequence to calculate the motion information of objects between adjacent frames. The optical flow image contains the time series information in the image sequence.

[0011] In the technical solution, the spatiotemporal information fusion module in S3 is composed of the following steps: S301: using the image feature extraction backbone network to extract T+1 frame image I T+1 And the corresponding optical flow image O T+1 The image features are used to obtain the temporal features and spatial features. S302: The attention structure can learn the importance weights of the feature matrix according to the importance; the attention is used to learn the temporal features to obtain the attention weight matrix, and the temporal features are multiplied by the weight matrix to obtain the temporal enhancement features. S303: The temporal enhancement features are added to the spatial features to obtain the spatiotemporal information fusion feature matrix.

[0012] In the technical solution, the motion information guidance feature is input into the smoking behavior detection model in S4, and the smoking behavior detection model is modified by the yolov5 detection model, and the yolov5 network is divided into different parts. The backbone part is replaced by the motion enhancement feature output by the motion information guidance module in step S3.

[0013] In the technical solution, the spatiotemporal information fusion feature is input into the smoking behavior detection module in S4, and the smoking behavior detection module is obtained by modifying the yolov5 target detection network. The yolov5 network consists of 4 parts: input end, backbone network, bottleneck module and prediction module. In order to make the yolov5 network adapt to the spatiotemporal information fusion feature matrix as input, the input end and the backbone network are deleted.

[0014] In the technical solution, the constructed smoking behavior detection network is trained using a data set with smoking behavior annotation information in S5. A loss function is used to measure the difference between the network prediction result and the actual annotation. The closer the prediction result is to the actual annotation, the smaller the loss function value. The training loss function of the smoking behavior detection network is set to the sum of position loss, confidence loss, and classification loss.

[0015] In this technical solution, the optimal network weight in S6 is the network parameter when the loss function is minimized in step S5. Two temporally consecutive monitoring images are used as input, and the smoking behavior detection network is used to load the optimal network weight to perform smoking behavior detection on the input images.

[0016] Compared with the prior art, the beneficial effect of the present invention's method for detecting smoking behavior of people based on spatiotemporal dual-stream information fusion is that: the present invention will combine the time information and spatial information of the video, not only detect the smoking behavior of people based on the spatial information of a single image frame, but also consider the action, form and other features contained in the time information. Through this smoking behavior detection method that utilizes the spatiotemporal dual-stream information fusion of the previous and next frame images, the accuracy of smoking behavior detection can be significantly improved, and false alarms and missed alarms can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is a flow chart of the present invention;

[0019] Figure 2 is a flowchart of the smoking behavior detection network of the present invention;

[0020] Figure 3 It is a flow chart of the spatiotemporal information fusion module of the present invention. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0022] In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "top", "bottom", "one side", "the other side", "front", "back", "middle part", "inside", "top end", "bottom end", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention; the terms "first", "second", and "third" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance; in addition, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] Example

[0024] The present invention provides a method for detecting smoking behavior of a person based on spatiotemporal dual-stream information fusion, the flow chart of which is as follows: Figure 1 As shown, the following steps are included:

[0025] S1: Data acquisition of smoking behavior of personnel and construction of data set;

[0026] S2: Generation of adjacent frame optical flow images;

[0027] S3: Build a spatiotemporal information fusion module to process the optical flow image and visible light image to obtain spatiotemporal information fusion features;

[0028] S4: Building a smoking behavior detection module;

[0029] S5: Use the dataset to construct Figure 2 The smoking behavior detection network shown is trained based on spatiotemporal dual-stream information fusion;

[0030] S6: Use the optimal network weights after training to detect smoking behavior.

[0031] In addition, preferably, the process of acquiring smoking behavior data and constructing a data set in S1 includes:

[0032] (1) Collect surveillance video data;

[0033] (2) Splitting the video data into visible light images frame by frame;

[0034] (3) Annotate the image data and use a rectangular frame to annotate the smoking behavior of the person in the image.

[0035] Specifically, the head and upper chest of the smoker in the image are surrounded by a rectangular frame, so that the rectangular frame contains the image of the person's mouth, cigarette, and hand, which is used to train the smoking behavior detection model. The visible light image and the corresponding annotation file together constitute the smoking behavior detection dataset.

[0036] In addition, preferably, the adjacent frame optical flow images in S2 are generated by taking two temporally continuous frames of images I T ,I T+1 As a pair of inputs, the deep learning-based optical flow estimation network RAFT is used. The RAFT optical flow estimation network extracts the features of each pixel, constructs a multi-scale correlation volume, and iteratively updates the optical flow field through a cyclic unit. Estimate the optical flow field between two frames of images and generate an optical flow image O T+1 Optical flow is a method that uses the correspondence between the previous frame and the current frame in an image sequence to calculate the motion information of objects between adjacent frames. The optical flow image contains the motion information in the image sequence.

[0037] In addition, if Figure 3 As shown, the motion information guidance module in S3 is composed of the following steps:

[0038] S301: Use the MobileNetV3 image feature extraction backbone network to extract T+1 frame images I through continuous convolution and downsampling. T+1 And the corresponding optical flow image O T+1 The image features are used to obtain the temporal features and spatial features;

[0039] S302: The attention structure can learn the spatial importance weights of the feature matrix according to different importance.

[0040] Use attention to learn the temporal features. Preferably, the attention operation uses the temporal features as input, performs maximum pooling and average pooling operations, and further performs activation function processing after convolution operation to obtain the attention weight matrix. The temporal features and the weight matrix perform matrix dot multiplication operation to obtain the temporal enhancement features.

[0041] S303: Add the time enhancement feature and the space feature to obtain a time-space information fusion feature matrix.

[0042] In addition, preferably, in S4, the spatiotemporal information fusion feature is input into the smoking behavior detection module, and the smoking behavior detection module is modified from the yolov5 target detection network. The yolov5 network consists of 4 parts: an input end, a backbone network, a bottleneck module and a prediction module.

[0043] In order to make the yolov5 network adapt to the spatiotemporal information fusion feature matrix as input, the input end and the backbone network are deleted.

[0044] In addition, the smoking behavior detection network constructed in S5 is preferably trained using the data set with annotation information. A loss function is used to measure the difference between the network prediction result and the actual annotation. The closer the prediction result is to the actual annotation, the smaller the loss function value. The training loss function of the smoking behavior detection network is set to the sum of position loss, confidence loss, and classification loss.

[0045] The technology of the present invention utilizes the temporal information and spatial information of the surveillance video to fuse, not only considering the spatial information of cigarettes and hands in the image, but also utilizing the temporal information such as hand movements contained in continuous video frames, thereby improving the accuracy of detecting people's smoking behavior and achieving efficient and accurate recognition of smoking behavior.

[0046] It should be noted that, in this article, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0047] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for detecting smoking behavior of people based on spatiotemporal dual-stream information fusion, characterized by: The following steps are included: S1: Data acquisition of smoking behavior of personnel and construction of data set; S2: Generation of adjacent frame optical flow images; S3: Build a spatiotemporal information fusion module to process the optical flow image and visible light image to obtain spatiotemporal information fusion features; S4: Building a smoking behavior detection module; S5: Use the dataset to train the smoking behavior detection network based on spatiotemporal dual-stream information fusion; S6: Use the optimal network weights after training to detect smoking behavior.

2. According to claim 1, a method for detecting smoking behavior of a person based on spatiotemporal dual-stream information fusion, characterized in that: The process of acquiring the smoking behavior data and constructing the data set in S1 includes: (1) Collect surveillance videos of people smoking in the scene; (2) Splitting the video data into visible light images frame by frame; (3) Annotate the image data and use rectangular boxes to annotate the smoking behavior of people in the image for training the smoking behavior detection model; Among them, visible light images and corresponding annotation files together constitute the smoking behavior detection dataset.

3. The method for detecting smoking behavior of a person based on spatiotemporal dual-stream information fusion according to claim 1 is characterized by: The adjacent frame optical flow images in S2 are generated by taking two temporally consecutive frames of images I T ,I T+1 As a pair of inputs, the optical flow estimation network based on deep learning is used to estimate the optical flow field between two frames of images and generate an optical flow image O. T+1 ; Optical flow is a method of calculating the motion information of objects between adjacent frames by using the correspondence between the previous frame and the current frame in an image sequence. The optical flow image contains the time series information in the image sequence.

4. The method for detecting smoking behavior of a person based on spatiotemporal dual-stream information fusion according to claim 1, characterized in that: The spatiotemporal information fusion module in S3 is composed of the following steps: S301, using the image feature extraction backbone network to extract T+1 frame image I T+1 And the corresponding optical flow image O T+1 The image features are used to obtain the temporal features and spatial features; S302, the attention structure can learn the importance weights of the feature matrix according to different importance, use attention to learn the time features, obtain the attention weight matrix, perform matrix dot multiplication operation on the time features and the weight matrix, and obtain the time enhancement feature; S303: Add the time enhancement feature and the space feature to obtain a time-space information fusion feature matrix.

5. The method for detecting smoking behavior of a person based on spatiotemporal dual-stream information fusion according to claim 1, characterized in that: In the S4, the spatiotemporal information fusion feature is input into the smoking behavior detection module, and the smoking behavior detection module is obtained by modifying the yolov5 target detection network. The yolov5 network consists of 4 parts: an input end, a backbone network, a bottleneck module and a prediction module; Among them, in order to make the yolov5 network adapt to the spatiotemporal information fusion feature matrix as input, the input end and the backbone network part are deleted.

6. The method for detecting smoking behavior of a person based on spatiotemporal dual-stream information fusion according to claim 1, characterized in that: In S5, a data set with smoking behavior annotation information is used to train the constructed smoking behavior detection network; Among them, the loss function is used to measure the difference between the network prediction result and the actual annotation. The closer the prediction result is to the actual annotation, the smaller the loss function value is. The training loss function of the smoking behavior detection network is set to the sum of position loss, confidence loss, and classification loss.

7. The method for detecting smoking behavior of a person based on spatiotemporal dual-stream information fusion according to claim 1, characterized in that: The optimal network weight in S6 is the network parameter obtained by training in step S5 to minimize the loss function; Among them, two temporally consecutive frames of monitoring images are used as input, and the smoking behavior detection network is used to load the optimal network weights to perform smoking behavior detection on the input images.

Images