Personnel operation behavior identification method based on lightweight YOLOv7

Abstract

The invention belongs to the technical field of new-generation information, and particularly relates to a lightweight YOLOv7-based personnel operation behavior recognition method, which comprises the following steps of S1, collecting, making or selecting a personnel operation behavior recognition data set; s2, a lightweight YOLOv7 network comprising a DSCBS module, an attention mechanism module and a DSCELAN module is constructed; s3, deploying the YOLOv7 network to an embedded device; and S4, the embedded device is used for personnel operation behavior identification. According to the application, a YOLOv7 network which can efficiently run on the embedded equipment is designed to identify the behaviors of the industrial equipment operators, so that the behaviors of the industrial equipment operators are standardized to reduce the occurrence frequency of industrial accidents caused by dangerous operation behaviors.

Description

Technical Field

[0001] The present invention belongs to the field of new generation information technology, and specifically relates to a personnel operation behavior recognition method based on lightweight YOLOv7. Background Art

[0002] As my country's industrialization gradually advances, industrial production tends to be automated and intelligent. Industrial safety accidents caused by improper behavior of industrial equipment operators occur from time to time. Ordinary recorders can only record the behavior of industrial equipment operators, but cannot play the role of reminding standardized personnel to operate industrial equipment safely.

[0003] In the prior art, there is a Chinese invention patent application with patent number CN202310966210.5 and titled "A detection method for network carrier inspection vehicles based on adaptively improved YOLOv7", which discloses a detection method for network carrier inspection vehicles based on adaptively improved YOLOv7, including the following steps: 1) using the construction of an intelligent platform for human-computer interaction, the camera input end is connected to the development board output end, the cloud server is connected to the mobile phone APP, the cloud server signal is detected and the data is recorded; 2) the operator uses the face to unlock the inspection vehicle, and the system collects and records the data; 3) the inspection vehicle starts the follow-up mode, and its distance acquisition device is set in front of the vehicle body for sensing the distance and direction; 4) the improved YOLOv7 model is trained, and the model is used to identify illegal operations and products, and the output is displayed through a display and a mobile phone APP; 5) through the recognition results, the inferior products are autonomously classified, and the packaging point is autonomously navigated, and the packaging work is performed by the specialist. After the patrol car completes the assigned task, it automatically cruises to the designated parking point. However, the above patent is not based on lightweight YOLOv7, which will result in excessive model parameters and computing consumption, and cannot run stably on embedded devices. Summary of the invention

[0004] In view of the above problems existing in the prior art, the present invention proposes a method for identifying personnel operation behavior based on lightweight YOLOv7.

[0005] In order to achieve the above effects, the technical solution adopted by the present invention is as follows:

[0006] A method for identifying human operation behavior based on lightweight YOLOv7 includes the following steps:

[0007] S1. Collect and create or select a data set for identifying human operation behaviors;

[0008] S2. Build a lightweight YOLOv7 network including DSCBS module, attention mechanism module, and DSC_ELAN module;

[0009] S3, deploy the YOLOv7 network to the embedded device;

[0010] S4. Using the embedded device to identify human operation behavior.

[0011] Furthermore, the personnel operation behavior recognition data set collected, produced or selected in step S1 includes common personnel operation behaviors, and the common personnel operation behaviors include normal operation of industrial equipment operators, sideways viewing, drinking water, smoking, operating central control, looking at mobile phones, picking up things sideways, tidying up and answering the phone.

[0012] Furthermore, in step S2, the DSCBS module will replace all CBS modules in the original YOLOv7 network except the first 3×3 convolutional layer; the attention mechanism module will replace the third CBS module of the original YOLOv7 backbone neural network; the DSC_ELAN module will replace all ELAN modules in the original YOLOv7 module, and finally a lightweight YOLOv7 network is obtained.

[0013] Furthermore, in step S2, the DSCBS module includes a depthwise separable convolution, a normalization layer, and a Silu activation function layer, and each layer is connected in sequence.

[0014] Furthermore, the formula of the Silu activation function is:

[0015]

[0016] Where x is the input content and e is a natural constant.

[0017] Furthermore, in step S2, the attention mechanism module in lightweight YOLOv7 has two branches, namely, a spatial attention mechanism module and a channel attention mechanism module.

[0018] Furthermore, in the spatial attention mechanism module, the input feature map tensor matrix C in ∈R H,W,c Where H is the height of the feature map, W is the width of the feature map, c is the number of channels of the feature map, and R is the real number domain; perform three 3×3 convolution operations on the input feature map to obtain three output feature maps, and then deform the dimensions of these three feature maps. The three output feature maps are denoted as Q s , K s 、V s , where Q s ∈R H×W,c , K s ∈R H×W,c , V s ∈R H×W,c , and finally the attention mechanism is used to calculate the formula:

[0019]

[0020] Where Q(Query) represents the query vector in the attention mechanism, K(Key) represents the key vector in the attention mechanism, and V(Value) represents the numerical vector in the attention mechanism, corresponding to the above feature graph Q s , K s , V s ;d k is the feature map matrix product QK T The standard deviation of

[0021] The formula for the softmax function is:

[0022]

[0023] Where xi represents the i-th input variable, and n represents the n input variables.

[0024] Furthermore, in the channel attention mechanism module, the input feature map tensor matrix C in ∈R H,W,c Where H is the height of the feature map, W is the width of the feature map, and c is the number of channels of the feature map. The input feature map is deformed in three dimensions respectively, and the three output feature maps Q c , K c 、V c , where Q c ∈R c,H×W , K c ∈R c,H×W , V c ∈R c,H×W , and finally the attention mechanism is used to calculate the formula:

[0025]

[0026] Where Q(Query) represents the query vector in the attention mechanism, K(Key) represents the key vector in the attention mechanism, and V(Value) represents the numerical vector in the attention mechanism, corresponding to the above feature graph Q s , K s , V s ;d k is the feature map matrix product QK T The standard deviation of

[0027] The formula for the softmax function is:

[0028]

[0029] Where xi represents the i-th input variable, and n represents the n input variables.

[0030] Furthermore, the DSC_ELAN module in the lightweight YOLOv7 has two branches, one of which passes through a 1×1 convolutional layer; the other branch consists of a 1×1 convolutional layer, and four DSCBS modules are connected in sequence. Finally, the output results of the two branches are spliced ​​together and then pass through a 1×1 convolutional layer.

[0031] Furthermore, in step S3, the embedded device supports the PyTorch deep learning framework.

[0032] Further, step S3 is specifically: using the PyQt framework developer to operate the behavior recognition system, deploying the trained lightweight YOLOv7 network into the software system, the system includes an image processing function module, a target detection function module, a behavior classification function module and an abnormal feedback function module, and the processing steps are as follows:

[0033] S31. In the image processing function module, after starting the lightweight YOLOv7-based industrial operator behavior recognition system, the user can use offline pictures, video files or online cameras to obtain video images in real time, display the images on the program user interface, process the images into the data dimension (640, 640, 3) RGB format required by lightweight YOLOv7, and pass them to the behavior recognition module for behavior recognition processing;

[0034] S32, the acquired image is positioned by the target detection function module. Once it is determined to be the position of the industrial equipment operator, the behavior recognition is completed, and the behavior category data (0-8) and the target position tuple data (x, y, w, h) and the confidence score are output. Then the data is passed to the behavior classification function module to classify the industrial equipment operator behavior;

[0035] S33, the behavior classification function module performs post-processing on the obtained behavior dictionary data, extracts the label with the maximum confidence score, divides the behavior into two categories: normal operation and abnormal operation, and transmits the data to the abnormal behavior feedback function module for risk assessment;

[0036] S34. The abnormal behavior feedback function module receives the output of the industrial equipment operator behavior classification function module, performs risk assessment according to the corresponding preset rules, triggers the corresponding warning feedback mechanism, prominently marks it on the user interface, records the abnormal behavior as a log, or sends the data to a remote server for further processing.

[0037] The technical effects achieved by the present invention are:

[0038] This application designs a YOLOv7 network that can run efficiently on embedded devices to identify the behavior of industrial equipment operators, so as to regulate the behavior of industrial equipment operators and reduce the frequency of industrial accidents caused by dangerous operating behaviors.

[0039] This application designs a lightweight YOLOv7 network that can run efficiently on embedded devices. The improved network has fewer parameters and computational consumption than the original YOLOv7 network, and can meet application scenarios that require real-time performance.

[0040] This application adopts the deep separable convolutional neural network technology and designs the efficient network modules of DSCBS and DSC_ELAN. Compared with the traditional YOLOv7 module, it has fewer parameters and computing consumption while ensuring the operating performance.

[0041] This application is based on lightweight YOLOv7. By using some lightweight network structure modules to replace the original YOLOv7 modules, the YOLOv7 model parameters and computing consumption can be reduced, so that it can run stably and efficiently on embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a method for identifying human operation behavior based on lightweight YOLOv7 is provided for an implementation example of the present invention.

[0043] Figure 2 This is the structure diagram of the DSCBS module in lightweight YOLOv7.

[0044] Figure 3 This is the structural diagram of the attention mechanism module in lightweight YOLOv7.

[0045] Figure 4 This is the structure diagram of the DSC_ELAN module in lightweight YOLOv7.

[0046] Figure 5 This is a diagram of the lightweight YOLOv7 network structure.

[0047] Figure 6 Flowchart of the personnel operation behavior identification system. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0050] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0051] In the description of the present application, it should be noted that the terms "upper", "vertical", "inner", "outer", etc. indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the product of the application is usually placed when in use, or the orientation or position relationship commonly understood by those skilled in the art. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0052] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0053] Example 1

[0054] like Figure 1 As shown, a method for identifying human operation behavior based on lightweight YOLOv7 includes the following steps:

[0055] S1. Collect and create or select a data set for identifying human operation behaviors;

[0056] S2. Build a lightweight YOLOv7 network including DSCBS module, attention mechanism module, and DSC_ELAN module;

[0057] Among them, DSCBS stands for Depthwise Separable Convolution-BatchNormalize-Silu, depth-separable convolution layer-normalization layer-Silu activation function, and DSC_ELAN stands for Depthwise Separable ConvolutionEffective Long-Range Aggregation Network, depth-separable efficient long-range aggregation network.

[0058] S3. Deploy the YOLOv7 network to an embedded device. The embedded device here can be vehicle-mounted or portable.

[0059] S4. Using the embedded device to identify human operation behavior.

[0060] This application designs a YOLOv7 network that can run efficiently on embedded devices to identify the behavior of industrial equipment operators, so as to regulate the behavior of industrial equipment operators and reduce the frequency of industrial accidents caused by dangerous operating behaviors. This application designs a lightweight YOLOv7 network, which can run efficiently on embedded devices. The improved network has fewer parameters and computational consumption than the original YOLOv7 network, which can meet the application scenarios with real-time requirements. This application adopts deep separable convolutional neural network technology and designs efficient network modules of DSCBS and DSC_ELAN. Compared with traditional YOLOv7 modules, it has fewer parameters and computational consumption while ensuring operating performance. This application is based on lightweight YOLOv7, and uses some lightweight network structure modules to replace the original YOLOv7 modules to reduce the YOLOv7 model parameter amount and computational consumption, so that it can run stably and efficiently on embedded devices.

[0061] Example 2

[0062] like Figure 1 As shown, a method for identifying human operation behavior based on lightweight YOLOv7 includes the following steps:

[0063] S1. Collect and create or select a data set for identifying human operation behaviors;

[0064] S2. Build a lightweight YOLOv7 network including DSCBS module, attention mechanism module, and DSC_ELAN module;

[0065] Among them, DSCBS stands for Depthwise Separable Convolution-BatchNormalize-Silu, depth-separable convolution layer-normalization layer-Silu activation function, and DSC_ELAN stands for Depthwise Separable ConvolutionEffective Long-Range Aggregation Network, depth-separable efficient long-range aggregation network.

[0066] S3. Deploy the YOLOv7 network to an embedded device. The embedded device here can be vehicle-mounted or portable.

[0067] S4. Using the embedded device to identify human operation behavior.

[0068] The personnel operation behavior recognition data set collected, produced or selected in step S1 includes common personnel operation behaviors, including normal operation, side-viewing, drinking, smoking, operating the central control, looking at the phone, leaning sideways to take things, tidying up and answering the phone by industrial equipment operators. The data can be collected in the form of pictures, videos or existing public data sets.

[0069] In step S2, the DSCBS module will replace all CBS modules in the original YOLOv7 network except the first 3×3 convolutional layer; the attention mechanism module will replace the third CBS module of the original YOLOv7 backbone neural network; the DSC_ELAN module will replace all ELAN modules in the original YOLOv7 module, and finally a lightweight YOLOv7 network is obtained.

[0070] The YOLOv7 network consists of a backbone neural network, YOLO-neck, and YOLO-head. The backbone neural network is responsible for downsampling the image, and is mainly composed of CBS (Convolution-BatchNormalize-Silu, convolution layer-normalization layer-Silu activation function) modules, ELAN (Effective Long-Range Aggregation Network) and MP (Maxpool) modules. YOLO-neck is responsible for feature fusion of the image, and mainly includes SPPCSPC (Spatial Pyramid Pooling Cross Stage Partial Channel) modules, UpSample (upsample) modules, ELAN modules, CBS modules, and MP modules. YOLO-head is responsible for outputting feature maps and calculating loss functions. Among them, the CBS module and the ELAN module are responsible for feature extraction and downsampling of the image, the MP module is a maximum pooling module, which is responsible for downsampling, the UpSample module is responsible for upsampling, and the SPPCSPC module is responsible for feature fusion to improve the receptive field of the model.

[0071] In the lightweight YOLOv7 network designed by the present invention, the CBS module is replaced by the DSCBS module, and the ELAN module is replaced by the DSC_ELAN module. In this way, the parameter amount and computational consumption of the YOLOv7 network are reduced, thereby achieving the purpose of improving operation efficiency.

[0072] The DSCBS module in step S2 includes a depthwise separable convolution, a normalization layer, and a Silu activation function layer, and each layer is connected in sequence. The formula of the Silu activation function is:

[0073]

[0074] Where x is the input content and e is a natural constant.

[0075] In step S2, the attention mechanism module in the lightweight YOLOv7 has two branches, namely the spatial attention mechanism module and the channel attention mechanism module. In the spatial attention mechanism module, the input feature map tensor matrix C in ∈R H,W,c Where H is the height of the feature map, W is the width of the feature map, c is the number of channels of the feature map, and R is the real number domain; perform three 3×3 convolution operations on the input feature map to obtain three output feature maps, and then deform the dimensions of these three feature maps. The three output feature maps are denoted as Q s , K s 、V s, where Q s ∈R H×W,c , K s ∈R H×W,c , V s ∈R H×W,c , and finally the attention mechanism is used to calculate the formula:

[0076]

[0077] Where Q(Query) represents the query vector in the attention mechanism, K(Key) represents the key vector in the attention mechanism, and V(Value) represents the numerical vector in the attention mechanism, corresponding to the above feature graph Q s , K s , V s ;d k is the feature map matrix product QK T The standard deviation of

[0078] The formula for the softmax function is:

[0079]

[0080] Where xi represents the i-th input variable, and n represents the n input variables.

[0081] In the channel attention mechanism module, the input feature map tensor matrix C in ∈R H,W,c Where H is the height of the feature map, W is the width of the feature map, and c is the number of channels of the feature map. The input feature map is deformed in three dimensions respectively, and the three output feature maps Q c , K c 、V c , where Q c ∈R c,H×W , K c ∈R c,H×W , V c ∈R c,H×W , and finally the attention mechanism is used to calculate the formula:

[0082]

[0083] Where Q(Query) represents the query vector in the attention mechanism, K(Key) represents the key vector in the attention mechanism, and V(Value) represents the numerical vector in the attention mechanism, corresponding to the above feature graph Q s , K s , V s ;d k is the feature map matrix product QK T The standard deviation of

[0084] The formula for the softmax function is:

[0085]

[0086] Where xi represents the i-th input variable, and n represents the n input variables.

[0087] The DSC_ELAN module in lightweight YOLOv7 has two branches, one of which passes through a 1×1 convolutional layer; the other branch consists of a 1×1 convolutional layer, and 4 DSCBS modules are connected in sequence. Finally, the output results of the two branches are spliced ​​together and then passed through a 1×1 convolutional layer.

[0088] The DSCBS module is used to replace the traditional CBS module. While retaining the receptive field of the original module, the module parameters and computational consumption are reduced, thereby improving the operating efficiency of the entire YOLOv7 network.

[0089] In step S3, the embedded device supports the PyTorch deep learning framework.

[0090] Step S3 is as follows: Taking the Raspberry Pi 4B as an example, this embedded device supports the pyTorch deep learning framework and can run the above-mentioned lightweight YOLOv7 network. The camera uses the Rasberry Pi NoIR Camera v2 supported by the Raspberry Pi, and the sound card and network card select Jetson Nano USB.

[0091] Using the PyQt framework, a human operation behavior recognition system is developed, and the trained lightweight YOLOv7 network is deployed to the software system. PyQt is an open source Python graphical interface library for the industrial graphical interface development program QT. PyQT supports the visual design tool QT Designer, provides a rich set of controls and a powerful layout manager to meet various interface design requirements. The PyQT library uses a unique signal and slot mechanism to achieve flexible communication between objects, enhance the interactivity of the application, and improve development efficiency. In addition, the application graphical interface developed based on PyQT has cross-platform compatibility and can run on multiple operating systems. Therefore, the lightweight YOLOv7 algorithm deployment and interface development program based on PyQT can be easily deployed on embedded devices.

[0092] The software system includes image processing function module, target detection function module, behavior classification function module and abnormal feedback function module. Its flow chart is as follows: Figure 6 As shown, the processing steps are as follows:

[0093] 1. In the image processing function module, after starting the lightweight YOLOv7-based industrial operator behavior recognition system, users can use offline pictures, video files or online cameras to obtain video images in real time, display the images on the program user interface, process the images into the data dimension (640,640,3) RGB format required by lightweight YOLOv7, and pass them to the behavior recognition module for behavior recognition processing;

[0094] 2. The acquired image will be passed through the industrial equipment operator target detection function module to locate the industrial equipment operator. Once it is determined to be the position of the industrial equipment operator, the behavior recognition is completed, and the behavior category data (0-8) and the target location tuple data (x, y, w, h) and the confidence score are output. The data is then passed to the behavior classification function module to classify the industrial equipment operator behavior;

[0095] 3. The behavior classification function module post-processes the obtained behavior dictionary data, extracts the label with the maximum confidence score, and divides the behavior into two categories: normal operation and abnormal operation (including side-gaze, drinking water, smoking, operating the central control, looking at the phone, taking things sideways, tidying up and answering the phone), and transmits the data to the abnormal behavior feedback function module for risk assessment;

[0096] 4. The abnormal behavior feedback function module receives the output of the industrial equipment operator behavior classification function module, performs risk assessment according to the corresponding preset rules, triggers the corresponding warning feedback mechanism, prominently marks it on the user interface, records the abnormal behavior as a log, or sends the data to a remote server for further processing (such as recording, analysis, etc.).

[0097] Example 3

[0098] like Figure 1 As shown, a method for identifying the behavior of industrial equipment operators based on lightweight YOLOv7 includes the following steps:

[0099] S1. Collect, create or select a data set for identifying human operation behaviors.

[0100] S2. Build a lightweight YOLOv7 network including the DSCBS module, the attention mechanism module, and the DSC_ELAN module.

[0101] S3. Deploy the YOLOv7 network to the vehicle embedded device.

[0102] S4. Using the vehicle-mounted embedded device to identify personnel operation behaviors.

[0103] The personnel operation behavior recognition data set collected, produced or selected in step S1 must include common personnel operation behaviors of industrial equipment operators, such as normal operation, side-viewing, drinking water, smoking, operating central control, looking at mobile phones, picking up things sideways, tidying up, and answering the phone.

[0104] In step S2, the DSCBS module will replace all CBS modules in the original YOLOv7 network except the first 3×3 convolutional layer; the attention mechanism module will replace the third CBS module of the original YOLOv7 backbone neural network; the DSC_ELAN module will replace all ELAN modules in the original YOLOv7 module. Finally, a lightweight YOLOv7 network is obtained, as shown in Figure 5 shown.

[0105] In step S2, the DSCBS module includes a depthwise separable convolution (DW, PW layer), a normalization layer (BN), and a Silu activation function layer connected in sequence, such as Figure 2 As shown. The formula of Silu activation function is:

[0106]

[0107] In step S2, the attention mechanism module in lightweight YOLOv7 has two branches, namely the spatial attention mechanism module and the channel attention mechanism module, as shown in Figure 3 shown.

[0108] In the above spatial attention mechanism module, the input feature map tensor matrix C in ∈R H,W,c Where H is the height of the feature map, W is the width of the feature map, and c is the number of channels of the feature map. Perform three 3×3 convolution operations on the input feature map to obtain three output feature maps, and then deform the dimensions of these three feature maps, which are recorded as Q s ∈R H×W,c , K s ∈R H×W,c , V s ∈R H×W,c . Finally, the attention mechanism is used to calculate the formula:

[0109] Where Q, K, V correspond to the above feature graph Q s , K s , V s ;d k is the feature map matrix product QK T The standard deviation of; the formula of the softmax function is:

[0110]

[0111] In the above-mentioned temporal attention mechanism module, the input feature map tensor matrix C in ∈R H,W,c Where H is the height of the feature map, W is the width of the feature map, and c is the number of channels of the feature map. The input feature map is deformed in three dimensions and recorded as Q c ∈R c,H×W , K c ∈R c,H×W , V c ∈R c,H×W . Finally, the attention mechanism is used to calculate the formula:

[0112]

[0113] In step S2, the DSC_ELAN module has two branches, one of which passes through a 1×1 convolutional layer; the other branch consists of a 1×1 convolutional layer and four DSCBS modules connected in sequence. Finally, the output results of the two branches are spliced ​​together and then passed through a 1×1 convolutional layer, as shown in Figure 4 shown.

[0114] Furthermore, in step S3, the embedded device must support the PyTorch deep learning framework.

[0115] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0118] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0119] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for identifying human operation behaviors based on lightweight YOLOv7. It is characterized in that The steps include: S1. Collect and create or select a data set for identifying human operation behaviors; S2. Build a lightweight YOLOv7 network including DSCBS module, attention mechanism module, and DSC_ELAN module; S3, deploy the YOLOv7 network to the embedded device; S4. Using the embedded device to identify human operation behavior.

2. According to claim 1, a method for identifying human operation behavior based on lightweight YOLOv7, It is characterized in that The personnel operation behavior recognition data set collected, produced or selected in step S1 includes common personnel operation behaviors, which include normal operation of industrial equipment operators, side-looking, drinking water, smoking, operating central control, looking at mobile phones, picking up things sideways, tidying up and answering the phone.

3. According to the method for identifying human operation behavior based on lightweight YOLOv7 in claim 1, It is characterized in that In step S2, the DSCBS module will replace all CBS modules in the original YOLOv7 network except the first 3×3 convolutional layer; the attention mechanism module will replace the third CBS module of the original YOLOv7 backbone neural network; the DSC_ELAN module will replace all ELAN modules in the original YOLOv7 module, and finally a lightweight YOLOv7 network is obtained.

4. According to the method for identifying human operation behavior based on lightweight YOLOv7 in claim 3, It is characterized in that The DSCBS module in step S2 includes a depthwise separable convolution, a normalization layer, and a Silu activation function layer, and each layer is connected in sequence.

5. According to claim 4, a method for identifying human operation behavior based on lightweight YOLOv7, It is characterized in that The formula of Silu activation function is: Where x is the input content and e is a natural constant.

6. According to claim 3, a method for identifying human operation behavior based on lightweight YOLOv7, It is characterized in that In step S2, the attention mechanism module in lightweight YOLOv7 has two branches, namely the spatial attention mechanism module and the channel attention mechanism module.

7. According to claim 6, a method for identifying human operation behavior based on lightweight YOLOv7, It is characterized in that In the spatial attention mechanism module, the input feature map tensor matrix C in ∈R H,W,c Where H is the height of the feature map, W is the width of the feature map, c is the number of channels of the feature map, and R is the real number domain; perform three 3×3 convolution operations on the input feature map to obtain three output feature maps, and then deform the dimensions of these three feature maps. The three output feature maps are denoted as Q s , K s 、V s , where Q s ∈R H×W,c , K s ∈R H×W,c , V s ∈R H×W,c , and finally the attention mechanism is used to calculate the formula: Where Q represents the query vector in the attention mechanism, K represents the key vector in the attention mechanism, and V represents the numerical vector in the attention mechanism, corresponding to the above feature graph Q s , K s , V s ;d k is the feature map matrix product QK T The standard deviation of The formula for the softmax function is: Where xi represents the i-th input variable, and n represents the n input variables.

8. According to claim 6, a method for identifying human operation behavior based on lightweight YOLOv7, It is characterized in that In the channel attention mechanism module, the input feature map tensor matrix C in ∈R H,W,c Where H is the height of the feature map, W is the width of the feature map, and c is the number of channels of the feature map. The input feature map is deformed in three dimensions respectively, and the three output feature maps Q c , K c 、V c , where Q c ∈R c,H×W , K c ∈R c,H×W , V c ∈R c,H×W , and finally the attention mechanism is used to calculate the formula: Where Q represents the query vector in the attention mechanism, K represents the key vector in the attention mechanism, and V represents the numerical vector in the attention mechanism, corresponding to the above feature graph Q s , K s , V s ;d k is the feature map matrix product QK T The standard deviation of The formula for the softmax function is: Where xi represents the i-th input variable, and n represents the n input variables.

9. According to claim 3, a method for identifying human operation behavior based on lightweight YOLOv7, It is characterized in that The DSC_ELAN module in lightweight YOLOv7 has two branches, one of which passes through a 1×1 convolutional layer; the other branch consists of a 1×1 convolutional layer, and 4 DSCBS modules are connected in sequence. Finally, the output results of the two branches are spliced ​​together and then passed through a 1×1 convolutional layer.

10. According to claim 1, a method for identifying human operation behavior based on lightweight YOLOv7, It is characterized in that In step S3, the embedded device supports the PyTorch deep learning framework, and the PyQt framework developer operates the behavior recognition system and deploys the trained lightweight YOLOv7 network into the software system. The system includes an image processing function module, a target detection function module, a behavior classification function module, and an abnormal feedback function module. The processing steps are as follows: S31. In the image processing function module, after starting the lightweight YOLOv7-based industrial operator behavior recognition system, the user uses offline pictures, video files or online cameras to obtain video images in real time, displays the images on the program user interface, processes the images into the data dimension (640, 640, 3) RGB format required by lightweight YOLOv7, and transmits them to the behavior recognition module for behavior recognition processing; S32, the acquired image is positioned by the target detection function module. Once it is determined to be the position of the industrial equipment operator, the behavior recognition is completed, and the behavior category data (0-8) and the target position tuple data (x, y, w, h) and the confidence score are output. Then the data is passed to the behavior classification function module to classify the industrial equipment operator behavior; S33, the behavior classification function module performs post-processing on the obtained behavior dictionary data, extracts the label with the maximum confidence score, divides the behavior into two categories: normal operation and abnormal operation, and transmits the data to the abnormal behavior feedback function module for risk assessment; S34. The abnormal behavior feedback function module receives the output of the industrial equipment operator behavior classification function module, performs risk assessment according to the corresponding preset rules, triggers the corresponding warning feedback mechanism, prominently marks it on the user interface, records the abnormal behavior as a log, or sends the data to a remote server for further processing.

Images