Gesture recognition method for nuclear power pipe gallery environment

Abstract

The invention relates to the technical field of nuclear power field gesture recognition, and aims to solve the problem that warning gesture information of workers is difficult to timely and efficiently recognize due to interference of environmental factors in existing nuclear power station pipe gallery environment field operation. The invention provides a gesture recognition method for a nuclear power pipe gallery environment. The gesture recognition method comprises the steps that RGB videos and depth information are collected; deploying a YOLO network model on a terminal device, and carrying out shunting real-time processing on the collected multi-modal information to obtain a region of interest; detecting key frames of the cut video sequence; extracting a gesture video frame sequence; performing video frame spatio-temporal information compression representation; performing cross-modal feature learning and gesture fusion recognition on the compressed RGB and depth video frames; and performing result processing on the judgment result of the gesture through the terminal. According to the invention, the gesture category of the worker can be accurately judged, and the safety of the worker in the pipe gallery environment is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of nuclear power site gesture recognition, and in particular to a gesture recognition method for a nuclear power pipeline gallery environment. Background Art

[0002] The work process in the nuclear power pipeline corridor environment area is inseparable from the efficient information exchange between operators and between operators and control machines. Especially in the collaborative work process of complex scenes, the accurate recognition of the operator's gestures will be affected by factors such as lighting changes, occlusion, and personal inattention. At present, the information interaction recognition of on-site workers in the pipeline corridor environment is mainly determined by mutual observation between the workers, which is easily affected by environmental factors and it is difficult to actively recognize and warn the gestures of the workers efficiently.

[0003] In the prior art, there are many methods for gesture recognition, including traditional manual feature extraction methods based on computer vision data, inertial sensor data and other modalities and learning methods based on deep neural networks. Among them, the neural network learning method has higher recognition performance and has become the mainstream of current algorithm design. The recognition method based on the deep neural network model can process end-to-end multimodal data, so that it can design a more efficient multimodal complementary feature learning model. RGB video and depth sequence in computer vision data are two common data sources. The two modal data of the same gesture action contain rich complementary information. RGB and depth modalities contain rich complementary information, and convolutional neural networks have great advantages in multimodal action recognition. Due to the limitations of single modality data in feature extraction, it is difficult to improve the recognition performance of the network model by learning complementary features of different modalities. Multi-stream learning mechanism and video sequence spatiotemporal information representation are effective methods to solve the above problems.

[0004] At present, gesture recognition methods based on deep neural network model learning are mainly divided into two categories: gesture recognition methods based on convolutional neural networks (CNN) and gesture recognition methods based on other deep networks. In the CNN gesture recognition method, a certain number of video frames are obtained as network input mainly through sparse sampling methods, and gestures are extracted and classified through the designed neural network. The recurrent convolutional network (RNN) memorizes the previous information and applies it to the calculation of the current output. It can process sequence data of any length and realize feature learning and recognition classification through orderly cyclic learning of the input sequence. It has achieved good application in the field of natural language processing, but it still needs to be further improved in gesture recognition.

[0005] Currently, gesture recognition based on deep learning network models is mostly based on single data modalities such as RGB video data or inertial sensors. Among them, gesture recognition methods based on RGB video information are greatly affected by environmental factors such as lighting, gesture recognition methods based on depth information are affected by factors such as sample resolution and lack of appearance texture information and have low performance, and recognition methods based on inertial sensor data have low recognition rates due to data source dimension limitations. Multimodal gesture recognition methods based on RGB video and depth information streams have better recognition performance and environmental adaptability, but are limited by the scale of network model parameters, which limits the application scenarios of recognition models. The RGB video information and depth information of the same gesture action have a high degree of frame-by-frame complementary characteristics. Designing a multimodal feature complementary learning model can efficiently learn complementary features of different modalities, thereby improving the recognition performance of the algorithm model. Summary of the invention

[0006] The purpose of the present invention is to provide a gesture recognition method for a nuclear power pipeline corridor environment, so as to solve the problem that the warning gesture information of the staff is difficult to be recognized in a timely and efficient manner due to interference from environmental factors in the existing nuclear power plant pipeline corridor environment on-site operations.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] On the one hand, the present invention provides a gesture recognition method for a nuclear power pipeline corridor environment, comprising:

[0009] Step 1: Collect RGB video and depth information;

[0010] Step 2: Deploy the YOLO network model on the terminal device to process the collected multimodal information in real time to obtain the region of interest;

[0011] Step 3: Detect key frames of the cropped video sequence;

[0012] Step 4: Extract gesture video frame sequence;

[0013] Step 5: Compressed representation of spatiotemporal information of video frames;

[0014] Step 6: Perform cross-modal feature learning and gesture fusion recognition on the compressed RGB and depth video frames;

[0015] Step 7: The terminal processes the result of the gesture determination.

[0016] In some embodiments, the collected RGB and depth sequence information is processed in real time through the YOLO network model to detect the region of interest in the video, crop the region of interest in the video frame, and remove background noise.

[0017] In some embodiments, key frames are marked by visual processing methods according to the visual characteristics of the gesture action.

[0018] In some embodiments, a continuous sequence of frames containing a gesture is obtained by adding and subtracting the preceding and succeeding frames.

[0019] In some embodiments, when a warning gesture is detected, L frames of images are recorded and traced back and forth to form a video sequence of 2L+1 frames.

[0020] In some embodiments, a video frame spatiotemporal information compression method is used to compress a 2L+1 frame video sequence into a single frame.

[0021] In some embodiments, the RGB and depth sequences are processed separately:

[0022] Assume that the RGB or Depth image sequence has T frames, expressed as <I1,I2,I3,...,I T >, define the moving image of its video sequence as MI:

[0023]

[0024] For the sequence frames in the RGB video, the compressed spatiotemporal information is represented in three channels respectively, and the motion information obtained in the calculation process is rounded off to finally obtain a motion image similar to the RGB three channels; for the depth sequence diagram, it is first converted into a grayscale image of [0,255] by scaling, and then the motion spatiotemporal information of the single-channel grayscale image is represented, and the motion image of the single channel is finally obtained by rounding off.

[0025] In some embodiments, the compressed RGB frame and depth map are fed into the recognition model through a dual-stream deep learning network model and a global feature compensation learning module.

[0026] On the other hand, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the gesture recognition method for a nuclear power pipeline corridor environment are implemented.

[0027] On the other hand, the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored, and when the computer-readable instructions are executed, the steps of the gesture recognition method for a nuclear power pipeline corridor environment are implemented.

[0028] Compared with the prior art, the gesture recognition method for a nuclear power pipeline corridor provided by the present invention has the following beneficial effects:

[0029] The present invention collects image video and depth sequence information, combines it with a deep learning network model, and performs spatiotemporal information learning and feature fusion on the video and depth information, so as to accurately judge the gesture category of the staff and ensure the safety of the personnel in the tunnel environment.

[0030] The present invention provides a set of practical, efficient and accurate recognition methods for common gestures and information interactions of workers in a complex nuclear power plant pipe gallery environment.

[0031] Furthermore, the RGB and depth map information collection method adopted by the present invention can effectively avoid interference from factors such as ambient lighting and occlusion, and provide complementary and reliable sample information for the accuracy of gesture recognition.

[0032] Furthermore, the present invention introduces the steps of cropping the video frame region of interest and compressing the video sequence to remove background noise to the maximum extent.

[0033] Furthermore, the present invention proposes a method for compressing and representing spatiotemporal information of video frames, which effectively reduces the amount of input data of the algorithm model and effectively reduces the amount of network model parameters.

[0034] Furthermore, the dual-stream deep learning network model designed in the present invention utilizes the collected data information more efficiently, ensures high-precision recognition without interference, and avoids operational errors caused by gesture recognition errors.

[0035] Furthermore, through image cropping of the video frame area and compressed representation of the sequence before and after the key frame, the amount of input data of the algorithm model is effectively reduced, which can be applied to lightweight network models. The overall recognition framework can be integrated into handheld devices, which is convenient and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for the technical description.

[0037] Figure 1 A flow chart of a gesture recognition method for a nuclear power pipeline gallery environment provided by the present invention;

[0038] Figure 2 The dual-stream deep learning network model provided by the present invention;

[0039] Figure 3 A schematic diagram of the global feature compensation learning module provided by the present invention. DETAILED DESCRIPTION

[0040] The following is further explained in detail through specific implementation methods.

[0041] like Figure 1 As shown, the present invention provides a gesture recognition method for a nuclear power pipeline corridor environment, comprising:

[0042] Step 1: Collect RGB video and depth information. Set up a camera capable of collecting RGB and depth information in the nuclear power plant tunnel environment to collect RGB and depth sample information.

[0043] Step 2: YOLO target detection region of interest, deploy the YOLO network model on the terminal device, and process the collected multimodal information in real time to obtain the region of interest, that is, the human image region of the staff.

[0044] The collected RGB and depth sequence information is processed in real time through the YOLO network model to detect the region of interest (staff target area) in the video and crop the region of interest in the video frame to remove background noise.

[0045] Step 3: Detect key frames of the cropped video sequence. According to the visual characteristics of the gesture action, the key frames are marked through visual processing methods.

[0046] Step 4: Extract the gesture video frame sequence. According to the key frame number found, the continuous frame sequence containing the gesture is obtained by adding and subtracting the forward and backward frames. This step processes the RGB video and depth information separately to obtain data information of the two modalities.

[0047] Step 5: Compress and represent the spatiotemporal information of video frames. The continuous video frame sequence obtained in step 4 is compressed and represented as a single video frame through the visual spatiotemporal information representation method.

[0048] Step 6: Gesture recognition. The RGB video and depth map sequences of the same gesture have high consistency and complementarity of feature information. A two-stream deep learning network model and a global feature compensation learning module are designed to realize gesture recognition in complex scenes. The two-stream deep learning network model is used to process the RGB and depth map video information streams simultaneously, and the global feature compensation learning module (GCB module) is used to learn the compensation features between the RGB and depth map video information streams. The designed GCB module is used to learn cross-modal features of the compressed RGB and depth video frames, and the high-level features are efficiently fused through the global attention mechanism, finally realizing efficient gesture recognition. The two-stream learning of RGB and depth map information videos can ensure that the algorithm model fully learns the underlying features of different modalities and the complementary features between modalities, so that the algorithm model can fully learn local and global features, which can improve the anti-interference ability of the algorithm model, thereby ensuring the recognition performance and generalization performance of the algorithm model. In addition, the two-stream features can be trained separately, which can realize flexible processing of single modality and multi-modality. .

[0049] Step 7: Handling of gesture recognition results: The terminal processes the gesture recognition results for information exchange between staff members or human-computer information interaction.

[0050] As an implementable manner, the gesture recognition method provided by the present invention includes:

[0051] Build data acquisition and storage processing equipment, set up cameras capable of collecting RGB and depth information at the nuclear power plant pipeline corridor environment, and implement RGB and depth sample information collection;

[0052] Deploy the YOLO network model on the terminal device to process the collected multimodal information in real time to obtain the region of interest, that is, the human image region of the staff;

[0053] Real-time detection of human-shaped image areas and marking of key frames through visual processing methods such as edge detection;

[0054] When a warning gesture such as "crossing arms above the head" is detected, L frames of images are recorded and traced back and forth to form a 2L+1 frame video sequence, which includes RGB video frames and depth map sequences;

[0055] The video frame spatiotemporal information compression method is used to compress the 2L+1 frame video sequence into a single frame, which contains rich spatiotemporal features;

[0056] Process RGB and depth sequences separately;

[0057] Assume that the RGB or Depth image sequence has T frames, expressed as <I1,I2,I3,...,I T >, define the moving image of its video sequence as MI:

[0058]

[0059] τ is an intermediate variable, taking values ​​from 1 to T.

[0060] For the sequence frames in the RGB video, the compressed spatiotemporal information can be represented in three channels respectively, and the motion information obtained in the calculation process can be rounded to obtain a motion image similar to the RGB three channels. For the depth sequence diagram, it is first converted into a grayscale image of [0,255] by scaling, and then the motion spatiotemporal information of the single-channel grayscale image is represented, and the motion image of the single channel is finally obtained by rounding.

[0061] like Figure 2 and Figure 3 As shown, a two-stream deep learning network model and a global feature compensation learning module are designed to feed the compressed RGB frame and depth map into the recognition model; Figure 2 The figure shows a two-stream network model that can process RGB and depth map information simultaneously and learn consistent features of the two modalities; Figure 3Shown is the GCB module, which can learn complementary features between RGB and depth map modalities; Figure 2 and 3 It can learn the consistency and complementary features of the two modalities to ensure the recognition and generalization performance of the algorithm model.

[0062] The terminal determines the gesture result and processes information exchange between staff members or human-computer information interaction.

[0063] In summary, the present invention collects image video and depth sequence information, combines it with a deep learning network model, and performs spatiotemporal information learning and feature fusion on the video and depth information, so as to accurately judge the gesture category of the staff and ensure the safety of the personnel in the tunnel environment.

[0064] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above-mentioned gesture recognition method for a nuclear power pipeline corridor environment are implemented.

[0065] In addition, the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed, the steps of the above-mentioned gesture recognition method for a nuclear power pipeline corridor environment are implemented.

[0066] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0067] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A gesture recognition method for a nuclear power pipeline corridor environment, characterized in that: include: Step 1: Collect RGB video and depth information; Step 2: Deploy the YOLO network model on the terminal device to process the collected multimodal information in real time to obtain the region of interest; Step 3: Detect key frames of the cropped video sequence; Step 4: Extract gesture video frame sequence; Step 5: Compressed representation of spatiotemporal information of video frames; Step 6: Perform cross-modal feature learning and gesture fusion recognition on the compressed RGB and depth video frames; Step 7: The terminal processes the result of the gesture determination.

2. The gesture recognition method for a nuclear power pipeline corridor environment according to claim 1 is characterized in that: In step 2, the collected RGB and depth sequence information is processed in real time through the YOLO network model to detect the region of interest in the video, crop the region of interest in the video frame, and remove background noise.

3. The gesture recognition method for a nuclear power pipeline corridor environment according to claim 1, characterized in that: In step 3, key frames are marked by visual processing methods according to the visual characteristics of the gesture action.

4. The gesture recognition method for a nuclear power pipeline corridor environment according to claim 1, characterized in that: In step 4, a continuous frame sequence containing the gesture is obtained by adding and subtracting the forward and backward frames.

5. The gesture recognition method for a nuclear power pipeline gallery environment according to claim 1, characterized in that: In step 4, when a warning gesture is detected, L frames of images are recorded and traced back and forth to form a video sequence of 2L+1 frames.

6. The gesture recognition method for a nuclear power pipeline corridor environment according to claim 5 is characterized in that: The video frame spatiotemporal information compression method is used to compress the 2L+1 frame video sequence into a single frame.

7. The gesture recognition method for a nuclear power pipeline corridor environment according to claim 1, characterized in that: In step 5, the RGB and depth sequences are processed separately: Assume that the RGB or Depth image sequence has T frames, represented as I1,I2,I3,...,I T , define the moving image of its video sequence as MI: For the sequence frames in the RGB video, the compressed spatiotemporal information is represented in three channels respectively, and the motion information obtained in the calculation process is rounded off to finally obtain a motion image similar to the RGB three channels; for the depth sequence diagram, it is first converted into a grayscale image of [0,255] by scaling, and then the motion spatiotemporal information of the single-channel grayscale image is represented, and the motion image of the single channel is finally obtained by rounding off.

8. The gesture recognition method for a nuclear power pipeline gallery environment according to claim 1, characterized in that: In step 6, the compressed RGB frame and depth map are sent to the recognition model through the two-stream deep learning network model and the global feature compensation learning module.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the processor executes the computer-readable instructions, the steps of the gesture recognition method for a nuclear power pipeline corridor environment described in any one of claims 1-8 are implemented.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed, the steps of the gesture recognition method for a nuclear power pipeline corridor environment described in any one of claims 1-8 are implemented.

Images