Driving license test action recognition method and device based on deep learning technology

Abstract

The invention relates to a driving license test action recognition method and device based on a deep learning technology, and the method comprises the following steps: transmitting a color control instruction to an indicator light on a driving license test vehicle, and transmitting a corresponding action voice through a voice module for indicating an examiner to complete a specified action; in the process that the examinee completes the specified action, action video data of the examinee are collected in real time through a camera and input into a pre-constructed deep learning model; and inputting video data of the deep learning model, firstly performing skeleton point data detection through an OpenPose algorithm, then performing action recognition according to the skeleton point data through the action recognition model, and judging whether an examinee completes a specified action or not according to an action recognition result. Compared with the prior art, the method can efficiently and accurately recognize the action completion effect of the applicant, and can adapt to different environments and diversity of the applicants.

Description

Technical Field

[0001] The present invention relates to the technical field of driver's license test motion recognition, and in particular to a driver's license test motion recognition method and device based on deep learning technology. Background Art

[0002] In the field of computer vision and deep learning, action recognition technology has become one of the hot topics of research, especially in automation systems and intelligent monitoring. Traditional action recognition methods rely on complex feature extraction and manual coding techniques, which not only increases the difficulty of implementation, but also limits the application performance of the system in complex environments. The rise of deep learning technology, especially the application of neural networks in feature learning, has brought revolutionary progress to action recognition. Deep learning can automatically extract complex high-level features, greatly improving the accuracy and efficiency of action recognition.

[0003] With the rapid development of deep learning technology, feature learning using neural networks has become a key way to solve the above problems. Deep learning can automatically learn and extract high-level features, greatly improving the accuracy and efficiency of action recognition. In this context, posture estimation and graph convolutional networks (GCN) have become two key technologies in this field.

[0004] One of the key technologies of action recognition, posture estimation technology, aims to identify the key points of the human body from images or videos to provide basic data for further action recognition. By accurately identifying and tracking the key points of the human skeleton, we can better understand the human body's movements and behaviors. Posture estimation technology enables a deeper understanding of human movements and behaviors. The successful application of this technology has promoted the development of the field of action recognition, especially in application scenarios that require accurate action recognition, such as motion analysis, health monitoring, and security monitoring, showing great application potential.

[0005] Another key technology, graph convolutional network (GCN), is a neural network designed for graph structure data, or a neural network specifically used to process graph structure data. It shows its unique advantages in action recognition. In action recognition, the human body can be regarded as a graph composed of skeleton points, where the skeleton points are the nodes of the graph and the connections between the skeletons are the edges of the graph. GCN can effectively learn the relationship between nodes on this graph structure, so as to perform accurate action recognition, even complex actions can be accurately recognized. This graph-based method provides new possibilities and opens up new paths for the recognition of complex actions and interactive actions.

[0006] Although posture estimation and GCN technology have achieved remarkable results in action recognition, how to efficiently integrate these technologies into specific applications, especially in specific scenarios such as driver's license tests, still faces challenges. In addition, considering the particularity of the driver's license test, the system needs to be able to handle a variety of complex actions and maintain high accuracy and real-time performance in a dynamic and changing environment. Therefore, developing an action recognition driver's license test system based on deep learning technology that can not only meet the needs of high accuracy but also adapt to the diversity and complexity of practical applications has become an important direction of research and application in this field. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method and device for driver's license test action recognition based on deep learning technology, which can not only meet the requirements of high accuracy, but also adapt to the diversity and complexity in practical applications.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A method for driver's license test action recognition based on deep learning technology, comprising the following steps:

[0010] Transmit color control instructions to the indicator lights on the driver's license test vehicle, and send corresponding action voices through the voice module to instruct the examinee to complete the specified actions;

[0011] When the examinee is completing the specified action, the camera collects the examinee's action video data in real time and inputs it into the pre-built deep learning model;

[0012] The video data input into the deep learning model is first detected through the OpenPose algorithm for skeleton point data, and then the action recognition model is used to perform action recognition based on the skeleton point data. The action recognition results are used to determine whether the examinee has completed the specified action.

[0013] Furthermore, the OpenPose algorithm obtains confidence maps and partial affinity fields of body key points through neural network processing according to image features of video data, determines the positions of key points according to the confidence maps of body key points, and connects each key point according to the partial affinity fields to form a complete human skeletal structure.

[0014] Furthermore, the OpenPose algorithm also includes adaptively adjusting the threshold for dividing the object to be identified and the environment according to changes in ambient lighting during the process of extracting image features.

[0015] Furthermore, during the training process of the OpenPose algorithm, random illumination change simulation and background noise filtering are performed on the input data, and then training is performed.

[0016] Furthermore, the OpenPose algorithm also includes performing skeleton point data detection after feature enhancement on the input image features.

[0017] Furthermore, the action recognition model constructs human joints into a graph structure based on the skeleton point data, and the graph structure uses joints as nodes and connections between joints as edges; uses graph convolution operations to learn and fuse features in spatial and temporal dimensions, and analyzes feature changes at different time steps to capture the dynamic patterns and laws of actions, thereby identifying action categories.

[0018] Furthermore, the action recognition model is a ST-GCN model.

[0019] Furthermore, the action voice emitted by the voice module corresponds one to one with the color displayed by the indicator light.

[0020] Furthermore, the indicator light is installed in the driving license test vehicle and faces the main driving seat.

[0021] The present invention also provides a driving license test action recognition device based on deep learning technology, comprising a memory, a processor, an indicator light, a voice module and a camera, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of the method described above;

[0022] The indicator light is used to display a corresponding color according to a color control instruction transmitted by the processor;

[0023] The voice module is used to issue corresponding action voice under the control of the processor;

[0024] The camera is used to collect the motion video data of the examinee in real time under the control of the processor.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] (1) The present invention first prompts the applicant to perform a specified action by displaying different colors and issuing voice messages, making full use of the dual prompts of vision and hearing to ensure that the applicant can clearly and accurately understand the action to be completed, thereby improving the accuracy of action execution;

[0027] Based on the input data from the camera, the system first marks the applicant's skeleton nodes through the OpenPose model, and then inputs the data into another model for action recognition, ST-GCN, for further processing and action recognition; and determines whether the applicant has successfully completed the specified action;

[0028] A motion recognition driver's license test system device has been realized that can efficiently and accurately identify the applicant's action completion effect. The combination of advanced posture estimation technology and graph convolutional network not only improves the accuracy of motion recognition, but also enhances the system's ability to recognize complex actions, realizing real-time motion capture and accurate recognition. It can not only adapt to different environments and the diversity of applicants, but also realize technological innovation and progress in the field of driver's license examinations, indicating huge market potential in the future.

[0029] (2) The present invention also takes into account the application challenges of action recognition in actual operations, such as lighting changes, background complexity, and the diversity of applicants' clothing, and adopts advanced data enhancement and model optimization strategies to ensure the stability and reliability of the system under various conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart of a method for identifying driver's license test actions based on deep learning technology provided in an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of an OpenPose network structure provided in an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of the ST-GCN network structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0034] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0036] Example 1

[0037] This embodiment provides a method for driver's license test action recognition based on deep learning technology, including the following steps:

[0038] Transmit color control instructions to the indicator lights on the driver's license test vehicle, and send corresponding action voices through the voice module to instruct the examinee to complete the specified actions;

[0039] When the examinee is completing the specified action, the camera collects the examinee's action video data in real time and inputs it into the pre-built deep learning model;

[0040] The video data input into the deep learning model is first detected through the OpenPose algorithm for skeleton point data, and then the action recognition model is used to perform action recognition based on the skeleton point data. The action recognition results are used to determine whether the examinee has completed the specified action.

[0041] like Figure 1 As shown, the actual scenario includes the following steps:

[0042] S1: The applicant will hear a voice message according to the different colors displayed by the machine, which will instruct the applicant to do a specified action;

[0043] The action voice issued by the voice module corresponds to the color displayed by the indicator light;

[0044] The indicator light is installed in the driving test vehicle and faces the main driver's seat;

[0045] S2: The applicant completes the corresponding actions according to the instructions of S1. This scheme makes full use of visual and auditory dual prompts to ensure that the applicant can clearly and accurately understand the actions to be completed, thereby improving the accuracy of action execution;

[0046] S3: The camera takes real-time video of the applicant and inputs the video data into the deep learning model in real time;

[0047] S4: The deep learning model first uses the improved OpenPose algorithm based on the input in S3 to enhance the accuracy and robustness of skeleton point detection, especially in the case of lighting changes and complex backgrounds. The improvement method includes the use of adaptive threshold processing and enhanced feature extraction technology;

[0048] S5: The skeleton point data obtained in S4 is used as the input of the action recognition model ST-GCN for action recognition, and the final recognition result is output to determine whether the applicant has completed the corresponding action. The combination of advanced posture estimation technology and graph convolutional network not only improves the accuracy of action recognition, but also enhances the system's recognition ability for complex actions, and realizes real-time action capture and accurate recognition. It can not only adapt to different environments and the diversity of applicants, but also achieve technological innovation and progress in the field of driver's license examinations, indicating huge market potential in the future. In addition, the present invention also considers the application challenges of action recognition in actual operations, such as lighting changes, background complexity, and the diversity of applicants' clothing, and adopts advanced data enhancement and model optimization strategies to ensure the stability and reliability of the system under various conditions. Through these technical innovations, the present invention not only improves the accuracy and practicality of action recognition, but also provides new ideas and methods for research in related fields.

[0049] Specifically, Figure 2 As shown in the figure, in the OpenPose algorithm, the OpenPose algorithm obtains the confidence map and partial affinity field of the key points of the body through neural network processing according to the image features of the video data, determines the position of the key points according to the confidence map of the key points of the body, and connects each key point according to the partial affinity field to form a complete human skeleton structure. The corresponding calculation expression is:

[0050]

[0051] Where ρt is the CNN for inference at stage t; t is the CNN for inference at stage t; L t is the partial affinity domain set at stage t; S t is the detection confidence atlas of body key points at stage t, and F is the image feature;

[0052] The processing and analysis of St and Lt can realize the detection and extraction of skeleton points. Specifically, St is first analyzed to find the possible key point positions, and then Lt is used to connect these key points to form a complete human skeleton structure.

[0053] During the implementation process, image processing and machine learning techniques, such as convolutional neural networks, are usually used to perform feature extraction and pattern recognition on the input image or video to accurately obtain skeleton point information.

[0054] The OpenPose algorithm also includes adaptively adjusting the threshold for dividing the object to be identified and the environment according to changes in ambient lighting during the process of extracting image features, and performing skeleton point data detection after feature enhancement of the input image features.

[0055] During the training process of the OpenPose algorithm, random illumination changes are simulated and background noise is filtered on the input data before training.

[0056] In the action recognition model, the action recognition model constructs human joints into a graph structure based on the skeleton point data. The graph structure uses joints as nodes and the connections between joints as edges. It uses graph convolution operations to learn and fuse features in spatial and temporal dimensions, and analyzes feature changes at different time steps to capture the dynamic patterns and rules of actions, thereby identifying action categories.

[0057] like Figure 3 As shown, the action recognition model used is the ST-GCN model, and the formula of ST-GCN is as follows:

[0058] fout=Λ-1 / 2(A+I)Λ-1 / 2finW

[0059] Λii=Σj(Aij+Iij)

[0060] In the formula, A is the adjacency matrix of the graph; I is the unit matrix; W is composed of weight vectors of multiple output channels; fin represents the input feature map, whose dimension is (c, V, T), where V is the number of nodes and T is the number of frames; fout is the output feature, which is the result obtained after a series of convolutions and calculations; Λii is the weight and parameter defined in the graph convolutional network, which is used to control the way information is transmitted and fused between nodes.

[0061] Example 2

[0062] This embodiment provides a driving license test action recognition device based on deep learning technology, including a memory, a processor, an indicator light, a voice module and a camera, the memory stores a computer program, and the processor calls the computer program to execute the steps of the method in Embodiment 1;

[0063] The indicator light is used to display the corresponding color according to the color control instruction transmitted by the processor;

[0064] The voice module is used to issue corresponding action voice under the control of the processor;

[0065] The camera is used to collect the examinee's motion video data in real time under the control of the processor.

[0066] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for driver's license test action recognition based on deep learning technology, characterized in that: The following steps are involved: Transmit color control instructions to the indicator lights on the driver's license test vehicle, and send corresponding action voices through the voice module to instruct the examinee to complete the specified actions; When the examinee is completing the specified action, the camera collects the examinee's action video data in real time and inputs it into the pre-built deep learning model; The video data input into the deep learning model is first detected through the OpenPose algorithm for skeleton point data, and then the action recognition model is used to perform action recognition based on the skeleton point data. The action recognition results are used to determine whether the examinee has completed the specified action.

2. The method for driver's license test action recognition based on deep learning technology according to claim 1, characterized in that: The OpenPose algorithm obtains the confidence map and partial affinity field of the key points of the body through neural network processing according to the image features of the video data, determines the position of the key points according to the confidence map of the key points of the body, and connects the key points according to the partial affinity field to form a complete human skeletal structure.

3. The method for driver's license test action recognition based on deep learning technology according to claim 2, characterized in that: The OpenPose algorithm also includes adaptively adjusting the threshold for dividing the object to be identified and the environment according to changes in ambient lighting during the process of extracting image features.

4. The method for driver's license test action recognition based on deep learning technology according to claim 3 is characterized in that: During the training process of the OpenPose algorithm, random illumination change simulation and background noise filtering are performed on the input data, and then training is performed.

5. The method for driver's license test action recognition based on deep learning technology according to claim 2, characterized in that: The OpenPose algorithm also includes performing skeleton point data detection after feature enhancement on the input image features.

6. The method for driver's license test action recognition based on deep learning technology according to claim 1, characterized in that: The action recognition model constructs human joints into a graph structure based on skeleton point data, with joints as nodes and connections between joints as edges; uses graph convolution operations to learn and fuse features in spatial and temporal dimensions, and analyzes feature changes at different time steps to capture the dynamic patterns and laws of actions, thereby identifying action categories.

7. The method for driver's license test action recognition based on deep learning technology according to claim 1, characterized in that: The action recognition model is a ST-GCN model.

8. The method for driver's license test action recognition based on deep learning technology according to claim 1, characterized in that: The action voice emitted by the voice module corresponds to the color displayed by the indicator light.

9. The method for driver's license test action recognition based on deep learning technology according to claim 1, characterized in that: The indicator light is installed in the driving license test vehicle and faces the main driving seat.

10. A driver's license test action recognition device based on deep learning technology, characterized in that: It comprises a memory, a processor, an indicator light, a voice module and a camera, the memory stores a computer program, and the processor calls the computer program to execute the steps of any method described in claim 1 to 9; The indicator light is used to display a corresponding color according to a color control instruction transmitted by the processor; The voice module is used to issue corresponding action voice under the control of the processor; The camera is used to collect the motion video data of the examinee in real time under the control of the processor.

Images