Driver specific action detection method

Abstract

A driver-specific action detection method relates to the field of image processing technology. The driver's behavior action image data is collected by a camera acquisition unit and imported into a key point detection module to obtain the two-dimensional coordinates of each joint of the driver respectively; the two-dimensional coordinates of each joint point are mapped to a three-dimensional coordinate system under the driving position through a three-dimensional key point mapping model; the coordinate points under the three-dimensional coordinate system are input into a pre-trained action recognition model to obtain the driver action monitoring result. Compared with the prior art, the technical solution of the present invention no longer relies solely on optical flow to extract information, thereby improving the readiness of the detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting specific actions of a driver. Background Art

[0002] With the increasing number of motor vehicles in my country, the incidence of traffic accidents is also increasing year by year. The driver is the dominant person in the vehicle. Driver behaviors such as not wearing a seat belt and answering or making phone calls during driving pose huge safety risks. Therefore, regulating driver behavior plays a very important role in reducing the occurrence of traffic accidents. By detecting the driver's actions, the driver can be reminded when the driver's driving actions are not standardized, avoiding traffic accidents caused by unsafe and standardized actions.

[0003] In the prior art, the features of the image of feature points and the trajectory of the movement direction of the limbs are mainly used. For example, the DT algorithm predicts by connecting the trajectory of key points (dense optical flow) of several consecutive frames and features such as HOG in the trajectory process. This method cannot effectively utilize the multi-scale of the image and the feature calculation is huge, and the robustness is poor. The detection accuracy is low in real scenes. Summary of the invention

[0004] An embodiment of the present invention provides a method for detecting driver-specific actions, which obtains image information collected by the driver of a driving vehicle during the driving process; pre-processes the image information and sends it to a pre-trained key point detection module to obtain the two-dimensional coordinates of each joint point of the driver; maps the two-dimensional coordinates of each joint point to a three-dimensional coordinate system under the driving position through a three-dimensional key point mapping model; inputs the coordinate points under the three-dimensional coordinate system into a pre-trained action recognition model to obtain the driver action monitoring result, thereby solving the problem of low detection accuracy in real scenarios that exists in current detection technologies.

[0005] In view of the above problems, the present invention is proposed to provide a method for detecting a specific action of a driver.

[0006] The embodiment of the present invention is implemented as follows: a driver specific action detection method is applied to a data acquisition end, and the method includes:

[0007] Acquire image data collected by the driver during driving;

[0008] The image data collected in the cockpit is transmitted to the data processing end.

[0009] In another embodiment, the present invention provides a method for detecting a specific driver action, which is applied to a data processing end, and the method includes:

[0010] Acquire image data collected by the driver during driving and transmitted by the data acquisition end;

[0011] The acquired image data is analyzed to obtain the driver action monitoring results.

[0012] Furthermore, the image data collected during the driver's driving process and transmitted by the data acquisition end are analyzed to obtain the driver's action monitoring result, which specifically includes the following steps:

[0013] Obtain the two-dimensional coordinates of each joint point of the driver;

[0014] Map the two-dimensional coordinates of each joint point in the image to the three-dimensional coordinate system under the driving position;

[0015] The three-dimensional coordinates of the joint points in the three-dimensional coordinate system are analyzed to obtain the driver's action monitoring results.

[0016] In a second aspect, an embodiment of the present invention provides a driver specific action detection system, the system comprising:

[0017] A data acquisition end, used to execute the method described in the data acquisition end of the above embodiment;

[0018] The data processing end communicates with the data acquisition end and is used to execute the method of the data processing end in the above embodiment.

[0019] In a third aspect, an embodiment of the present invention provides a driver specific action detection device, which is applied to a data acquisition end, and includes:

[0020] The acquisition module is used to acquire the image information in the cab collected during the driving process and transmit the image information to the detection module.

[0021] In another embodiment, the present invention provides a driver specific action detection device, which is applied to a data processing end and is characterized by comprising:

[0022] The detection module is used to pre-process the input image, obtain the two-dimensional coordinates of each joint point of the driver, map the two-dimensional coordinates of each joint point in the image to the three-dimensional coordinate system under the driving position, analyze the coordinate points in the three-dimensional coordinate system, and obtain the driver action monitoring results.

[0023] Furthermore, the acquisition module includes a camera acquisition unit, and the camera acquisition unit is used to acquire image information in the cab.

[0024] Furthermore, the detection module includes a key point detection module, a three-dimensional key point mapping model and a motion recognition model. The key point detection module is used to obtain the two-dimensional coordinates of each joint point of the driver based on image data analysis. The three-dimensional key point mapping model is used to map the two-dimensional key points on the image to the three-dimensional coordinate system under the driving position after the key point detection module outputs the two-dimensional coordinates of the joint points. The motion recognition model is used to identify the coordinate points in the three-dimensional coordinate system under the driving position to obtain the driver motion monitoring results.

[0025] Furthermore, the preprocessing is to perform image normalization processing on the collected images.

[0026] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0027] The technical solution described in the present invention collects the driver's behavior and action image data through a camera acquisition unit and imports it into a key point detection module to obtain the two-dimensional coordinates of each joint of the driver respectively; the two-dimensional coordinates of each joint point are mapped to the three-dimensional coordinate system under the driving position through a three-dimensional key point mapping model; the coordinate points under the three-dimensional coordinate system are input into a pre-trained action recognition model to obtain the driver action monitoring result. Compared with the prior art, the technical solution described in the present invention no longer relies solely on optical flow to extract information, thereby improving the readiness of the detection results.

[0028] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0029] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0031] Figure 1 A diagram of an application environment of a driver specific action detection method in one embodiment;

[0032] Figure 2 A schematic diagram of the structure of a driver specific action detection system disclosed in an embodiment of the present invention;

[0033] Figure 3 This is a flow chart of a method for detecting a specific driver action disclosed in Embodiment 1 of the present invention;

[0034] Figure 4 This is a schematic diagram of the key point detection module and the limb affinity module disclosed in the third embodiment of the present invention;

[0035] Figure 5 A schematic diagram of the structure of a three-dimensional key point mapping model disclosed in an embodiment of the present invention;

[0036] Figure 6 It is a schematic diagram of the structure of the action recognition model disclosed in the prior art;

[0037] Figure 7 It is a schematic diagram of the structure of the temporal convolution (TCN) module disclosed in the prior art.

[0038] Reference numerals:

[0039] 100-data acquisition end, 101-acquisition module, 1011-camera acquisition unit, 200-data processing end, 201-detection module, 2011-key point detection module, 2012-three-dimensional key point mapping model, 2013-action recognition model. Specific embodiments

[0040] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] Embodiment 1

[0042] like Figure 1-3 As shown, an embodiment of the present invention provides a method for detecting a specific driver action, the method comprising:

[0043] S1, driver image acquisition, the camera acquisition unit 1011 acquires the image data of the driver during driving, and sends the acquired image data to the key point detection module 2011 after pre-processing;

[0044] S2, obtaining the two-dimensional coordinates of the joint points. The key point detection module 2011 receives and analyzes the image data sent by the camera acquisition unit 1011, obtains the two-dimensional coordinates of each joint point of the driver, and sends the two-dimensional coordinates of each joint point of the driver to the three-dimensional key point mapping model 2012;

[0045] S3, mapping three-dimensional coordinates, the three-dimensional key point mapping model 2012 receives the two-dimensional coordinates of each joint point of the driver sent by the key point detection module 2011, maps the two-dimensional coordinates of each joint point in the image to the three-dimensional coordinate system under the driving position, and sends the mapped three-dimensional coordinate system to the action recognition model 2013;

[0046] S4, action analysis: the action recognition model 2013 receives the mapped three-dimensional coordinate system sent by the three-dimensional key point mapping model 2012 and analyzes it to obtain the driver action detection result.

[0047] Compared with the prior art, the technical solution of the present invention no longer relies solely on optical flow to extract information, thereby improving the readiness of the detection results.

[0048] Embodiment 2

[0049] The embodiment of the present invention also discloses a driver specific action detection system, such as Figure 1-2 As shown, including:

[0050] The data acquisition end 100 acquires the image data of the driver during driving, pre-processes the acquired image data, and then sends it to the data processing end 200 for analysis and processing;

[0051] The data processing end 200 receives the image data and analyzes the image data to obtain the driver action detection result;

[0052] Compared with the prior art, the technical solution of the present invention no longer relies solely on optical flow to extract information, thereby improving the readiness of the detection results.

[0053] Embodiment 3

[0054] The embodiment of the present invention also discloses a driver specific action detection device, such as Figure 1-7 As shown, including:

[0055] The acquisition module 101 is used to acquire the image information in the cab collected during driving and transmit the image information to the detection module 201. The acquisition module 101 includes a camera acquisition unit 1011, which is used to collect the image information in the cab;

[0056] Specifically, the camera acquisition unit 1011 acquires images of a series of behaviors of the driver in the cockpit, and pre-processes the acquired image data, wherein the pre-processing includes performing image normalization processing on the images.

[0057] The detection module 201 is used to pre-process the input image, obtain the two-dimensional coordinates of each joint point of the driver, map the two-dimensional coordinates of each joint point in the image to the three-dimensional coordinate system under the driving position, analyze the coordinate points in the three-dimensional coordinate system, and obtain the driver's action monitoring result. The detection module 201 includes a key point detection module 2011, a three-dimensional key point mapping model 2012 and a motion recognition model 2013; the key point detection module 2011 is used to obtain the two-dimensional coordinates of each joint point of the driver according to image data analysis. In this embodiment, a limb affinity module is also used to obtain the two-dimensional coordinates of each joint point of the driver; the three-dimensional key point mapping model 2012 is used to map the two-dimensional key points on the image to the three-dimensional coordinate system under the driving position after the key point detection module 2011 outputs the two-dimensional coordinates of the joint points. The motion recognition model 2013 is used to identify the coordinate points in the three-dimensional coordinate system under the driving position to obtain the driver's action monitoring result.

[0058] Specifically, the key point detection module 2011 and the limb affinity module: the image is scaled to 368x368 pixels, combined with the annotation file, and input into the key point network and affinity network for training, wherein a variety of convolution kernels (3x3, 5x5) are used for feature extraction, and the maximum pooling layer is used for downsampling to further extract high-level abstract image information, and ReLu6 is used as the activation function to improve the network's fitting ability and speed up the network's training, and further normalized using GroupNormalization, and finally output the feature map;

[0059] After extracting the image feature map, the key points of the human body and the connection lines of each limb are obtained after post-processing. The key point format is:

[0060] ((x1,y1,class1),(x2,y2,class2)...(x16,y16,class16),(x17,y17,class17));

[0061] 3D Keypoint Mapping Model 2012:

[0062] After the key point detection module 2011 outputs the key point positions of the human body, the two-dimensional key points on the image are mapped to the three-dimensional coordinate system of the space through the three-dimensional key point mapping model 2012. The network structure is as follows: Figure 5 As shown, a linear model is used to output the input two-dimensional key points as the three-dimensional coordinates of the human body in the spatial coordinate system.

[0063] ((x1,y1,z1,class1),(x2,y2,z2,class2)...(x16,y16,z16,class16),(x17,y17,z17,clas s17));

[0064] Action Recognition Model 2013:

[0065] After extracting the key points of the three-dimensional coordinates of the human body, the graph neural network is used for human action recognition. A 5-dimensional matrix (N, C, T, V, M) is used to represent each video group. N represents the number of videos. Usually a group has N videos. C represents the characteristics of the joints. Usually a joint contains 4 features such as (x, y, z, acc). T represents the number of key frames. Generally, a video has 150 frames. V represents the number of joints. Usually one person annotates 18 joints. M represents the number of people in a frame. The two people with the highest average confidence are selected. The input of ST-GCN is in the shape of (N, C, T, V, M). Its network structure is as follows: Figure 6 As shown, the graph convolution (GCN) formula is:

[0066] aggre(x)=D -1 AX

[0067] It can be simplified as:

[0068]

[0069] Among them, X is the node feature, X i is the feature of node i. A is the adjacency matrix, A ij represents the weight of the edge between node i and node j, and D is the degree matrix of A;

[0070] The temporal convolution (TCN) module structure is as follows Figure 7 As shown in the figure, Input is the input layer, X is the input frame, T is the index of the frame, and d is the expansion factor. Output is the output layer, y is the output result, and finally the GCN and TCN modules are combined to output the action.

[0071] The specific working principle is that the camera acquisition unit 1011 acquires the image data of the driver during the driving process, pre-processes the acquired image data and sends it to the key point detection module 2011, the key point detection module 2011 receives the image data sent by the camera acquisition unit 1011 and analyzes it to obtain the two-dimensional coordinates of each joint point of the driver, and sends the two-dimensional coordinates of each joint point of the driver to the three-dimensional key point mapping model 2012, the three-dimensional key point mapping model 2012 receives the two-dimensional coordinates of each joint point of the driver sent by the key point detection module 2011, maps the two-dimensional coordinates of each joint point in the image to the three-dimensional coordinate system under the driving position, and sends the mapped three-dimensional coordinate system to the action recognition model 2013, the action recognition model 2013 receives the mapped three-dimensional coordinate system sent by the three-dimensional key point mapping model 2012 and analyzes it to obtain the driver action detection result. Compared with the prior art, the technical solution described in the present invention no longer relies solely on optical flow to extract information, thereby improving the readiness of the detection result.

[0072] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0073] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0074] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0075] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0076] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0077] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. Driver specific action detection method, including: Data acquisition end and data processing end, among which, The data acquisition end includes an acquisition module, and the acquisition module includes a camera acquisition unit; The data processing end includes a detection module, which includes a key point detection module, a three-dimensional key point mapping model and an action recognition model; S1, driver image acquisition, the camera acquisition unit acquires the image data of the driver during driving, pre-processes the acquired image data and sends it to the key point detection module; S2, obtaining the two-dimensional coordinates of the joint points. The key point detection module receives the image data sent by the camera acquisition unit, analyzes the image data in combination with the limb affinity module, obtains the two-dimensional coordinates of each joint point of the driver, and sends the two-dimensional coordinates of each joint point of the driver to the three-dimensional key point mapping model; The key point detection module and limb affinity module scale the image to 368x368 pixels, combine the annotation file, and input the key point network and affinity network for training. Among them, multiple convolution kernels (3x3, 5x5) are used for feature extraction, and the maximum pooling layer is used for downsampling to further extract high-level abstract image information. ReLu6 is used as the activation function to improve the network's fitting ability and speed up the network training. Group Normalization is further used for normalization, and finally the feature map is output. After extracting the image feature map, the key points of the human body and the connection lines of each limb are obtained after post-processing; S3, mapping three-dimensional coordinates, the three-dimensional key point mapping model receives the two-dimensional coordinates of each joint point of the driver sent by the key point detection module, maps the two-dimensional coordinates of each joint point in the image to the three-dimensional coordinate system under the driving position, and sends the mapped three-dimensional coordinate system to the action recognition model; After the key point detection module outputs the position of the key points of the human body, the two-dimensional key points on the image are mapped to the three-dimensional coordinate system of the space through the three-dimensional key point mapping model, and the input two-dimensional key points are output as the three-dimensional coordinates of the human body in the spatial coordinate system using a linear model; S4, action analysis, the action recognition model receives the mapped three-dimensional coordinate system sent by the three-dimensional key point mapping model and analyzes it to obtain the driver action detection result.

2. The method for detecting a specific driver action as claimed in claim 1, wherein: The preprocessing is to perform image normalization on the acquired images.

Images