A steering wheel control monitoring method, device, medium and equipment based on PYNQ-Z2

Abstract

The present application provides a steering wheel control monitoring method, device, medium and equipment based on PYNQ‑Z2, by obtaining first control data of the driver on the steering wheel; inputting the first control data into a monitoring model to obtain the driver's control status of the steering wheel; wherein the monitoring model is arranged on a PYNQ‑Z2 development board, the network structure of the monitoring model is arranged on the processor end, and multiple convolution layers and multiple pooling layers are respectively arranged on multiple programmable logic units of the PYNQ‑Z2 development board; that is, by arranging the network structure and the convolution layer and the pooling layer of the monitoring model on the processor end and the programmable logic end of the PYNQ‑Z2 development board respectively, using multiple programmable logic units to synchronously calculate to increase the calculation speed, thereby reducing the delay, and using multiple convolution layers and pooling layers on multiple programmable logic units to calculate the control data and comprehensively obtain the control status, thereby using the complementary advantages of multiple convolution layers and pooling layers to improve the monitoring accuracy.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and specifically to a steering wheel control monitoring method, device, medium and equipment based on PYNQ-Z2. Background Art

[0002] With the development of the automobile industry, intelligent and complex automotive electrical systems such as assisted driving systems have become popular. Based on the assisted driving function, a hands-off detection function has also been developed to match the assisted driving function. The driver's monitoring of the steering wheel is an indispensable part of the modern automobile safety system. The driver's loss of control of the steering wheel is one of the main causes of traffic accidents. An effective monitoring system can monitor the driver's behavior in real time, identify potential dangerous situations, and take measures to prevent accidents.

[0003] The current steering wheel control monitoring technology mainly includes sensor-based steering angle monitoring, torque monitoring, and driver steering wheel control detection based on acceleration and gyroscope sensors. These methods can reflect the driver's control of the steering wheel to a certain extent, but they are usually deployed on CPU and GPU general platforms. Since the computing power source in the car is single and mainly depends on the vehicle platform, when the computing power chip handles a large number of general needs, its delay and detection reliability are difficult to meet the needs. Therefore, there is an urgent need for a method that can quickly and accurately determine the driver's control of the steering wheel. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a steering wheel control monitoring method, device, medium and equipment based on PYNQ-Z2.

[0005] According to one aspect of the present application, a steering wheel control monitoring method based on PYNQ-Z2 is provided, including: obtaining first control data of a driver on a steering wheel; wherein the first control data represents operation data of the driver acting on the steering wheel; inputting the first control data into a monitoring model to obtain the driver's control status of the steering wheel; wherein the monitoring model is arranged on a PYNQ-Z2 development board, the PYNQ-Z2 development board includes a processor end and a programmable logic end, the programmable logic end includes a plurality of programmable logic units, the plurality of programmable logic units are respectively connected to the processor end, the monitoring model includes a plurality of convolutional layers and a plurality of pooling layers, the network structure of the monitoring model is arranged on the processor end, and the plurality of convolutional layers and the plurality of pooling layers are respectively arranged on the plurality of programmable logic units.

[0006] In one embodiment, inputting the first control data into a monitoring model to obtain the driver's control status of the steering wheel includes: based on the network structure of the monitoring model, using the multiple programmable logic units to calculate the first control data respectively to obtain the driver's control status of the steering wheel.

[0007] In one embodiment, the network structure based on the monitoring model uses the multiple programmable logic units to calculate the first control data respectively to obtain the driver's control status of the steering wheel, including: if the first control data includes multiple feature maps, the multiple programmable logic units are used to calculate the multiple feature maps respectively to obtain multiple first intermediate quantities; based on the multiple first intermediate quantities, the driver's control status of the steering wheel is calculated; if the first control data includes a single feature map, the multiple programmable logic units are used to calculate the single feature map respectively to obtain multiple second intermediate quantities; based on the multiple second intermediate quantities, the driver's control status of the steering wheel is calculated.

[0008] In one embodiment, before obtaining the driver's control data of the steering wheel, the steering wheel control monitoring method based on PYNQ-Z2 also includes: selecting the tester's second control data of the steering wheel and the corresponding labeled control status; preprocessing the second control data and the labeled control status to obtain training samples; and optimizing the model parameters of the monitoring model based on the training samples.

[0009] In one embodiment, the optimization of the model parameters of the monitoring model based on the training samples includes: initializing the initial model parameters of the monitoring model using a random normal distribution; based on the number of the multiple programmable logic units, simultaneously inputting multiple training samples into the monitoring model to train the monitoring model; wherein the learning rate of the monitoring model gradually decreases during the training process; evaluating the trained monitoring model to obtain an evaluation result; if the evaluation result is worse than a preset evaluation threshold, adjusting the model parameters or network structure of the monitoring model based on the evaluation result; if the evaluation result is better than the evaluation threshold, determining that the model parameters corresponding to the evaluation result are the optimized model parameters.

[0010] In one embodiment, the simultaneously inputting the plurality of training samples into the monitoring model to train the monitoring model includes: respectively inputting the plurality of training samples into the plurality of programmable logic units; respectively calculating the gradient value of each programmable logic unit; respectively calculating the contribution value of each programmable logic unit; wherein the contribution value represents the influence rate of the calculation result of the corresponding programmable logic unit on the output result of the monitoring model; and deleting the convolution layer and the corresponding pooling layer arranged on the programmable logic unit whose gradient value is less than a preset gradient threshold or whose contribution value is less than a preset contribution threshold.

[0011] In one embodiment, the preprocessing of the second control data and the labeled control state to obtain training samples includes: performing any one or more combinations of the following operations on the second control data and the labeled control state: size standardization processing, pixel value normalization processing, Gaussian filtering processing, image registration processing and adaptive gamma correction processing.

[0012] According to another aspect of the present application, a steering wheel control monitoring device based on PYNQ-Z2 is provided, including: a control data acquisition module, used to obtain first control data of the driver on the steering wheel; wherein the first control data represents the operation data of the driver acting on the steering wheel; a control state identification module, used to input the first control data into a monitoring model to obtain the control state of the driver on the steering wheel; wherein the monitoring model is arranged on a PYNQ-Z2 development board, the PYNQ-Z2 development board includes a processor end and a programmable logic end, the programmable logic end includes a plurality of programmable logic units, the plurality of programmable logic units are respectively connected to the processor end, the monitoring model includes a plurality of convolutional layers and a plurality of pooling layers, the network structure of the monitoring model is arranged on the processor end, and the plurality of convolutional layers and the plurality of pooling layers are respectively arranged on the plurality of programmable logic units.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute any of the above methods.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor is used to execute any of the above-described methods.

[0015] The present application provides a steering wheel control monitoring method, device, medium and equipment based on PYNQ-Z2, which obtains first control data of the driver on the steering wheel; wherein the first control data represents the operation data of the driver on the steering wheel; the first control data is input into a monitoring model to obtain the driver's control state of the steering wheel; wherein the monitoring model is arranged on a PYNQ-Z2 development board, the PYNQ-Z2 development board includes a processor end and a programmable logic end, the programmable logic end includes a plurality of programmable logic units, the plurality of programmable logic units are respectively connected to the processor end, and the monitoring model includes a plurality of programmable logic units. The network structure of the monitoring model is arranged on the processor end, and the multiple convolution layers and the multiple pooling layers are arranged on multiple programmable logic units respectively; that is, by arranging the network structure of the monitoring model and the convolution layers and the pooling layers on the processor end and the programmable logic end of the PYNQ-Z2 development board respectively, multiple programmable logic units are used to perform synchronous calculations to increase the calculation speed, thereby reducing latency, and multiple convolution layers and pooling layers on the multiple programmable logic units are used to calculate the control data and comprehensively obtain the control status, thereby utilizing the complementary advantages of multiple convolution layers and pooling layers to improve the monitoring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 It is a flowchart of a steering wheel control monitoring method based on PYNQ-Z2 provided by an exemplary embodiment of the present application.

[0018] Figure 2 It is a structural schematic diagram of a steering wheel control monitoring device based on PYNQ-Z2 provided by an exemplary embodiment of the present application.

[0019] Figure 3 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0021] Figure 1FIG. 1 is a flow chart of a steering wheel control monitoring method based on PYNQ-Z2 provided by an exemplary embodiment of the present application. Figure 1 As shown, the steering wheel control monitoring method based on PYNQ-Z2 includes the following steps:

[0022] Step 110: Obtain first steering wheel control data of the driver.

[0023] The first control data represents the operation data of the driver on the steering wheel. The present application acquires the first control data of the driver on the steering wheel in real time during the actual operation of the vehicle to obtain the actual control of the driver on the steering wheel.

[0024] Step 120: Input the first control data into the monitoring model to obtain the driver's control status of the steering wheel.

[0025] Among them, the monitoring model is arranged on the PYNQ-Z2 development board, the PYNQ-Z2 development board includes a processor end and a programmable logic end, the programmable logic end includes multiple programmable logic units, and the multiple programmable logic units are respectively connected to the processor end. The monitoring model includes multiple convolution layers and multiple pooling layers, and the network structure of the monitoring model is arranged on the processor end, and the multiple convolution layers and multiple pooling layers are respectively arranged on multiple programmable logic units. In this application, the first control data of the driver's steering wheel collected is input into the monitoring model to determine the actual control state of the driver's steering wheel. In addition, in this application, the monitoring model is arranged on the PYNQ-Z2 development board, the network structure is arranged on the processor end, that is, the network structure is arranged on the ARM processor of the PS end (Processing System), the convolution layer and the pooling layer are arranged on the programmable logic unit, that is, the convolution layer and the pooling layer are arranged on the FPGA (Field Programmable Gate Array) of the PL (Programmable Logic) end, and the programmable logic unit is used for parallel calculation to improve the calculation efficiency of the monitoring model, thereby reducing the recognition delay of the driver's control state of the steering wheel, which is conducive to the rapid response and driving safety of the intelligent driving system.

[0026] The present application provides a steering wheel control monitoring method based on PYNQ-Z2, by obtaining first control data of the driver on the steering wheel; wherein the first control data represents the operation data of the driver acting on the steering wheel; the first control data is input into a monitoring model to obtain the driver's control state of the steering wheel; wherein the monitoring model is arranged on a PYNQ-Z2 development board, the PYNQ-Z2 development board includes a processor end and a programmable logic end, the programmable logic end includes a plurality of programmable logic units, the plurality of programmable logic units are respectively connected to the processor end, the monitoring model includes a plurality of convolutional layers and a plurality of pooling layers, the network structure of the monitoring model is arranged on the processor end, and the plurality of convolutional layers and the plurality of pooling layers are respectively arranged on the plurality of programmable logic units; that is, by arranging the network structure and the convolutional layers and the pooling layers of the monitoring model on the processor end and the programmable logic end of the PYNQ-Z2 development board respectively, using a plurality of programmable logic units to perform synchronous calculations to increase the calculation speed, thereby reducing the delay, and using a plurality of convolutional layers and pooling layers on a plurality of programmable logic units to calculate the control data and obtain the control state comprehensively, thereby using the complementary advantages of a plurality of convolutional layers and pooling layers to improve the monitoring accuracy.

[0027] In one embodiment, the specific implementation of the above step 120 may be: based on the network structure of the monitoring model, a plurality of programmable logic units are used to calculate the first control data respectively to obtain the driver's control status of the steering wheel.

[0028] This application implements the integration of a deep learning processing unit (Deep Learning Processor Unit, referred to as DPU) in the PYNQ-Z2 development board, wherein the DPU is a programmable logic unit designed specifically for deep learning reasoning, which can accelerate the calculation process of neural networks at the hardware level, especially when processing deep learning operations such as convolution and pooling. It shows high efficiency. This application selects a DPU IP core suitable for the PYNQ-Z2 platform, and configures the parameters of the DPU IP core, such as processing power, interface type, and operating frequency. After completing the configuration of the DPU IP core and the system integration design, a bitstream file (.bit) is generated, and the generated bitstream file is loaded into the PYNQ-Z2 platform to realize the layout of the monitoring model. In addition, this application uses the DPU IP core with efficient parallel processing calculations to calculate a part of the first control data respectively, thereby realizing the effective utilization of hardware resources and improving the operating efficiency of the monitoring model.

[0029] This application deploys the driver's steering wheel control monitoring model to the PYNQ-Z2 platform by utilizing the API and tool library provided by the PYNQ platform, using the fast parallel processing characteristics of the FPGA to optimize the model's calculation and execution processes, and rationally utilizing the API and tool library to ensure that the model can be efficiently executed on the DPU while minimizing data transmission and processing delays. Finally, the performance test is performed by running the model on the PYNQ-Z2 platform, and the key indicators of the model in the driver's steering wheel control monitoring task, such as accuracy, processing speed, and resource consumption, are evaluated based on the test results.

[0030] In one embodiment, the specific implementation method of the above step 120 may be: if the first control data includes multiple feature maps, multiple programmable logic units are used to calculate the multiple feature maps respectively to obtain multiple first intermediate quantities; based on the multiple first intermediate quantities, the driver's control status of the steering wheel is calculated; if the first control data includes a single feature map, multiple programmable logic units are used to calculate the single feature map respectively to obtain multiple second intermediate quantities; based on the multiple second intermediate quantities, the driver's control status of the steering wheel is calculated.

[0031] This application compiles the monitoring model written in a high-level language into an optimized FPGA execution instruction set, that is, the DPU IP corresponding to the FPGA execution instruction command executes the corresponding computing operation. Among them, the compilation process takes into account the unique architecture and parallel processing capabilities of the FPGA, by using batch parallelism and convolution kernel parallelism. The compiled monitoring model can be directly deployed on the PYNQ-Z2 platform, taking advantage of the parallel processing advantages of the FPGA to achieve higher execution speed and lower power consumption.

[0032] The present application can process multiple input samples (multiple feature maps) at the same time, so that each programmable logic unit processes one or more feature maps, thereby obtaining multiple first intermediate quantities, and obtaining the driver's control status of the steering wheel based on a comprehensive calculation of the multiple first intermediate quantities, thereby effectively improving the monitoring efficiency.

[0033] For a single feature map, calculations of different convolution kernels can be performed in parallel through multiple programmable logic units to obtain multiple second intermediate quantities, and the driver's control status of the steering wheel can be obtained based on the comprehensive calculation of the multiple second intermediate quantities, thereby effectively improving the monitoring accuracy.

[0034] In one embodiment, before step 110, the above-mentioned steering wheel control monitoring method based on PYNQ-Z2 may also include: selecting the tester's second control data of the steering wheel and the corresponding labeled control status; preprocessing the second control data and the labeled control status to obtain training samples; and optimizing the model parameters of the monitoring model based on the training samples.

[0035] The present application obtains the measured second control data and the corresponding real labeled control status by collecting the tester's control data of the steering wheel and the corresponding labeled control status during the operation of the vehicle, and pre-processes the second control data and the corresponding labeled control status to obtain accurate control data and the corresponding real control status as training samples to train and optimize the model parameters of the monitoring model.

[0036] In one embodiment, a specific optimization method for the model parameters of the monitoring model may be: initializing the initial model parameters of the monitoring model using a random normal distribution; based on the number of multiple programmable logic units, simultaneously inputting multiple training samples into the monitoring model to train the monitoring model; wherein the learning rate of the monitoring model gradually decreases during the training process; evaluating the trained monitoring model to obtain an evaluation result; if the evaluation result is worse than a preset evaluation threshold, adjusting the model parameters or network structure of the monitoring model based on the evaluation result; if the evaluation result is better than the evaluation threshold, determining that the model parameters corresponding to the evaluation result are the optimized model parameters.

[0037] The present application first uses a random normal distribution to initialize the initial model parameters of the monitoring model to reduce the gradient vanishing or exploding problem in the initial training stage and promote the stable progress of the training process. Then the preprocessed training samples are input into the monitoring model in batches (i.e., multiple training samples are input at a time) for training. After training, the performance of the monitoring model is evaluated. If the evaluation result is poor, the model parameters or network structure are further optimized until the performance of the monitoring model meets the requirements. Among them, during the training process of the monitoring model, the present application can also adopt a learning rate decay strategy to avoid overfitting, while ensuring that the monitoring model can effectively converge to the optimal model parameters. Specifically, the present application can reduce the set initial learning rate in equal proportion or arithmetic progression to achieve fast iteration in the early stage and slow iteration in the later stage.

[0038] In one embodiment, a specific method for optimizing the model parameters of the monitoring model may be: inputting multiple training samples into multiple programmable logic units respectively; calculating the gradient value of each programmable logic unit respectively; calculating the contribution value of each programmable logic unit respectively; wherein the contribution value represents the influence rate of the calculation result of the corresponding programmable logic unit on the output result of the monitoring model; deleting the convolution layer and the corresponding pooling layer arranged on the programmable logic unit whose gradient value is less than a preset gradient threshold or whose contribution value is less than a preset contribution threshold.

[0039] The present application uses gradient descent and back propagation to perform iterative updates, and updates model parameters based on the training samples input in each batch. Preferably, the present application can be divided into multiple batches according to the type of training samples, wherein each batch contains all types of training samples, for example, each batch contains training samples under different categories, locations, sizes, weather conditions, time periods, etc., to ensure the diversity of training samples in each batch.

[0040] During the iteration process, this application calculates the gradient value of each programmable logic unit (corresponding convolution layer or pooling layer) and its contribution rate to the output result of the monitoring model, and deletes neurons (corresponding convolution layer or pooling layer) with small gradient values ​​and low contribution rates to achieve lightweight monitoring model, thereby further improving the computational efficiency of the monitoring model.

[0041] The present application can also use a validation set to verify the training results of the trained monitoring model, and further optimize the model parameters based on the verification results. If the verification results of multiple consecutive verification samples do not show obvious improvement, for example, the difference between the verification results of multiple consecutive verification samples and the target value exceeds the preset difference threshold, then the training is stopped in advance to retain the current model parameters, and the training samples are used for training again.

[0042] In one embodiment, the specific method of obtaining the training samples can be: performing any one or more combinations of the following operations on the second control data and the labeled control status: size standardization processing, pixel value normalization processing, Gaussian filtering processing, image registration processing and adaptive gamma correction processing.

[0043] Specifically, the present application can pre-process the second control data and the corresponding annotated control state by means of size standardization, pixel value normalization, Gaussian filtering, image registration, and adaptive gamma correction, etc., to provide rich and accurate training data for monitoring model training. In addition, the present application also takes into account the impact of light on monitoring, and the second control data selects data collected under weather conditions such as sunny, cloudy, rainy, and snowy days, as well as data collected during different time periods such as daytime, night, early morning, and dusk.

[0044] Figure 2 : is a schematic diagram of the structure of a steering wheel control monitoring device based on PYNQ-Z2 provided by an exemplary embodiment of the present application. Figure 2As shown, the steering wheel control monitoring device 20 based on PYNQ-Z2 includes: a control data acquisition module 21, used to obtain the first control data of the driver on the steering wheel; wherein the first control data represents the operation data of the driver acting on the steering wheel; a control state recognition module 22, used to input the first control data into the monitoring model to obtain the driver's control state of the steering wheel; wherein the monitoring model is arranged on the PYNQ-Z2 development board, the PYNQ-Z2 development board includes a processor end and a programmable logic end, the programmable logic end includes a plurality of programmable logic units, and the plurality of programmable logic units are respectively connected to the processor end, the monitoring model includes a plurality of convolutional layers and a plurality of pooling layers, the network structure of the monitoring model is arranged on the processor end, and the plurality of convolutional layers and the plurality of pooling layers are respectively arranged on the plurality of programmable logic units.

[0045] The present application provides a steering wheel control monitoring device based on PYNQ-Z2, which obtains the first control data of the driver on the steering wheel through the control data acquisition module 21; wherein the first control data represents the operation data of the driver on the steering wheel; the control state recognition module 22 inputs the first control data into the monitoring model to obtain the control state of the driver on the steering wheel; wherein the monitoring model is arranged on the PYNQ-Z2 development board, and the PYNQ-Z2 development board includes a processor end and a programmable logic end, and the programmable logic end includes a plurality of programmable logic units, and the plurality of programmable logic units are respectively connected to the processor end, and ... The model includes multiple convolution layers and multiple pooling layers, the network structure of the monitoring model is arranged on the processor end, and the multiple convolution layers and the multiple pooling layers are arranged on multiple programmable logic units respectively; that is, by arranging the network structure and convolution layers and the pooling layers of the monitoring model on the processor end and the programmable logic end of the PYNQ-Z2 development board respectively, multiple programmable logic units are used to perform synchronous calculations to increase the calculation speed, thereby reducing latency, and multiple convolution layers and pooling layers on multiple programmable logic units are used to calculate the control data and comprehensively obtain the control status, thereby utilizing the complementary advantages of multiple convolution layers and pooling layers to improve the monitoring accuracy.

[0046] In one embodiment, the control state recognition module 22 may be further configured as follows: based on the network structure of the monitoring model, a plurality of programmable logic units are used to calculate the first control data respectively to obtain the driver's control state of the steering wheel.

[0047] In one embodiment, the control state identification module 22 can be further configured as follows: if the first control data includes multiple feature maps, multiple programmable logic units are used to calculate the multiple feature maps respectively to obtain multiple first intermediate quantities; based on the multiple first intermediate quantities, the driver's control state of the steering wheel is calculated; if the first control data includes a single feature map, multiple programmable logic units are used to calculate the single feature map respectively to obtain multiple second intermediate quantities; based on the multiple second intermediate quantities, the driver's control state of the steering wheel is calculated.

[0048] In one embodiment, the above-mentioned PYNQ-Z2-based steering wheel control monitoring device 20 can be further configured to: select the tester's second control data of the steering wheel and the corresponding labeled control status; pre-process the second control data and the labeled control status to obtain training samples; and optimize the model parameters of the monitoring model based on the training samples.

[0049] In one embodiment, the above-mentioned steering wheel control monitoring device 20 based on PYNQ-Z2 can be further configured as: using random normal distribution to initialize the initial model parameters of the monitoring model; based on the number of multiple programmable logic units, inputting multiple training samples into the monitoring model at the same time to train the monitoring model; wherein the learning rate of the monitoring model gradually decreases during the training process; evaluating the trained monitoring model to obtain an evaluation result; if the evaluation result is worse than a preset evaluation threshold, adjusting the model parameters or network structure of the monitoring model based on the evaluation result; if the evaluation result is better than the evaluation threshold, determining that the model parameters corresponding to the evaluation result are the optimized model parameters.

[0050] In one embodiment, the above-mentioned steering wheel control monitoring device 20 based on PYNQ-Z2 can be further configured as: inputting multiple training samples into multiple programmable logic units respectively; calculating the gradient value of each programmable logic unit respectively; calculating the contribution value of each programmable logic unit respectively; wherein the contribution value represents the influence rate of the calculation result of the corresponding programmable logic unit on the output result of the monitoring model; deleting the convolution layer and the corresponding pooling layer arranged on the programmable logic unit whose gradient value is less than the preset gradient threshold or the contribution value is less than the preset contribution threshold.

[0051] In one embodiment, the above-mentioned PYNQ-Z2-based steering wheel control monitoring device 20 can be further configured to: perform any one or more combinations of the following operations on the second control data and the labeled control status: size standardization processing, pixel value normalization processing, Gaussian filtering processing, image registration processing and adaptive gamma correction processing.

[0052] Below, reference Figure 3The electronic device according to the embodiment of the present application is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signal from them.

[0053] Figure 3 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0054] like Figure 3 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0055] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0056] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the methods of the various embodiments of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0057] In one example, the electronic device 10 may further include: an input device 13 and an output device 14 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0058] When the electronic device is a stand-alone device, the input device 13 may be a communication network connector, which is used to receive the collected input signals from the first device and the second device.

[0059] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0060] The output device 14 can output various information to the outside, including the determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0061] Of course, to simplify, Figure 3Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0062] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0063] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0064] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0065] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0066] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0067] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0068] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0069] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0070] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A steering wheel control monitoring method based on PYNQ-Z2, characterized in that: include: Acquiring first control data of the driver on the steering wheel; wherein the first control data represents operation data of the driver acting on the steering wheel; Inputting the first control data into the monitoring model to obtain the driver's control status of the steering wheel; wherein the monitoring model is arranged on a PYNQ-Z2 development board, the PYNQ-Z2 development board includes a processor end and a programmable logic end, the programmable logic end includes a plurality of programmable logic units, the plurality of programmable logic units are respectively connected to the processor end, the monitoring model includes a plurality of convolutional layers and a plurality of pooling layers, the network structure of the monitoring model is arranged on the processor end, and the plurality of convolutional layers and the plurality of pooling layers are respectively arranged on the plurality of programmable logic units; The step of inputting the first control data into a monitoring model to obtain the driver's control status of the steering wheel includes: If the first control data includes a plurality of characteristic graphs, the plurality of programmable logic units are used to respectively calculate the plurality of characteristic graphs to obtain a plurality of first intermediate quantities; Based on the multiple first intermediate quantities, calculating the driver's control state of the steering wheel; If the first control data includes a single characteristic graph, the multiple programmable logic units are used to calculate the single characteristic graph respectively to obtain multiple second intermediate quantities; Based on the multiple second intermediate quantities, the driver's control status of the steering wheel is calculated.

2. The steering wheel control monitoring method based on PYNQ-Z2 according to claim 1 is characterized in that: The step of inputting the first control data into a monitoring model to obtain the driver's control status of the steering wheel includes: Based on the network structure of the monitoring model, the plurality of programmable logic units are used to respectively calculate the first control data to obtain the driver's control status of the steering wheel.

3. The steering wheel control monitoring method based on PYNQ-Z2 according to claim 1 is characterized in that: Before obtaining the driver's steering wheel control data, the steering wheel control monitoring method based on PYNQ-Z2 also includes: Select the tester's second control data of the steering wheel and the corresponding annotated control state; Preprocessing the second control data and the labeled control state to obtain training samples; The model parameters of the monitoring model are optimized based on the training samples.

4. The steering wheel control monitoring method based on PYNQ-Z2 according to claim 3 is characterized in that: The optimizing the model parameters of the monitoring model based on the training sample includes: Initializing the initial model parameters of the monitoring model using a random normal distribution; Based on the number of the plurality of programmable logic units, the plurality of training samples are simultaneously input into the monitoring model to train the monitoring model; wherein the learning rate of the monitoring model is gradually reduced during the training process; Evaluating the trained monitoring model to obtain an evaluation result; If the evaluation result is worse than a preset evaluation threshold, adjusting the model parameters or network structure of the monitoring model based on the evaluation result; If the evaluation result is better than the evaluation threshold, the model parameters corresponding to the evaluation result are determined to be optimized model parameters.

5. The steering wheel control monitoring method based on PYNQ-Z2 according to claim 4 is characterized in that: The step of simultaneously inputting a plurality of the training samples into the monitoring model to train the monitoring model comprises: Inputting the plurality of training samples into the plurality of programmable logic units respectively; Calculate the gradient value of each programmable logic unit respectively; Calculating the contribution value of each programmable logic unit respectively; wherein the contribution value represents the influence rate of the calculation result of the corresponding programmable logic unit on the output result of the monitoring model; Delete the convolution layer and the corresponding pooling layer arranged on the programmable logic unit whose gradient value is less than the preset gradient threshold or the contribution value is less than the preset contribution threshold.

6. The steering wheel control monitoring method based on PYNQ-Z2 according to claim 3 is characterized in that: The preprocessing of the second control data and the labeled control state to obtain a training sample comprises: The second control data and the annotated control state are subjected to any one or more combinations of the following operations: size standardization processing, pixel value normalization processing, Gaussian filtering processing, image registration processing, and adaptive gamma correction processing.

7. A steering wheel control monitoring device based on PYNQ-Z2, characterized in that: include: A control data acquisition module, used to acquire first control data of the driver on the steering wheel; wherein the first control data represents the operation data of the driver acting on the steering wheel; A control state identification module, used for inputting the first control data into a monitoring model to obtain the driver's control state of the steering wheel; wherein the monitoring model is arranged on a PYNQ-Z2 development board, the PYNQ-Z2 development board includes a processor end and a programmable logic end, the programmable logic end includes a plurality of programmable logic units, the plurality of programmable logic units are respectively connected to the processor end, the monitoring model includes a plurality of convolutional layers and a plurality of pooling layers, the network structure of the monitoring model is arranged on the processor end, and the plurality of convolutional layers and the plurality of pooling layers are respectively arranged on the plurality of programmable logic units; The control state recognition module is further configured as follows: If the first control data includes a plurality of characteristic graphs, the plurality of programmable logic units are used to respectively calculate the plurality of characteristic graphs to obtain a plurality of first intermediate quantities; Based on the multiple first intermediate quantities, calculating the driver's control state of the steering wheel; If the first control data includes a single characteristic graph, the multiple programmable logic units are used to calculate the single characteristic graph respectively to obtain multiple second intermediate quantities; Based on the multiple second intermediate quantities, the driver's control status of the steering wheel is calculated.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the method described in any one of claims 1 to 6.

Images