Intelligent vehicle-mounted terminal data acquisition system and method

Abstract

The invention relates to an intelligent vehicle-mounted terminal data acquisition system and method. The method comprises the following steps: acquiring a driving monitoring video of a to-be-analyzed driver in a predetermined time period, wherein the driving monitoring video is acquired by a camera; transmitting the driving monitoring video to a processor of the intelligent vehicle-mounted terminal; in the processor, performing driving behavior analysis and depth feature constraint fusion processing on the driving monitoring video to obtain a global constraint driving behavior time sequence feature map; in the processor, a detection result is determined based on the global constraint driving behavior time sequence feature map. Therefore, whether the risk of fatigue driving exists or not can be judged, and a fatigue warning is given out in time to remind a driver to take measures, so that traffic accidents are prevented.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent vehicle-mounted technology, and in particular, to an intelligent vehicle-mounted terminal data acquisition system and method. Background Art

[0002] During driving, fatigue driving is one of the important causes of traffic accidents. Specifically, fatigue driving refers to the decline of the driver's attention, reaction ability and judgment due to fatigue, lack of rest or insufficient sleep during long-term driving, thereby increasing the risk of traffic accidents. This state may lead to serious consequences, including casualties.

[0003] With the development of technology, smart in-vehicle terminals have become an important part of modern cars. These terminals can collect and process a large amount of data to provide various convenient functions such as navigation, entertainment, etc. However, driver fatigue detection remains an important and challenging problem. In order to avoid the risk of accidents caused by driver fatigue driving, an optimized solution is expected. Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a method for collecting data from an intelligent vehicle terminal, the method comprising:

[0006] Obtaining driving monitoring video of the driver to be analyzed within a predetermined time period collected by a camera;

[0007] Transmitting the driving monitoring video to a processor of an intelligent vehicle terminal;

[0008] In the processor, driving behavior analysis and deep feature constraint fusion processing are performed on the driving monitoring video to obtain a global constraint driving behavior time series feature graph;

[0009] In the processor, a detection result is determined based on the global restrictive driving behavior temporal characteristic diagram.

[0010] Optionally, in the processor, driving behavior analysis and deep feature constraint fusion processing are performed on the driving monitoring video to obtain a global constraint driving behavior time series feature graph, including: performing data preprocessing on the driving monitoring video to obtain a set of driving monitoring video clips; performing driving behavior time series feature extraction and feature constraint fusion on the set of driving monitoring video clips to obtain the global constraint driving behavior time series feature graph.

[0011] Optionally, performing data preprocessing on the driving monitoring video to obtain a set of driving monitoring video segments includes: performing video segmentation on the driving monitoring video to obtain the set of driving monitoring video segments.

[0012] Optionally, the set of driving monitoring video clips is subjected to driving behavior temporal feature extraction and feature constraint fusion to obtain the global constrained driving behavior temporal feature graph, including: using a deep learning network model to capture the spatiotemporal features of driving behavior on the set of driving monitoring video clips to obtain a sequence of local temporal feature graphs of driving behavior; and passing the sequence of local temporal feature graphs of driving behavior through a deep feature constraint fusion module based on an autocorrelation attention mechanism to obtain the global constrained driving behavior temporal feature graph.

[0013] Optionally, a deep learning network model is used to capture the spatiotemporal features of driving behavior on the set of driving monitoring video clips to obtain a sequence of local temporal feature graphs of driving behavior, including: passing the set of driving monitoring video clips through a driving behavior temporal feature capturer based on a three-dimensional convolutional neural network model to obtain a sequence of local temporal feature graphs of driving behavior.

[0014] Optionally, the sequence of the local temporal feature graphs of the driving behavior is passed through a deep feature constraint fusion module based on an autocorrelation attention mechanism to obtain the global constrained driving behavior temporal feature graph, including: processing the sequence of the local temporal feature graphs of the driving behavior with the following deep feature constraint fusion formula to obtain the global constrained driving behavior temporal feature graph; wherein the deep feature constraint fusion formula is:

[0015]

[0016] F=[a1f1;a2f2;···;f i f i ;···;a t f t ]

[0017] Among them, h i is the global mean pooling feature vector of the ith driving behavior local temporal feature graph in the sequence of the driving behavior local temporal feature graphs, W iis the i-th weight coefficient matrix, is the i-th weight coefficient vector, B i is the i-th bias vector, tanh(·) represents the hyperbolic tangent function processing, e i is the i-th attention scoring coefficient, e k is the kth attention scoring coefficient, exp(·) represents the exponential function with base e, a i is the i-th attention weight coefficient, a t is the tth attention weight coefficient, t is the length of the sequence of the local temporal feature graph of the driving behavior, f i is the local temporal feature graph of the i-th driving behavior, f t is the t-th local time series feature graph of driving behavior, Concat[·;·] represents the cascade function, and F is the global constraint driving behavior time series feature graph.

[0018] Optionally, in the processor, the detection result is determined based on the global restrictive driving behavior timing characteristic diagram, including: using a nonlinear response compensation module to process the global restrictive driving behavior timing characteristic diagram to obtain a corrected global restrictive driving behavior timing characteristic diagram; optimizing the feature distribution of the corrected global restrictive driving behavior timing characteristic diagram to obtain an optimized corrected global restrictive driving behavior timing characteristic diagram; passing the optimized corrected global restrictive driving behavior timing characteristic diagram through a classifier-based driving behavior detector to obtain the detection result, and the detection result is used to indicate whether a fatigue warning is issued.

[0019] Optionally, using a nonlinear response compensation module to process the global restrictive driving behavior time series characteristic diagram to obtain a corrected global restrictive driving behavior time series characteristic diagram, comprising: processing the global restrictive driving behavior time series characteristic diagram using the following nonlinear response compensation formula to obtain the corrected global restrictive driving behavior time series characteristic diagram; wherein the nonlinear response compensation formula is:

[0020]

[0021] Among them, L 0i is the pixel value of each position of the global constraint driving behavior time series feature map, A, B, C and D are adjustment parameters with different values, L 1i is the pixel value of each position of the corrected global constraint driving behavior time series feature map.

[0022] Optionally, the optimized corrected global restrictive driving behavior time-series feature graph is passed through a classifier-based driving behavior detector to obtain the detection result, and the detection result is used to indicate whether a fatigue warning is issued, including: expanding the optimized corrected global restrictive driving behavior time-series feature graph into a classification feature vector according to a row vector or a column vector; using multiple fully connected layers of the classifier to fully connect encode the classification feature vector to obtain an encoded classification feature vector; and, passing the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0023] In a second aspect, the present disclosure provides an intelligent vehicle terminal data acquisition system, the system comprising:

[0024] A driving monitoring video acquisition module is used to acquire driving monitoring videos of the driver to be analyzed within a predetermined time period collected by a camera;

[0025] A monitoring video transmission module, used for transmitting the driving monitoring video to a processor of an intelligent vehicle terminal;

[0026] A constraint fusion processing module, used for performing driving behavior analysis and deep feature constraint fusion processing on the driving monitoring video in the processor to obtain a global constraint driving behavior time series feature graph;

[0027] A detection result determination module is used to determine the detection result in the processor based on the global restrictive driving behavior time series characteristic diagram.

[0028] The above technical solution is adopted to obtain the driving monitoring video of the driver to be analyzed collected by the camera within a predetermined time period; the driving monitoring video is transmitted to the processor of the intelligent vehicle terminal; in the processor, the driving monitoring video is subjected to driving behavior analysis and deep feature constraint fusion processing to obtain a global constraint driving behavior time series feature graph; in the processor, the detection result is determined based on the global constraint driving behavior time series feature graph. In this way, it is possible to determine whether there is a risk of fatigue driving, and issue a fatigue warning in time to remind the driver to take measures to prevent traffic accidents.

[0029] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0031] Figure 1 The present invention is a flow chart of a method for collecting data of an intelligent vehicle terminal according to an exemplary embodiment.

[0032] Figure 2 The figure is a block diagram of an intelligent vehicle terminal data acquisition system according to an exemplary embodiment.

[0033] Figure 3 It is a block diagram of an electronic device according to an exemplary embodiment.

[0034] Figure 4 It is an application scenario diagram of a smart vehicle terminal data collection method according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0036] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0037] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0038] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0041] Fatigue driving refers to a state in which the driver's attention, reaction ability and judgment are reduced due to fatigue, lack of rest or insufficient sleep during long-term driving, thereby increasing the risk of traffic accidents. This state may lead to serious consequences, including casualties.

[0042] Fatigue driving can seriously impair the driver's ability, leading to: prolonged reaction time, increased driving errors, decreased vehicle control, impaired judgment, and decreased cognitive function.

[0043] These effects can significantly increase the risk of a traffic accident, including: collision, rollover, running off the road, pedestrian or cyclist injury. Drowsy driving is a serious risk behavior that significantly increases the risk of a traffic accident. By taking precautions and paying attention to body signals, drivers can reduce the risk of drowsy driving and ensure the safety of themselves and others.

[0044] Intelligent in-vehicle terminals have become an important part of modern cars, which can collect and process a large amount of data to provide various convenient functions such as navigation, entertainment, etc. However, driver fatigue detection remains an important and challenging problem.

[0045] Smart in-vehicle terminals can detect signs of driver fatigue through various sensors and algorithms. These sensors can monitor the driver's behavior and physiological data, such as: drooping eyelids, reduced blinking frequency; nodding, shaking left and right; weakened grip, unstable steering; heart rate, brain waves.

[0046] In order to avoid the risk of accidents caused by driver fatigue, an optimized driver fatigue detection solution is needed. This solution can accurately detect signs of fatigue and minimize false alarms. It can monitor the driver's status in real time and issue an alarm immediately when signs of fatigue are detected. It does not interfere with the driver's driving and is as comfortable as possible.

[0047] By optimizing driver fatigue detection in smart vehicle terminals, the risk of accidents caused by fatigue driving can be greatly reduced. Through accurate and reliable real-time detection, non-intrusive operation and cost-effectiveness, driver safety can be ensured and lives can be saved.

[0048] First, in response to the above technical problems, the technical concept of this application is to use the data collection capabilities of the intelligent vehicle terminal, combined with the deep learning algorithm to achieve real-time monitoring and detection of the driver's fatigue state. That is, by analyzing the driving behavior characteristics of the driver in the driving monitoring video, capturing the driver's behavior pattern, so as to determine whether there is a risk of fatigue driving, and timely issue a fatigue warning to remind the driver to take measures to prevent traffic accidents.

[0049] In order to solve the above problems, the present disclosure provides a smart vehicle terminal data acquisition system and method, which obtains the driving monitoring video of the driver to be analyzed within a predetermined time period collected by the camera; transmits the driving monitoring video to the processor of the smart vehicle terminal; in the processor, the driving monitoring video is subjected to driving behavior analysis and deep feature constraint fusion processing to obtain a global constraint driving behavior time series feature graph; in the processor, the detection result is determined based on the global constraint driving behavior time series feature graph. In this way, it is possible to determine whether there is a risk of fatigue driving, and issue a fatigue warning in time to remind the driver to take measures to prevent traffic accidents.

[0050] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0051] Figure 1 is a flow chart of a method for collecting data from an intelligent vehicle terminal according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0052] Step 101: Acquire a driving monitoring video of the driver to be analyzed within a predetermined time period collected by a camera;

[0053] Step 102: transmitting the driving monitoring video to a processor of an intelligent vehicle terminal;

[0054] Step 103: In the processor, the driving monitoring video is subjected to driving behavior analysis and deep feature constraint fusion processing to obtain a global constraint driving behavior time series feature graph;

[0055] Step 104: In the processor, determine a detection result based on the global restrictive driving behavior temporal characteristic diagram.

[0056] Based on this, in the technical solution of the present application, first, a driving monitoring video of the driver to be analyzed within a predetermined time period collected by a camera is obtained. Here, the driving monitoring video can provide rich information, such as the driver's eye activities, head postures, hand movements, etc., which can be used to analyze the driver's behavioral characteristics. Fatigue driving is usually manifested in these behavioral characteristics, such as reduced blinking frequency, frequent head nodding, yawning, etc. After that, the driving monitoring video is transmitted to the processor of the intelligent vehicle terminal. Among them, in the intelligent vehicle terminal data acquisition system, the processor refers to a central processing unit for processing data and performing computing tasks. The processor is usually an integrated circuit chip that is responsible for executing various algorithms, logics and operations to achieve the functions and goals of the system. As an important part of the intelligent vehicle terminal data acquisition system, the processor plays a key role. For example, the processor is responsible for receiving, processing and analyzing data collected from various sensors (such as cameras). It executes various algorithms and models to extract useful information from the raw data, such as driving behavior characteristics, signs of fatigue driving, etc. At the same time, the processor can execute complex algorithms such as deep learning models and neural networks to identify patterns of driving behavior. These models require a lot of computing resources and processing power, which processors can provide.

[0057] In one embodiment of the present disclosure, in the processor, driving behavior analysis and deep feature constraint fusion processing are performed on the driving monitoring video to obtain a global constraint driving behavior time series feature graph, including: data preprocessing of the driving monitoring video to obtain a set of driving monitoring video clips; driving behavior time series feature extraction and feature constraint fusion are performed on the set of driving monitoring video clips to obtain the global constraint driving behavior time series feature graph.

[0058] Then, in the processor, the driving monitoring video is segmented to obtain a set of driving monitoring video clips. In this way, dividing a long driving monitoring video into small video clips helps to improve the efficiency of data processing and analysis. The system can process each small video clip in parallel to speed up data processing. At the same time, it improves the attention to subtle changes in the driving monitoring video.

[0059] In one embodiment of the present disclosure, data preprocessing is performed on the driving monitoring video to obtain a set of driving monitoring video segments, including: video segmentation is performed on the driving monitoring video to obtain the set of driving monitoring video segments.

[0060] In one embodiment of the present disclosure, the driving behavior temporal feature extraction and feature constraint fusion are performed on the set of driving monitoring video clips to obtain the global constraint driving behavior temporal feature graph, including: using a deep learning network model to capture the driving behavior spatiotemporal features of the set of driving monitoring video clips to obtain a sequence of local temporal feature graphs of driving behavior; and passing the sequence of local temporal feature graphs of driving behavior through a deep feature constraint fusion module based on an autocorrelation attention mechanism to obtain the global constraint driving behavior temporal feature graph.

[0061] Next, in the processor, the set of driving monitoring video clips is passed through a driving behavior temporal feature capturer based on a three-dimensional convolutional neural network model to obtain a sequence of local temporal feature maps of driving behavior. Here, a three-dimensional convolutional neural network (3D CNN) is a neural network architecture for processing video data. Unlike traditional two-dimensional convolutional neural networks (2D CNN), 3D CNN also has convolution operations in the time dimension and can directly process the spatiotemporal information of video data. By performing convolution operations in dimensions such as time, height, and width, 3D CNN can effectively capture the temporal features in the video. Among them, the driving monitoring video contains the changes in the driver's behavior over time. Through 3D CNN, the temporal features in the video can be effectively captured to identify different behavior patterns of the driver, such as yawning, frequent eye closure, etc.

[0062] In one embodiment of the present disclosure, a deep learning network model is used to capture the spatiotemporal features of driving behavior on the set of driving monitoring video clips to obtain a sequence of local temporal feature graphs of driving behavior, including: passing the set of driving monitoring video clips through a driving behavior temporal feature capturer based on a three-dimensional convolutional neural network model to obtain a sequence of local temporal feature graphs of driving behavior.

[0063] Considering that although the sequence of the local temporal feature graphs of driving behavior can characterize the behavioral characteristics of the driver in each driving monitoring video clip, there is a lack of interaction and communication between the driving behavior characteristics of each driving monitoring video clip. This will lose the overall temporal driving feature distribution and behavior pattern. In the technical solution of the present application, in the processor, the sequence of the local temporal feature graphs of driving behavior is passed through a deep feature constraint fusion module based on the autocorrelation attention mechanism to obtain a global constrained driving behavior temporal feature graph. In this way, by introducing the autocorrelation attention mechanism, it can help the model better capture the correlation between different local features, thereby integrating local information, obtaining a more globally constrained driving behavior temporal feature distribution, and enhancing the model's ability to characterize the driving behavior temporal features.

[0064] In one embodiment of the present disclosure, the sequence of the local temporal feature graphs of the driving behavior is passed through a deep feature constraint fusion module based on an autocorrelation attention mechanism to obtain the global constrained driving behavior temporal feature graph, including: processing the sequence of the local temporal feature graphs of the driving behavior with the following deep feature constraint fusion formula to obtain the global constrained driving behavior temporal feature graph; wherein the deep feature constraint fusion formula is:

[0065]

[0066] F=[a1f1;a2f2;···;a i f i ;···;a t f t ]

[0067] Among them, h i is the global mean pooling feature vector of the ith driving behavior local temporal feature graph in the sequence of the driving behavior local temporal feature graphs, W i is the i-th weight coefficient matrix, is the i-th weight coefficient vector, B i is the i-th bias vector, tanh(·) represents the hyperbolic tangent function processing, e i is the i-th attention scoring coefficient, e k is the kth attention scoring coefficient, exp(·) represents the exponential function with base e, a i is the i-th attention weight coefficient, a t is the tth attention weight coefficient, t is the length of the sequence of the local temporal feature graph of the driving behavior, f i is the local temporal feature graph of the i-th driving behavior, f t is the t-th local time series feature graph of driving behavior, Concat[·;·] represents the cascade function, and F is the global constraint driving behavior time series feature graph.

[0068] Considering that the driving behavior information expressed in the global constraint driving behavior time series characteristic diagram may have a nonlinear response relationship for the judgment of fatigue driving. For example, different degrees of change in blinking frequency may have different importance for judging whether the driver is tired. However, this nonlinear response relationship may be easily overlooked or difficult to reflect in the process of feature extraction, that is, in the process of feature extraction, the nonlinear relationship and importance difference between the driver's fatigue driving and driving behavior pattern may not be fully considered. This may lead to the neglect of important features and over-emphasis on minor features. Therefore, in the technical solution of the present application, it is expected that in the processor, a nonlinear response compensation module is used to process the global constraint driving behavior time series characteristic diagram to compensate and correct the above-mentioned nonlinear response relationship, thereby obtaining a corrected global constraint driving behavior time series characteristic diagram.

[0069] In one embodiment of the present disclosure, in the processor, a detection result is determined based on the global constraint driving behavior timing characteristic diagram, including: using a nonlinear response compensation module to process the global constraint driving behavior timing characteristic diagram to obtain a corrected global constraint driving behavior timing characteristic diagram; optimizing the feature distribution of the corrected global constraint driving behavior timing characteristic diagram to obtain an optimized corrected global constraint driving behavior timing characteristic diagram; passing the optimized corrected global constraint driving behavior timing characteristic diagram through a classifier-based driving behavior detector to obtain the detection result, and the detection result is used to indicate whether a fatigue warning is issued.

[0070] Further, the global constraint driving behavior time series characteristic diagram is processed using a nonlinear response compensation module to obtain a corrected global constraint driving behavior time series characteristic diagram, including: processing the global constraint driving behavior time series characteristic diagram using the following nonlinear response compensation formula to obtain the corrected global constraint driving behavior time series characteristic diagram; wherein the nonlinear response compensation formula is:

[0071]

[0072] Among them, L 0i is the pixel value of each position of the global constraint driving behavior time series feature map, A, B, C and D are adjustment parameters with different values, L 1i is the pixel value of each position of the corrected global constraint driving behavior time series feature map.

[0073] In the technical solution of the present application, each driving behavior local temporal feature map in the sequence of the driving behavior local temporal feature maps respectively expresses the image semantic features of the temporal association in the local time domain determined by video segmentation in the global time domain of the driving monitoring video. Therefore, after the sequence of the driving behavior local temporal feature maps is subjected to the autocorrelation attention mechanism, the overall feature distribution in the local time domain can be strengthened based on the overall spatiotemporal semantic distribution of the temporal association image semantic features in the local time domain, which will increase the overall temporal association image semantic feature distribution difference between each local time domain, and considering the correction of the local nonlinear feature response by the nonlinear response compensation module, the main trend of the spatiotemporal semantic distribution in each local time domain will be further strengthened, thereby further increasing the local feature distribution difference based on the spatial distribution representation of the local time domain feature matrix under the channel dimension of the corrected global constraint driving behavior temporal feature map, so that the overall expression effect of the corrected global constraint driving behavior temporal feature map will be affected, and the accuracy of the classification result obtained by the classifier will be affected.

[0074] Based on this, the present application optimizes the characteristic distribution of the corrected global restrictive driving behavior time series characteristic diagram to obtain an optimized corrected global restrictive driving behavior time series characteristic diagram, including the following steps:

[0075] The absolute value sum a1 of each characteristic value of the corrected global restrictive driving behavior time series characteristic diagram and the square root a2 of the square sum are calculated, that is:

[0076]

[0077] Among them, f i represents the eigenvalue of the ith position of the corrected global restrictive driving behavior time-series characteristic graph, S represents the characteristic graph scale of the corrected global restrictive driving behavior time-series characteristic graph, a1 represents the sum of the absolute values ​​of each eigenvalue of the corrected global restrictive driving behavior time-series characteristic graph, and a2 represents the square root of the sum of the squares of each eigenvalue of the corrected global restrictive driving behavior time-series characteristic graph.

[0078] After subtracting a2 from each characteristic value of the corrected global restrictive driving behavior time series characteristic graph, take the absolute value and multiply it by the characteristic graph scale S and the inverse of a1 of the corrected global restrictive driving behavior time series characteristic graph, and then take the inverse of the product to obtain the first corrected global restrictive driving behavior time series intermediate characteristic graph, and after subtracting a1 from each characteristic value of the corrected global restrictive driving behavior time series characteristic graph, take the absolute value and multiply it by the square root of the characteristic graph scale S and the inverse of a1, and then take the inverse of the product to obtain the second corrected global restrictive driving behavior time series intermediate characteristic graph, that is:

[0079] f 1i =[|f i -a2|×S×(a1 -1 )] -1

[0080]

[0081] Among them, f i represents the characteristic value of the ith position of the corrected global restrictive driving behavior time series characteristic graph, S represents the characteristic graph scale of the corrected global restrictive driving behavior time series characteristic graph, a1 represents the sum of the absolute values ​​of each characteristic value of the corrected global restrictive driving behavior time series characteristic graph, a2 represents the square root of the sum of the squares of each characteristic value of the corrected global restrictive driving behavior time series characteristic graph, f 1i represents the feature value of the ith position of the intermediate feature map of the global constraint driving behavior time series after the first correction, f 2i Represents the feature value of the i-th position of the intermediate feature map of the global constraint driving behavior time series after the second correction.

[0082] Finally, the optimized corrected global restrictive driving behavior temporal feature graph is obtained by subtracting the dot product of the second corrected global restrictive driving behavior temporal intermediate feature graph and the weighted hyperparameter from the first corrected global restrictive driving behavior temporal intermediate feature graph.

[0083] Therefore, the difference of the characteristic value of the corrected global restrictive driving behavior time series feature map relative to the different and modulated representations of the overall feature set of the corrected global restrictive driving behavior time series feature map is used as the semantic change intensity information, and a phase-like transformation corresponding to the position-based intensity modulation is performed through different and modulated representations, so that the aggregation enhancement of the semantic change phase perception can improve the axial aggregation receptive field along the feature aggregation direction by performing a spatial translation operation based on alternating stacking under the scale balance of the feature set of the corrected global restrictive driving behavior time series feature map, thereby improving the perception effect of the aggregated semantics of the corrected global restrictive driving behavior time series feature map for the detailed semantic changes, so as to improve the overall expression effect of the corrected global restrictive driving behavior time series feature map, and improve the accuracy of the classification result obtained by the classifier.

[0084] Furthermore, in the processor, the optimized corrected global restrictive driving behavior time series characteristic diagram is passed through a classifier-based driving behavior detector to obtain a detection result, and the detection result is used to indicate whether a fatigue warning is issued.

[0085] In one embodiment of the present disclosure, the optimized corrected global restrictive driving behavior time-series feature graph is passed through a classifier-based driving behavior detector to obtain the detection result, and the detection result is used to indicate whether a fatigue warning is issued, including: expanding the optimized corrected global restrictive driving behavior time-series feature graph into a classification feature vector according to a row vector or a column vector; using multiple fully connected layers of the classifier to fully connect encode the classification feature vector to obtain an encoded classification feature vector; and, passing the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0086] In summary, the above solution can realize real-time monitoring and detection of driver fatigue status by utilizing the data collection capability of the intelligent vehicle terminal and combining it with the deep learning algorithm. That is, by analyzing the driving behavior characteristics of the driver in the driving monitoring video and capturing the driver's behavior pattern, it is possible to determine whether there is a risk of fatigue driving, and issue a fatigue warning in time to remind the driver to take measures to prevent traffic accidents.

[0087] Figure 2 FIG. 1 is a block diagram of an intelligent vehicle terminal data acquisition system according to an exemplary embodiment. Figure 2 As shown, the intelligent vehicle terminal data acquisition system 200 includes:

[0088] The driving monitoring video acquisition module 201 is used to acquire the driving monitoring video of the driver to be analyzed within a predetermined time period collected by the camera;

[0089] A monitoring video transmission module 202, used to transmit the driving monitoring video to a processor of an intelligent vehicle terminal;

[0090] A constraint fusion processing module 203 is used to perform driving behavior analysis and deep feature constraint fusion processing on the driving monitoring video in the processor to obtain a global constraint driving behavior time series feature graph;

[0091] The detection result determination module 204 is used to determine the detection result in the processor based on the global restrictive driving behavior time series characteristic diagram.

[0092] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0093] like Figure 3 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0094] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0095] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0096] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0097] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0098] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0099] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, a test parameter acquisition module may also be described as a "module for acquiring device test parameters corresponding to a target device".

[0102] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0103] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] According to one or more embodiments of the present disclosure, Example 1 provides a method for collecting data from an intelligent vehicle terminal, the method comprising:

[0105] Obtaining driving monitoring video of the driver to be analyzed within a predetermined time period collected by a camera;

[0106] Transmitting the driving monitoring video to a processor of an intelligent vehicle terminal;

[0107] In the processor, driving behavior analysis and deep feature constraint fusion processing are performed on the driving monitoring video to obtain a global constraint driving behavior time series feature graph;

[0108] In the processor, a detection result is determined based on the global restrictive driving behavior temporal characteristic diagram.

[0109] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein in the processor, driving behavior analysis and deep feature constraint fusion processing are performed on the driving monitoring video to obtain a global constraint driving behavior time series feature graph, including:

[0110] Performing data preprocessing on the driving monitoring video to obtain a set of driving monitoring video clips;

[0111] The driving behavior temporal feature extraction and feature constraint fusion are performed on the set of driving monitoring video clips to obtain the global constraint driving behavior temporal feature graph.

[0112] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, performing data preprocessing on the driving monitoring video to obtain a set of driving monitoring video clips, including:

[0113] The driving monitoring video is segmented to obtain a set of driving monitoring video segments.

[0114] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 3, performing driving behavior temporal feature extraction and feature constraint fusion on the set of driving monitoring video clips to obtain the global constraint driving behavior temporal feature graph, including:

[0115] Using a deep learning network model to capture the spatiotemporal features of driving behavior on the set of driving monitoring video clips to obtain a sequence of local temporal feature graphs of driving behavior;

[0116] The sequence of the local temporal feature graphs of the driving behavior is passed through a deep feature constraint fusion module based on an autocorrelation attention mechanism to obtain the global constrained driving behavior temporal feature graph.

[0117] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 4, using a deep learning network model to capture the spatiotemporal features of driving behavior on the set of driving monitoring video clips to obtain a sequence of local temporal feature graphs of driving behavior, including:

[0118] The collection of driving monitoring video clips is passed through a driving behavior temporal feature capturer based on a three-dimensional convolutional neural network model to obtain a sequence of local temporal feature graphs of the driving behavior.

[0119] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 5, wherein the sequence of the local temporal feature graphs of the driving behavior is passed through a deep feature constraint fusion module based on an autocorrelation attention mechanism to obtain the global constrained driving behavior temporal feature graph, including:

[0120] The sequence of the local time series feature graphs of the driving behavior is processed by the following deep feature constraint fusion formula to obtain the global constraint driving behavior time series feature graph; wherein the deep feature constraint fusion formula is:

[0121]

[0122] F=[a1f1;a2f2;···;a i f i ;···;a t f t ]

[0123] Among them, h i is the global mean pooling feature vector of the ith driving behavior local temporal feature graph in the sequence of the driving behavior local temporal feature graphs, W i is the i-th weight coefficient matrix, is the i-th weight coefficient vector, Bi is the i-th bias vector, tanh(·) represents the hyperbolic tangent function processing, e i is the i-th attention scoring coefficient, e k is the kth attention scoring coefficient, exp(·) represents the exponential function with base e, a i is the i-th attention weight coefficient, a t is the tth attention weight coefficient, t is the length of the sequence of the local temporal feature graph of the driving behavior, f i is the local temporal feature graph of the i-th driving behavior, f t is the t-th local time series feature graph of driving behavior, Concat[·;·] represents the cascade function, and F is the global constraint driving behavior time series feature graph.

[0124] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 6, wherein in the processor, determining a detection result based on the global restrictive driving behavior time series characteristic diagram includes:

[0125] Using a nonlinear response compensation module to process the global restrictive driving behavior time series characteristic diagram to obtain a corrected global restrictive driving behavior time series characteristic diagram;

[0126] Performing feature distribution optimization on the corrected global restrictive driving behavior time series feature graph to obtain an optimized corrected global restrictive driving behavior time series feature graph;

[0127] The optimized corrected global constraint driving behavior time series characteristic diagram is passed through a classifier-based driving behavior detector to obtain the detection result, and the detection result is used to indicate whether a fatigue warning is issued.

[0128] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 7, using a nonlinear response compensation module to process the global restrictive driving behavior time series characteristic diagram to obtain a corrected global restrictive driving behavior time series characteristic diagram, including:

[0129] The global restrictive driving behavior time series characteristic diagram is processed using the following nonlinear response compensation formula to obtain the corrected global restrictive driving behavior time series characteristic diagram; wherein the nonlinear response compensation formula is:

[0130]

[0131] Among them, L 0i is the pixel value of each position of the global constraint driving behavior time series feature map, A, B, C and D are adjustment parameters with different values, L 1iis the pixel value of each position of the corrected global constraint driving behavior time series feature map.

[0132] According to one or more embodiments of the present disclosure, Example 9 provides the method of Example 8, wherein the optimized corrected global constraint driving behavior time series feature graph is passed through a classifier-based driving behavior detector to obtain the detection result, and the detection result is used to indicate whether a fatigue warning is issued, including:

[0133] Expanding the optimized corrected global constraint driving behavior time series feature graph into a classification feature vector according to a row vector or a column vector;

[0134] Performing full connection encoding on the classification feature vector using multiple fully connected layers of the classifier to obtain an encoded classification feature vector; and

[0135] The encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

[0136] According to one or more embodiments of the present disclosure, Example 10 provides an intelligent vehicle terminal data acquisition system, the system comprising:

[0137] A driving monitoring video acquisition module is used to acquire driving monitoring videos of the driver to be analyzed within a predetermined time period collected by a camera;

[0138] A monitoring video transmission module, used for transmitting the driving monitoring video to a processor of an intelligent vehicle terminal;

[0139] A constraint fusion processing module, used for performing driving behavior analysis and deep feature constraint fusion processing on the driving monitoring video in the processor to obtain a global constraint driving behavior time series feature graph;

[0140] A detection result determination module is used to determine the detection result in the processor based on the global restrictive driving behavior time series characteristic diagram.

[0141] Figure 4 FIG. 1 is an application scenario diagram of a method for collecting data from an intelligent vehicle terminal according to an exemplary embodiment. Figure 4 As shown, in this application scenario, first, a driving monitoring video of the driver to be analyzed within a predetermined time period collected by a camera is obtained (for example, Figure 4 Then, the acquired driving monitoring video is input to a server (for example, Figure 4 In S) shown in , the server is capable of processing the driving monitoring video based on the intelligent vehicle terminal data acquisition algorithm to determine the detection result based on the global constraint driving behavior time series characteristic diagram.

[0142] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0143] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0144] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A method for collecting data from an intelligent vehicle terminal, characterized in that: include: Obtaining driving monitoring video of the driver to be analyzed within a predetermined time period collected by a camera; Transmitting the driving monitoring video to a processor of an intelligent vehicle terminal; In the processor, driving behavior analysis and deep feature constraint fusion processing are performed on the driving monitoring video to obtain a global constraint driving behavior time series feature graph; In the processor, a detection result is determined based on the global restrictive driving behavior temporal characteristic diagram.

2. The intelligent vehicle terminal data collection method according to claim 1, characterized in that: In the processor, the driving monitoring video is subjected to driving behavior analysis and deep feature constraint fusion processing to obtain a global constraint driving behavior time series feature graph, including: Performing data preprocessing on the driving monitoring video to obtain a set of driving monitoring video clips; The driving behavior temporal feature extraction and feature constraint fusion are performed on the set of driving monitoring video clips to obtain the global constraint driving behavior temporal feature graph.

3. The intelligent vehicle terminal data collection method according to claim 2, characterized in that: Performing data preprocessing on the driving monitoring video to obtain a set of driving monitoring video clips includes: The driving monitoring video is segmented to obtain a set of driving monitoring video segments.

4. The intelligent vehicle terminal data collection method according to claim 3 is characterized in that: The driving behavior temporal feature extraction and feature constraint fusion are performed on the set of driving monitoring video clips to obtain the global constraint driving behavior temporal feature graph, including: Using a deep learning network model to capture the spatiotemporal features of driving behavior on the set of driving monitoring video clips to obtain a sequence of local temporal feature graphs of driving behavior; The sequence of the local temporal feature graphs of the driving behavior is passed through a deep feature constraint fusion module based on an autocorrelation attention mechanism to obtain the global constrained driving behavior temporal feature graph.

5. The intelligent vehicle terminal data collection method according to claim 4, characterized in that: The deep learning network model is used to capture the spatiotemporal features of driving behavior on the set of driving monitoring video clips to obtain a sequence of local temporal feature graphs of driving behavior, including: The collection of driving monitoring video clips is passed through a driving behavior temporal feature capturer based on a three-dimensional convolutional neural network model to obtain a sequence of local temporal feature graphs of the driving behavior.

6. The intelligent vehicle terminal data collection method according to claim 5, characterized in that: The sequence of the local time series feature graphs of the driving behavior is passed through a deep feature constraint fusion module based on an autocorrelation attention mechanism to obtain the global constraint time series feature graph of the driving behavior, including: The sequence of the local time series feature graphs of the driving behavior is processed by the following deep feature constraint fusion formula to obtain the global constraint driving behavior time series feature graph; wherein the deep feature constraint fusion formula is: F=[a1f1;a2f2;···;a i f i ;···;a t f t ] Among them, h i is the global mean pooling feature vector of the ith driving behavior local temporal feature graph in the sequence of the driving behavior local temporal feature graphs, W i is the i-th weight coefficient matrix, is the i-th weight coefficient vector, B i is the i-th bias vector, tanh(·) represents the hyperbolic tangent function processing, e i is the i-th attention scoring coefficient, e k is the kth attention scoring coefficient, exp(·) represents the exponential function with base e, a i is the i-th attention weight coefficient, a t is the tth attention weight coefficient, t is the length of the sequence of the local temporal feature graph of the driving behavior, f i is the local temporal feature graph of the i-th driving behavior, f t is the t-th local time series feature graph of driving behavior, Concat[·;·] represents the cascade function, and F is the global constraint driving behavior time series feature graph.

7. The intelligent vehicle terminal data collection method according to claim 6, characterized in that: In the processor, determining a detection result based on the global restrictive driving behavior time series characteristic diagram includes: Using a nonlinear response compensation module to process the global restrictive driving behavior time series characteristic diagram to obtain a corrected global restrictive driving behavior time series characteristic diagram; Performing feature distribution optimization on the corrected global restrictive driving behavior time series feature graph to obtain an optimized corrected global restrictive driving behavior time series feature graph; The optimized corrected global constraint driving behavior time series characteristic diagram is passed through a classifier-based driving behavior detector to obtain the detection result, and the detection result is used to indicate whether a fatigue warning is issued.

8. The intelligent vehicle terminal data collection method according to claim 7, characterized in that: The global restrictive driving behavior time series characteristic diagram is processed using a nonlinear response compensation module to obtain a corrected global restrictive driving behavior time series characteristic diagram, including: The global restrictive driving behavior time series characteristic diagram is processed using the following nonlinear response compensation formula to obtain the corrected global restrictive driving behavior time series characteristic diagram; wherein the nonlinear response compensation formula is: Among them, L 0i is the pixel value of each position of the global constraint driving behavior time series feature map, A, B, C and D are adjustment parameters with different values, L 1i is the pixel value of each position of the corrected global constraint driving behavior time series feature map.

9. The intelligent vehicle terminal data collection method according to claim 8, characterized in that: The optimized corrected global constraint driving behavior time series feature graph is passed through a classifier-based driving behavior detector to obtain the detection result, which is used to indicate whether a fatigue warning is issued, including: Expanding the optimized corrected global constraint driving behavior time series feature graph into a classification feature vector according to a row vector or a column vector; Performing full connection encoding on the classification feature vector using multiple fully connected layers of the classifier to obtain an encoded classification feature vector; and The encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

10. An intelligent vehicle terminal data acquisition system, characterized in that: include: A driving monitoring video acquisition module is used to acquire driving monitoring videos of the driver to be analyzed within a predetermined time period collected by a camera; A monitoring video transmission module, used for transmitting the driving monitoring video to a processor of an intelligent vehicle terminal; A constraint fusion processing module, used for performing driving behavior analysis and deep feature constraint fusion processing on the driving monitoring video in the processor to obtain a global constraint driving behavior time series feature graph; A detection result determination module is used to determine the detection result in the processor based on the global restrictive driving behavior time series characteristic diagram.

Images