Fatigue driving detection method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a fatigue driving detection method and device, electronic equipment and a nonvolatile storage medium. The method comprises the steps that a to-be-detected video stream is acquired, and the to-be-detected video stream comprises an image of a driver in the vehicle driving process; variable receptive field convolution is adopted, feature extraction is carried out on image frames in the video stream to be detected, first feature maps of different scales are obtained, and when feature extraction is carried out on different positions of the image frames by the variable receptive field convolution, the sizes of corresponding receptive fields are different; performing scale alignment on the first feature maps of different scales, and performing weighted fusion on the first feature maps after scale alignment to obtain a target feature map; according to the target feature map, the driving state corresponding to the driver in the image frame is determined, and the driving state comprises fatigue driving and non-fatigue driving. According to the invention, the technical problem of low precision of the detection result of fatigue driving in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision applications, and more specifically, to a fatigue driving detection method, device, electronic device, and non-volatile storage medium. Background Art

[0002] In the related art, computer vision technology is usually combined with target detection algorithms to realize automatic monitoring of driver fatigue driving. However, since the manifestation of fatigue driving may be more complicated, the solutions in the related art may have the problem of false detection or missed detection during detection, resulting in the problem of low accuracy of monitoring results.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a fatigue driving detection method, device, electronic device and non-volatile storage medium to at least solve the technical problem of low accuracy of fatigue driving detection results in related technologies.

[0005] According to one aspect of an embodiment of the present application, a fatigue driving detection method is provided, including: obtaining a video stream to be detected, wherein the video stream to be detected contains an image of a driver during vehicle driving; using variable receptive field convolution to perform feature extraction on image frames in the video stream to be detected, and obtain first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field sizes are different; aligning the scales of the first feature maps of different scales, and performing weighted fusion on the scale-aligned first feature maps to obtain a target feature map; determining the driving state corresponding to the driver in the image frame based on the target feature map, wherein the driving state includes: fatigue driving and non-fatigue driving.

[0006] Optionally, variable receptive field convolution is used to perform feature extraction on image frames in the video stream to be detected, and first feature maps of different scales are obtained, including: determining the receptive field size of the convolution kernel of the variable receptive field convolution based on local features of different pixel positions in the image frame; using the convolution kernel to extract features of the local area corresponding to the pixel position in the image frame to obtain a sub-feature map; determining a first weight coefficient of the convolution kernel corresponding to the pixel position based on the local features, wherein the first weight coefficient is used to characterize the contribution of each local area in the image frame to the output first feature map; and determining the first feature map corresponding to the image frame based on the sub-feature map corresponding to each local area and the first weight coefficient.

[0007] Optionally, aligning the scales of first feature maps of different scales includes: obtaining a first feature map, wherein the scales of the first feature maps obtained by performing feature extraction on the image frame at different convolutional layers of the convolutional neural network are different, and the first feature map includes: a first scale feature map, a second scale feature map, and a third scale feature map, the detail information in the image frame contained in the second scale feature map is more than the detail information in the image frame contained in the third scale feature map, and less than the detail information in the image frame contained in the first scale feature map, and the semantic information in the image frame contained in the second scale feature map is less than the semantic information in the image frame contained in the third scale feature map, and more than the semantic information in the image frame contained in the first scale feature map; performing upsampling or downsampling operations on the first scale feature map and / or the third scale feature map so that the scales of the first scale feature map and / or the third scale feature map after the upsampling or downsampling operations are consistent with the scale of the second scale feature map.

[0008] Optionally, weighted fusion is performed on the scale-aligned first feature maps to obtain a target feature map, including: determining a second weight coefficient corresponding to each first feature map, wherein the second weight coefficient is used to characterize the contribution of each first feature map during fusion; using a densely connected feature pyramid, densely connecting the first feature maps according to the second weight coefficient to obtain a first fused image; obtaining a second fused image corresponding to the image frame, wherein the second fused image is obtained by fusing local features and context features corresponding to the image frame; and determining the target feature map based on the first fused image and the second fused image.

[0009] Optionally, obtaining a second fused image corresponding to the image frame includes: performing a convolution operation on the image frame to obtain a local feature map, and performing context feature extraction on the image frame to obtain a context feature map, wherein the local feature map is used to characterize detail information of the image frame, and the context feature map is used to characterize overall information of the image frame; fusing the local feature map and the context feature map to obtain an initial fused map, and performing an asymmetric convolution operation on the initial fused map to obtain a second fused image, wherein the asymmetric convolution operation is used to enhance the information expression capability of the feature map in different directions.

[0010] Optionally, the convolutional neural network used for feature extraction of image frames in the video stream to be detected includes multiple convolutional layers, each convolutional layer corresponds to a loss function, and the loss weights corresponding to the loss functions of different convolutional layers are different; the training steps of the convolutional neural network include: determining the loss function value corresponding to each convolutional layer based on the predicted output value of each convolutional layer in the convolutional neural network and the corresponding true annotation value; determining the hierarchical perception loss function value based on the loss function value and the loss weights corresponding to each convolutional layer, and updating the network parameters in the convolutional neural network based on the hierarchical perception loss function value.

[0011] Optionally, before performing feature extraction on the image frames in the video stream to be detected, the method also includes: adjusting the size of the image frames to a target size, wherein the target size is an input size supported by the convolutional neural network used for feature extraction on the image frames; and normalizing the pixel values ​​of each point in the image frames of the target size, wherein the normalization is used to scale the pixel values ​​of each point to a preset range.

[0012] According to another aspect of an embodiment of the present application, a fatigue driving detection device is also provided, including: an image acquisition module, used to acquire a video stream to be detected, wherein the video stream to be detected contains an image of the driver during vehicle driving; a feature extraction module, used to use a variable receptive field convolution to perform feature extraction on image frames in the video stream to be detected, and obtain first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field sizes are different; a feature fusion module, used to align the scales of the first feature maps of different scales, and perform weighted fusion on the scale-aligned first feature maps to obtain a target feature map; a state judgment module, used to determine the driving state corresponding to the driver in the image frame based on the target feature map, wherein the driving states include: fatigue driving and non-fatigue driving.

[0013] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the fatigue driving detection method is executed when the program is run.

[0014] According to another aspect of the embodiment of the present application, a non-volatile storage medium is provided, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the fatigue driving detection method by running the computer program.

[0015] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of the fatigue driving detection method when the computer program is executed by a processor.

[0016] In an embodiment of the present application, a video stream to be detected is obtained, wherein the video stream to be detected contains an image of the driver during vehicle driving; variable receptive field convolution is used to perform feature extraction on image frames in the video stream to be detected to obtain first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field sizes are different; the scales of the first feature maps of different scales are aligned, and the first feature maps after scale alignment are weighted fused to obtain a target feature map; based on the target feature map, the driving state corresponding to the driver in the image frame is determined, wherein the driving state includes: fatigue driving and non-fatigue driving. By improving the backbone network and introducing mechanisms such as variable receptive field convolution, the purpose of enhancing feature representation capability, improving the accuracy of identifying the driver's fatigue state, and reducing false detection and missed detection is achieved, thereby solving the technical problem of low accuracy of fatigue driving detection results in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for fatigue driving detection provided in an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a method flow for fatigue driving detection provided according to an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of a method flow for real-time detection of fatigue driving provided in accordance with an embodiment of the present application;

[0021] Figure 4 It is a structural schematic diagram of a fatigue driving detection device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In order to facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0025] Convolution: Convolution is a mathematical operation used to extract features from images in convolutional neural networks. The convolution operation performs a weighted summation of local areas on the input image through a sliding window (convolution kernel) to generate a feature map. This process helps the model capture edges, textures, and other important features in the image, allowing for higher-level visual tasks such as object detection and classification.

[0026] Feature Pyramid Network (FPN): is a structure that enhances the capabilities of convolutional neural networks and is used to process multi-scale features. The Feature Pyramid Network extracts feature maps at different scales and fuses these feature maps, enabling the model to more effectively detect objects of different sizes. FPN is widely used in target detection tasks because it can significantly improve the model's detection capabilities and accuracy for objects of various sizes.

[0027] Dense Connection Feature Pyramid (DCFP): It is a feature pyramid structure that densely connects different feature layers, aiming to reduce the loss of information during transmission and retain the details between feature layers. This structure improves the expressiveness of feature maps through multi-level connections and balances the capture of global and local information.

[0028] Enhanced Context Feature Fusion: is a module that improves object detection capabilities by aggregating multi-scale context information. It combines context information from different scales to enhance the model's detection capabilities in complex backgrounds and improve the handling of object-background confusion.

[0029] Hierarchical Perception Loss: It is a loss function that applies different weights to features at different levels in the feature map. This loss function optimizes the model's ability to detect different targets, especially small targets and occluded targets, during the training process, thereby improving the overall detection performance.

[0030] In related technologies, computer vision technology is usually combined with target detection algorithms to achieve automatic monitoring of driver fatigue driving. For example, target detection algorithms based on deep learning, such as the YOLO series (You Only Look Once), SSD (Single Shot MultiBox Detector), FasterR-CNN (Faster Region Convolutional Neural Network), etc. These algorithms usually use convolutional neural networks as the backbone network and combine different feature extraction and region proposal technologies to achieve detection and positioning of target objects.

[0031] However, these methods may have the following problems in the process of monitoring driver fatigue status in real-time video streams or static images:

[0032] 1) Insufficient accuracy: Since the manifestations of fatigue driving may be complex, the relevant technical solutions may have false detection or missed detection during detection, resulting in low accuracy of monitoring results. For example, fatigue manifestations such as closing the eyes and yawning of the driver may be misidentified or missed by other facial movements.

[0033] 2) High real-time requirements: During driving, the driver's status needs to be monitored in a timely and accurate manner to provide timely warnings and interventions. However, related technical solutions may have certain deficiencies in speed and real-time performance and cannot meet the needs of real-time monitoring. For example, complex network structures and high computational workloads may cause delays and affect real-time performance.

[0034] 3) High resource consumption: Training deep learning models requires a large amount of labeled data and computing resources, and in actual deployment, model reasoning may also require high computing resources and memory consumption, which may be a burden for mobile embedded systems. For example, embedded devices such as vehicle-mounted cameras and vehicle-mounted computers have limited computing power and memory capacity, making it difficult to support the operation of large models.

[0035] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0036] According to an embodiment of the present application, a method embodiment of fatigue driving detection is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing a fatigue driving detection method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0038] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the fatigue driving detection method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned fatigue driving detection method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0040] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0041] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0042] Under the above operating environment, the embodiment of the present application provides a method for detecting fatigue driving. Figure 2 is a schematic diagram of a method flow for fatigue driving detection provided in an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0043] Step S202, obtaining a video stream to be detected, wherein the video stream to be detected includes an image of the driver during the driving process of the vehicle;

[0044] Step S204, using variable receptive field convolution to extract features from image frames in the video stream to be detected, and obtaining first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field sizes are different;

[0045] Step S206, aligning the scales of the first feature maps of different scales, and performing weighted fusion on the scale-aligned first feature maps to obtain a target feature map;

[0046] Step S208, determining the driving state corresponding to the driver in the image frame according to the target feature map, wherein the driving state includes: fatigue driving and non-fatigue driving.

[0047] Through the above steps, by improving the backbone network and introducing mechanisms such as variable receptive field convolution, the purpose of enhancing feature representation capabilities, improving the accuracy of identifying the driver's fatigue state, and reducing false detections and missed detections is achieved, thereby solving the technical problem of low accuracy of fatigue driving detection results in related technologies.

[0048] The above method flow in the embodiment of the present application can be applied to the architecture of network element devices including embedded smart cameras, mobile embedded systems, communication modules, etc. The various network element devices are connected through the communication module to realize data transmission and interaction. Among them, the smart camera is used to obtain the image to be identified, the mobile embedded system is used to run the algorithm to realize target detection and prediction, and the communication module is used to transmit the monitoring results. The algorithm model of the embodiment of the present application is deployed on the mobile embedded system, and the image processing, target detection and communication functions are realized by software.

[0049] The following is a further introduction to the fatigue driving detection method in steps S202 to S208 of the embodiment of the present application.

[0050] The embodiment of the present application proposes a target detection method based on YOLO improvement, aiming to improve monitoring accuracy, realize real-time monitoring and reduce resource consumption. By improving the YOLOv5 model, introducing ghost convolution, multi-head attention mechanism and adaptive feature pyramid mechanism, it can significantly reduce the consumption of computing resources under the premise of ensuring high accuracy and high real-time performance, adapt to the needs of embedded systems, realize efficient monitoring of driver fatigue status, help improve driving safety, and reduce traffic accidents caused by fatigue driving. Figure 3 As shown, a detailed introduction is given below.

[0051] First, the image to be recognized is obtained through the smart camera, that is, the video stream to be detected is obtained. For example, the real-time video stream can be captured through the OpenCV library to obtain continuous image frames to ensure that the system can monitor the driver's status in real time, as shown in the following formula:

[0052] Frame t =OpenCV.capture(t)

[0053] Among them, Frame t represents the image frame captured at time t.

[0054] After obtaining the image frames in the video stream to be detected, data enhancement and optimization may be performed on the image frames. The specific steps are as follows.

[0055] In some embodiments of the present application, before performing feature extraction on image frames in a video stream to be detected, the method also includes the following steps: adjusting the size of the image frame to a target size, wherein the target size is an input size supported by a convolutional neural network used to extract features from the image frame; and normalizing the pixel values ​​of each point in the image frame of the target size, wherein the normalization is used to scale the pixel values ​​of each point to a preset range.

[0056] Specifically, each frame of the image is preprocessed, including resizing and normalization. In the preprocessing stage, the image is adjusted to a size acceptable to the model, that is, the above-mentioned target size, such as 640*640 pixels, as shown in the following formula:

[0057] Image resized =Resize(Frame t , 640×640)

[0058] Then, normalization is performed to scale the pixel values ​​to between 0 and 1 for easier network processing, as shown below:

[0059]

[0060] Such a preprocessing process ensures the consistency and standardization of the image, so that the improved backbone network and feature pyramid can better extract image features.

[0061] After preprocessing the image frame, the variable receptive field convolution can be used to extract and represent the features of the preprocessed image frame. The specific steps are as follows.

[0062] In some embodiments of the present application, variable receptive field convolution is used to perform feature extraction on image frames in a video stream to be detected, and obtaining first feature maps of different scales includes the following steps: determining the receptive field size of the convolution kernel of the variable receptive field convolution based on local features of different pixel positions in the image frame; using the convolution kernel, performing feature extraction on the local area corresponding to the pixel position in the image frame to obtain a sub-feature map; determining the first weight coefficient of the convolution kernel corresponding to the pixel position based on the local features, wherein the first weight coefficient is used to characterize the contribution of each local area in the image frame to the output first feature map; determining the first feature map corresponding to the image frame based on the sub-feature map corresponding to each local area and the first weight coefficient.

[0063] The above-mentioned adaptive receptive field convolution (ARFC) is a convolution operation that can dynamically adjust the receptive field size of the convolution kernel. It aims to adjust the range of the convolution operation according to the different features of the input image to better capture the multi-scale information of the target. In traditional convolution operations, the receptive field size is fixed, while ARFC introduces an adaptive weight mechanism so that the convolution kernel can dynamically adjust its receptive field at different positions according to the input features, thereby improving the flexibility of feature extraction and detection accuracy, especially in complex scenes and small target detection.

[0064] In the process of feature extraction using variable receptive field convolution, the receptive field size of the convolution kernel is adjusted according to the local features of the image during the convolution operation, and an adaptive weight mechanism is used to enhance the flexibility of the convolution kernel, so that the first feature map containing more semantic information can be generated, especially enhancing the ability to detect small targets and targets in complex scenes. The details are as follows.

[0065] In the embodiment of the present application, a receptive field function R(p) is defined to represent the receptive field of the convolution kernel in the input image, and the receptive field is adaptively adjusted as the input features change:

[0066] R(p)=f(p;θ)

[0067] Where p represents the pixel position of the input image; θ is a set of adaptive adjustment parameters, which can be dynamically adjusted according to the input features. The sub-feature map output by ARFC can be expressed as:

[0068]

[0069] Among them, O(i, j) is the sub-feature map output at position (i, j); the convolution kernel K R( p ) (m, n, c) is the receptive field convolution kernel at position (m, n) dynamically adjusted by the R(p) function according to the input features, and the corresponding channel is c; I(i+m, j+n, c) is the pixel value of the input image at position (i+m, j+n), and the corresponding channel is c; M×N is the size of the convolution kernel; C is the total number of channels of the input image. In this way, the convolution operation is no longer based on a fixed-size convolution kernel, but the size and shape of the convolution kernel are adaptively adjusted according to the changes in the local features of the input image.

[0070] At the same time, in order to further enhance the flexibility of the convolution kernel, an adaptive weight mechanism may be introduced in this embodiment, which adjusts the weight of the convolution kernel based on the local features of the input image:

[0071] K R(p)(m,n,c)=α(p)·K(m,n,c)

[0072] Wherein: α(p) is an adaptive weight function, which represents the influence of the input feature on the convolution kernel weight, that is, the first weight coefficient mentioned above. According to the sub-feature map corresponding to each local area and the first weight coefficient, the first feature map corresponding to the image frame can be determined.

[0073] After obtaining the first feature maps of different scales, the features can be fused at multiple levels through the multi-scale fusion module. First, the first feature maps of each scale need to be adjusted to the same size. The specific steps are as follows.

[0074] In some embodiments of the present application, aligning the scales of first feature maps of different scales includes the following steps: obtaining a first feature map, wherein the scales of the first feature maps obtained by performing feature extraction on an image frame at different convolutional layers of a convolutional neural network are different, and the first feature map includes: a first-scale feature map, a second-scale feature map, and a third-scale feature map, the detail information in the image frame contained in the second-scale feature map is more than the detail information in the image frame contained in the third-scale feature map, and is less than the detail information in the image frame contained in the first-scale feature map, and the semantic information in the image frame contained in the second-scale feature map is less than the semantic information in the image frame contained in the third-scale feature map, and is more than the semantic information in the image frame contained in the first-scale feature map; performing upsampling or downsampling operations on the first-scale feature map and / or the third-scale feature map so that the scales of the first-scale feature map and / or the third-scale feature map after the upsampling or downsampling operations are consistent with the scale of the second-scale feature map.

[0075] Specifically, the scales of the first feature maps extracted from different levels of the convolutional network are different. For example, three scales of first feature maps are extracted from different levels of the convolutional network: the first scale feature map F s 1 , the second scale feature map F s 2 , the third scale feature map F s 3 , wherein the first scale feature map is a low-level feature, containing more detail information, the second scale feature map is a medium-level feature, having both certain semantics and detail information, and the third scale feature map is a high-level feature, containing more semantic information. In this embodiment, the first feature maps of different scales can be adjusted to the same size by upsampling or downsampling operations, as follows.

[0076] Upsample the high-level feature map (third-scale feature map) to align its size with the middle-level feature map (second-scale feature map):

[0077]

[0078] Downsample the low-level feature map (first-scale feature map) to align its size with the middle-level feature map (second-scale feature map):

[0079]

[0080] In this way, the first feature maps of all scales are aligned to the same size, and then point-by-point operations can be performed to perform weighted fusion on the scale-aligned first feature maps. The specific steps are as follows.

[0081] In some embodiments of the present application, weighted fusion is performed on the scale-aligned first feature maps to obtain a target feature map, including the following steps: determining a second weight coefficient corresponding to each first feature map, wherein the second weight coefficient is used to characterize the contribution of each first feature map during fusion; using a densely connected feature pyramid, based on the second weight coefficient, densely connecting the first feature maps to obtain a first fused image; obtaining a second fused image corresponding to the image frame, wherein the second fused image is obtained by fusing local features and context features corresponding to the image frame; and determining the target feature map based on the first fused image and the second fused image.

[0082] Specifically, the aligned first feature maps can be fused according to different weighting coefficients (i.e., second weighting coefficients) to generate the final first fused image F fuse During the fusion process, the second weight coefficients of the feature maps of different scales can be dynamically adjusted according to the characteristics of the input image. The specific formula is as follows:

[0083]

[0084] Among them: α, β, γ are the second weight coefficients, which are used to indicate the contribution of feature maps of different scales during fusion; in this embodiment, an adaptive adjustment mechanism can be introduced to dynamically adjust the weight coefficients during fusion according to the image content, so that the feature map obtained after fusion is more suitable for detection requirements in different scenarios, as shown in the following formula:

[0085] α, β, γ = f(I; θ)

[0086] Among them, I is the first feature map of the input; θ is the parameter set for adaptive weight adjustment; f is the weight adjustment function, which is adaptively adjusted based on the features of the input image. Through this multi-scale feature fusion method, the model can more effectively capture the global information and detail information of the target, effectively fuse high-level semantic features with low-level detail features, enhance the model's multi-scale perception ability of the target, improve the detection ability of targets of different scales, and enhance the robustness of the model.

[0087] In addition, when the first feature map is fused according to the second weight coefficient, a densely connected feature pyramid (Dense Connection Feature Pyramid, DCFP) can also be used to connect the feature maps of all levels to each other through dense connection operations to achieve global fusion of information.

[0088] Among them, the densely connected feature pyramid forms a pyramid structure by establishing dense connections between different feature layers of the network, realizing the effective fusion of low-level and high-level features. This structure can transfer information between different feature layers, enabling the model to capture both detailed information and global semantic information in the image, thereby improving the accuracy and robustness of target detection, as follows.

[0089] In the traditional feature pyramid, the feature maps of each layer are connected layer by layer, while in the densely connected feature pyramid, direct connections are established between all feature layers. The formula is as follows:

[0090]

[0091] in: Represent the first feature maps of different resolutions and semantic depths respectively, ⊕ represents the connection operation of the feature map (which can be element-by-element addition or concatenation operation); Represents the feature map in the densely connected feature pyramid. Through dense connections, each feature layer can directly obtain the feature information of other layers, thereby realizing the fusion of global semantic information and local detail information. Through this densely connected feature pyramid structure, the model can achieve more effective feature fusion between features at different levels, thereby improving the detection ability of complex targets.

[0092] On the other hand, the embodiment of the present application can also obtain a second fused image by fusing based on context features, and the specific steps are as follows.

[0093] In some embodiments of the present application, obtaining a second fused image corresponding to an image frame includes the following steps: performing a convolution operation on the image frame to obtain a local feature map, and performing context feature extraction on the image frame to obtain a context feature map, wherein the local feature map is used to characterize detail information of the image frame, and the context feature map is used to characterize overall information of the image frame; fusing the local feature map and the context feature map to obtain an initial fused map, and performing an asymmetric convolution operation on the initial fused map to obtain a second fused image, wherein the asymmetric convolution operation is used to enhance the information expression capability of the feature map in different directions.

[0094] Specifically, in an embodiment of the present application, an enhanced context feature fusion module (ECFF) can be used to fuse local features with context features, wherein in the target detection task of deep learning, capturing context information is critical for accurately identifying targets. The enhanced context feature fusion module helps to improve the model's target detection accuracy in complex scenarios and reduce the false detection rate by integrating the context information around the target into the feature representation. The core idea of ​​this module is to further fuse more global context information on the basis of extracting local features, so that the model can more comprehensively understand the scene relationships in the image. The details are as follows.

[0095] First, a local feature map F can be extracted from the input image frame using a standard convolution operation local =f local (X), f local is a local feature extraction function; X is the input image frame. This feature mainly reflects the detailed information of the target.

[0096] At the same time, in order to capture contextual information, the contextual feature extraction operation can be performed by expanding the receptive field or applying a multi-scale feature extraction module. context , to obtain a wider context feature map F context =f context (X).

[0097] After that, the local feature map and the context feature map are weighted fused to obtain the enhanced feature map F enhanced (ie the initial fusion graph mentioned above): F enhanced =α·F local +β·F context , where α and β are fusion weight coefficients, which are used to control the contribution ratio of local features and context features.

[0098] In order to further improve the fusion effect, an asymmetric convolution operation can be applied to the fused initial feature map to enhance the information representation in different directions in the feature map, as shown in the following formula:

[0099] F asym =Conv asym (F enhanced )

[0100] Among them, Conv asym Represents an asymmetric convolution operation; Fasy m is the feature map after asymmetric convolution. asym Input into the subsequent detection network for processing to obtain the final second fusion image F output =g(F asym ), where g is the subsequent network processing operation. By introducing the enhanced context feature fusion module, the model can better understand the scene relationship in the image, especially in the object detection task under complex background, significantly improving the detection accuracy and reducing false detection.

[0101] Finally, based on the first fused image and the second fused image, a target feature map can be determined for final detection of the target to detect whether the driver is driving fatigued.

[0102] In addition, when training the convolutional neural network used for feature extraction of image frames, the embodiment of the present application uses a hierarchical perceptual loss function for optimization, and the specific steps are as follows.

[0103] In some embodiments of the present application, a convolutional neural network used for feature extraction of image frames in a video stream to be detected includes multiple convolutional layers, each convolutional layer corresponds to a loss function, and the loss weights corresponding to the loss functions of different convolutional layers are different; the training steps of the convolutional neural network include: determining the loss function value corresponding to each convolutional layer based on the predicted output value of each convolutional layer in the convolutional neural network and the corresponding true annotation value; determining the hierarchical perception loss function value based on the loss function value and the loss weights corresponding to each convolutional layer, and updating the network parameters in the convolutional neural network based on the hierarchical perception loss function value.

[0104] The above-mentioned Hierarchical Perception Loss function introduces a multi-level perception mechanism, which enables the model to optimize the target features at different levels. The core idea of ​​this mechanism is to set different loss weights at different feature levels of the neural network to balance the learning of global and local features, thereby improving the detection accuracy of the model.

[0105] Specifically, in order to allow the model to learn local and global features at the same time, the embodiment of the present application defines different loss functions at different feature levels. Assume that the neural network has a total of L feature layers, the output of each layer is Fl (where l = 1, 2, ..., L represents the layer number), and the corresponding prediction output is Pl. Define a loss function Ll for each layer, and the total loss function can be expressed as:

[0106]

[0107] Among them, α l represents the loss weight of level l, and G represents the ground truth.

[0108] Regarding the setting of the above loss weights, in this embodiment, a smaller weight can be given to low-level local features so that the model does not overly focus on details; while a larger weight can be given to high-level global features to ensure that the model can better capture the overall structural information. The specific weights can be tuned through experiments or set based on experience.

[0109] When optimizing the objective function, the total loss function L can be optimized using the gradient descent method to update the model parameters θ. By applying different loss function weights at different levels, the model can pay more attention to important feature levels during training, thereby showing better detection effects in practical applications.

[0110] Among them, the total loss function is shown in the following formula:

[0111]

[0112] The loss function of each layer is shown in the following formula:

[0113]

[0114] Where N is the number of samples, L is the loss function, such as cross entropy loss or mean square error loss, and P l i and G i They represent the predicted output and true label of the i-th sample respectively.

[0115] Through this optimization process, the layer-aware loss function enables the model to learn multi-level features in a more balanced way during training, thereby improving the accuracy and robustness of detection.

[0116] In addition, as an optional implementation, when performing fatigue driving detection, it is not only possible to make judgments based on the collected video stream, but also to fuse the visual data obtained by the smart camera with other sensor data (such as infrared sensors, radars or heart rate monitoring sensors). The multimodal fusion technology in deep learning is used to integrate data from different sources to improve the accuracy and robustness of driver status detection. For example, this can be achieved through feature-level fusion (fusing features from different data sources in the feature extraction stage) or decision-level fusion (fusing detection results from different data sources in the detection result stage). In this way, the advantages of different sensors can be utilized to reduce blind spots or interference that may be caused by a single sensor.

[0117] By introducing variable receptive field convolution and multi-scale fusion modules, the improved YOLOv5 model can better adapt to targets of different scales, thereby significantly improving detection accuracy, especially in complex scenes, it can more accurately identify small targets and dense targets; by adopting enhanced context feature fusion technology and densely connected feature pyramids, the model can better capture global and local information in the image, making it more robust in the face of occlusion, complex background, etc., and reducing the occurrence of missed detection and false detection; the use of hierarchical perception loss function enables the model to learn target features more effectively at different levels, and at the same time optimizes the model's reasoning speed by streamlining calculations and parameters. This is very beneficial for real-time detection applications and can be efficiently deployed in resource-limited embedded systems; the improved model reduces computational complexity and memory usage while maintaining high-precision detection. This enables the model to run on edge devices or low-power devices, further expanding the model's application scenarios, especially on mobile terminals and IoT devices.

[0118] According to an embodiment of the present application, an embodiment of a fatigue driving detection device is also provided. Figure 4 Schematic diagram of a fatigue driving detection device provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0119] An image acquisition module 40 is used to acquire a video stream to be detected, wherein the video stream to be detected includes an image of the driver during the driving process of the vehicle;

[0120] A feature extraction module 42 is used to use variable receptive field convolution to perform feature extraction on the image frames in the video stream to be detected, and obtain first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field size is different;

[0121] A feature fusion module 44 is used to align the scales of the first feature maps of different scales, and perform weighted fusion on the scale-aligned first feature maps to obtain a target feature map;

[0122] The state judgment module 46 is used to determine the driving state corresponding to the driver in the image frame according to the target feature map, wherein the driving state includes: fatigue driving and non-fatigue driving.

[0123] Optionally, variable receptive field convolution is used to perform feature extraction on image frames in the video stream to be detected, and first feature maps of different scales are obtained, including: determining the receptive field size of the convolution kernel of the variable receptive field convolution based on local features of different pixel positions in the image frame; using the convolution kernel to extract features of the local area corresponding to the pixel position in the image frame to obtain a sub-feature map; determining a first weight coefficient of the convolution kernel corresponding to the pixel position based on the local features, wherein the first weight coefficient is used to characterize the contribution of each local area in the image frame to the output first feature map; and determining the first feature map corresponding to the image frame based on the sub-feature map corresponding to each local area and the first weight coefficient.

[0124] Optionally, aligning the scales of first feature maps of different scales includes: obtaining a first feature map, wherein the scales of the first feature maps obtained by performing feature extraction on the image frame at different convolutional layers of the convolutional neural network are different, and the first feature map includes: a first scale feature map, a second scale feature map, and a third scale feature map, the detail information in the image frame contained in the second scale feature map is more than the detail information in the image frame contained in the third scale feature map, and less than the detail information in the image frame contained in the first scale feature map, and the semantic information in the image frame contained in the second scale feature map is less than the semantic information in the image frame contained in the third scale feature map, and more than the semantic information in the image frame contained in the first scale feature map; performing upsampling or downsampling operations on the first scale feature map and / or the third scale feature map so that the scales of the first scale feature map and / or the third scale feature map after the upsampling or downsampling operations are consistent with the scale of the second scale feature map.

[0125] Optionally, weighted fusion is performed on the scale-aligned first feature maps to obtain a target feature map, including: determining a second weight coefficient corresponding to each first feature map, wherein the second weight coefficient is used to characterize the contribution of each first feature map during fusion; using a densely connected feature pyramid, densely connecting the first feature maps according to the second weight coefficient to obtain a first fused image; obtaining a second fused image corresponding to the image frame, wherein the second fused image is obtained by fusing local features and context features corresponding to the image frame; and determining the target feature map based on the first fused image and the second fused image.

[0126] Optionally, obtaining a second fused image corresponding to the image frame includes: performing a convolution operation on the image frame to obtain a local feature map, and performing context feature extraction on the image frame to obtain a context feature map, wherein the local feature map is used to characterize detail information of the image frame, and the context feature map is used to characterize overall information of the image frame; fusing the local feature map and the context feature map to obtain an initial fused map, and performing an asymmetric convolution operation on the initial fused map to obtain a second fused image, wherein the asymmetric convolution operation is used to enhance the information expression capability of the feature map in different directions.

[0127] Optionally, the convolutional neural network used for feature extraction of image frames in the video stream to be detected includes multiple convolutional layers, each convolutional layer corresponds to a loss function, and the loss weights corresponding to the loss functions of different convolutional layers are different; the training steps of the convolutional neural network include: determining the loss function value corresponding to each convolutional layer based on the predicted output value of each convolutional layer in the convolutional neural network and the corresponding true annotation value; determining the hierarchical perception loss function value based on the loss function value and the loss weights corresponding to each convolutional layer, and updating the network parameters in the convolutional neural network based on the hierarchical perception loss function value.

[0128] Optionally, before performing feature extraction on image frames in the video stream to be detected, it also includes: adjusting the size of the image frame to a target size, wherein the target size is an input size supported by a convolutional neural network used for feature extraction on the image frame; and normalizing the pixel value of each point in the image frame of the target size, wherein the normalization is used to scale the pixel value of each point to a preset range.

[0129] It should be noted that the various modules in the above-mentioned fatigue driving detection device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0130] It should be noted that the fatigue driving detection device provided in this embodiment can be used to perform Figure 2 The fatigue driving detection method shown, therefore, the relevant explanations and instructions on the above fatigue driving detection method are also applicable to the embodiments of the present application and will not be repeated here.

[0131] An embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following fatigue driving detection method by running the computer program: obtaining a video stream to be detected, wherein the video stream to be detected includes an image of the driver during vehicle driving; using variable receptive field convolution to extract features from image frames in the video stream to be detected, and obtain first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field sizes are different; aligning the scales of the first feature maps of different scales, and weighted fusion of the first feature maps after scale alignment to obtain a target feature map; determining the driving state corresponding to the driver in the image frame based on the target feature map, wherein the driving state includes: fatigue driving and non-fatigue driving.

[0132] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the fatigue driving detection method described in each embodiment of the present application: obtaining a video stream to be detected, wherein the video stream to be detected contains an image of the driver during vehicle driving; using variable receptive field convolution to extract features from image frames in the video stream to be detected, and obtain first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field sizes are different; aligning the scales of the first feature maps of different scales, and weighted fusion of the first feature maps after scale alignment to obtain a target feature map; determining the driving state corresponding to the driver in the image frame based on the target feature map, wherein the driving state includes: fatigue driving and non-fatigue driving.

[0133] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0134] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0139] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for detecting fatigue driving, characterized in that: include: Acquire a video stream to be detected, wherein the video stream to be detected includes an image of a driver during vehicle driving; Using variable receptive field convolution to extract features from image frames in the video stream to be detected, and obtaining first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field sizes are different; Aligning the scales of the first feature maps of different scales, and performing weighted fusion on the scale-aligned first feature maps to obtain a target feature map; Based on the target feature map, the driving state corresponding to the driver in the image frame is determined, wherein the driving state includes: fatigue driving and non-fatigue driving.

2. The fatigue driving detection method according to claim 1, characterized in that: Using variable receptive field convolution, feature extraction is performed on the image frames in the video stream to be detected, and the first feature maps of different scales are obtained, including: Determining the receptive field size of the convolution kernel of the variable receptive field convolution according to local features of different pixel positions in the image frame; Using the convolution kernel, extracting features from a local area corresponding to the pixel position in the image frame to obtain a sub-feature map; Determining a first weight coefficient of the convolution kernel corresponding to the pixel position according to the local feature, wherein the first weight coefficient is used to characterize the contribution of each local area in the image frame to outputting the first feature map; The first feature map corresponding to the image frame is determined based on the sub-feature maps corresponding to each of the local areas and the first weight coefficient.

3. The fatigue driving detection method according to claim 1, characterized in that: Aligning the scales of the first feature maps of different scales includes: Acquire the first feature map, wherein the scales of the first feature maps obtained by performing feature extraction on the image frame at different convolutional layers of a convolutional neural network are different, and the first feature map includes: a first-scale feature map, a second-scale feature map, and a third-scale feature map, detail information in the image frame contained in the second-scale feature map is more than the detail information in the image frame contained in the third-scale feature map, and less than the detail information in the image frame contained in the first-scale feature map, and semantic information in the image frame contained in the second-scale feature map is less than the semantic information in the image frame contained in the third-scale feature map, and more than the semantic information in the image frame contained in the first-scale feature map; An upsampling or downsampling operation is performed on the first scale feature map and / or the third scale feature map so that a scale of the first scale feature map and / or the third scale feature map after the upsampling or downsampling operation is consistent with a scale of the second scale feature map.

4. The fatigue driving detection method according to claim 3, characterized in that: The first feature map after scale alignment is weightedly fused to obtain a target feature map including: Determine a second weight coefficient corresponding to each of the first feature maps, wherein the second weight coefficient is used to characterize a contribution of each of the first feature maps during fusion; Using a densely connected feature pyramid, and according to the second weight coefficient, performing a dense connection operation on the first feature map to obtain a first fused image; Acquire a second fused image corresponding to the image frame, wherein the second fused image is obtained by fusing the local features and context features corresponding to the image frame; The target feature map is determined according to the first fused image and the second fused image.

5. The fatigue driving detection method according to claim 4, characterized in that: Acquiring a second fused image corresponding to the image frame includes: Performing a convolution operation on the image frame to obtain a local feature map, and performing context feature extraction on the image frame to obtain a context feature map, wherein the local feature map is used to represent detail information of the image frame, and the context feature map is used to represent overall information of the image frame; The local feature map and the context feature map are fused to obtain an initial fused map, and an asymmetric convolution operation is performed on the initial fused map to obtain the second fused image, wherein the asymmetric convolution operation is used to enhance the information expression capability of different directions in the feature map.

6. The fatigue driving detection method according to claim 1, characterized in that: The convolutional neural network used for extracting features from the image frames in the video stream to be detected includes a plurality of convolutional layers, each of the convolutional layers corresponds to a loss function, and the loss functions of different convolutional layers correspond to different loss weights; the training steps of the convolutional neural network include: Determine the loss function value corresponding to each convolutional layer according to the predicted output value of each convolutional layer in the convolutional neural network and the corresponding true annotation value; According to the loss function value and the loss weight corresponding to each of the convolutional layers, a layer-aware loss function value is determined, and according to the layer-aware loss function value, the network parameters in the convolutional neural network are updated.

7. The fatigue driving detection method according to claim 1, characterized in that: Before extracting features from the image frames in the video stream to be detected, the method further includes: Adjusting the size of the image frame to a target size, wherein the target size is an input size supported by a convolutional neural network used for extracting features from the image frame; The pixel value of each point in the image frame of the target size is normalized, wherein the normalization is used to scale the pixel value of each point to a preset range.

8. A fatigue driving detection device, characterized in that: include: An image acquisition module, used to acquire a video stream to be detected, wherein the video stream to be detected contains an image of the driver during the driving process of the vehicle; A feature extraction module, configured to use a variable receptive field convolution to perform feature extraction on the image frames in the video stream to be detected, and obtain first feature maps of different scales, wherein when the variable receptive field convolution performs feature extraction at different positions of the image frame, the corresponding receptive field sizes are different; A feature fusion module, used to align the scales of the first feature maps of different scales, and perform weighted fusion on the scale-aligned first feature maps to obtain a target feature map; The state judgment module is used to determine the driving state corresponding to the driver in the image frame according to the target feature map, wherein the driving state includes: fatigue driving and non-fatigue driving.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the fatigue driving detection method described in any one of claims 1 to 7 is executed when the program is run.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the fatigue driving detection method described in any one of claims 1 to 7 by running the computer program.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the fatigue driving detection method described in any one of claims 1 to 7 are implemented.

Images