Cockpit early warning prompt information determination method and device and storage medium

Abstract

This application provides a method, apparatus, and storage medium for determining cockpit warning information. The method involves acquiring a video stream of the driver's facial pose features. The video stream includes a first preset number of image frames. Based on the first preset number of image frames and a warning model, corresponding feature maps are determined. Based on the warning model, inter-frame feature fusion is performed on the feature maps to obtain a fused feature map. Based on the fused feature map, the classification result of the facial pose feature video stream is determined, and based on the classification result, corresponding warning information is determined. This technical solution can replace manual feature design, has a fast calculation speed, and can achieve real-time warnings, making warning reminders more rapid.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and in particular to a method, device and storage medium for determining cockpit warning information. Background Technology

[0002] In recent years, the number of motor vehicles on the road has been countless, making traffic safety a topic of particular concern in the transportation industry. Traffic accidents can cause significant losses to families and property. Many traffic accidents are caused by driver fatigue. Driver fatigue has outward signs, such as yawning or frequent nodding. Therefore, cameras can be used to detect drivers yawning and dozing off to prevent traffic accidents caused by fatigue.

[0003] Currently, driver fatigue monitoring systems mainly detect information about the driver's face, eyes, or head through vehicle-mounted cameras. The traditional method is to extract features manually. However, the traditional solution requires manually designing features and obtaining a dense vector through feature encoding, which is a complex calculation process with a large number of parameters.

[0004] Therefore, there is an urgent need for a method to determine cockpit warning information that can replace manual feature design, has a fast calculation speed, and can achieve the purpose of real-time warning, making warning reminders more rapid. Summary of the Invention

[0005] This application provides a method, device, and storage medium for determining cockpit warning information, which can replace manual design features, has a fast calculation speed, and can achieve the purpose of real-time warning, making warning reminders faster.

[0006] Firstly, this application provides a method for determining cockpit warning information, including:

[0007] Acquire a video stream of the driver's facial pose features; wherein the video stream of facial pose features includes a first preset number of image frames;

[0008] Based on the first preset number of image frames and the early warning model, determine the corresponding feature map;

[0009] Based on the early warning model, inter-frame feature fusion is performed on the feature map to obtain a fused feature map;

[0010] Based on the fused feature map, the classification result of the facial pose feature video stream is determined, and based on the classification result, corresponding early warning information is determined.

[0011] In one example, determining the corresponding feature map based on the first preset number of image frames and the warning model includes:

[0012] Cluster analysis is performed on the first preset number of image frames to obtain a second preset number of clustered image frame sets;

[0013] Based on the early warning model, features are extracted from the image frame set to generate corresponding feature maps; wherein, the number of feature maps is a second preset number.

[0014] In one example, the step of extracting features from the image frame set based on the early warning model and generating a corresponding feature map includes:

[0015] One clustering image frame is obtained in each of the image frame sets; wherein the number of clustering image frames is equal to a second preset number;

[0016] Based on the aforementioned early warning model, features are extracted from each of the clustered image frames to generate a corresponding feature map.

[0017] In one example, the step of performing inter-frame feature fusion on the feature map based on the early warning model to obtain a fused feature map includes:

[0018] Based on the early warning model, inter-frame information exchange is performed on the feature map to obtain the exchanged inter-frame feature map;

[0019] Based on the early warning model, the exchanged inter-frame feature map is fused with the clustered image frame to obtain a fused feature map.

[0020] In one example, the warning model is obtained by training multiple video streams of facial pose features labeled with classification results.

[0021] In one example, determining the corresponding early warning message based on the classification result includes:

[0022] If the classification result indicates a state of fatigue, then the corresponding early warning message will be issued.

[0023] If the classification result indicates a non-fatigue state, then it is determined that the corresponding warning message will not be issued.

[0024] In one example, the fatigue state is characterized by eye features, mouth features, or head features. If the classification result indicates a fatigue state, then the corresponding warning message is issued, including:

[0025] If the fatigue state is characterized by eye features, then a warning message corresponding to the eye features will be issued.

[0026] If the fatigue state is characterized by a mouth feature, then a warning message corresponding to the mouth feature will be issued.

[0027] If the fatigue state is characterized by head features, then a warning message corresponding to the head features will be issued.

[0028] Secondly, this application provides a cockpit warning information determination device, comprising:

[0029] An acquisition unit is used to acquire a video stream of the driver's facial pose features; wherein the video stream of facial pose features includes a first preset number of image frames;

[0030] The first determining unit is used to determine the corresponding feature map based on the first preset number of image frames and the early warning model;

[0031] The fusion unit is used to perform inter-frame feature fusion on the feature map based on the early warning model to obtain a fused feature map.

[0032] The second determining unit is used to determine the classification result of the facial pose feature video stream based on the fused feature map, and to determine the corresponding warning information based on the classification result.

[0033] In one example, the first defined unit includes:

[0034] The first determining module is used to perform cluster analysis on the first preset number of image frames to obtain a second preset number of clustered image frame sets;

[0035] The generation module is used to extract features from the image frame set based on the early warning model and generate corresponding feature maps; wherein the number of feature maps is a second preset number.

[0036] In one example, the generated module includes:

[0037] The acquisition submodule is used to acquire one clustered image frame in each of the image frame sets; wherein the number of clustered image frames is equal to a second preset number;

[0038] The extraction submodule is used to extract features from each clustered image frame based on the early warning model and generate a corresponding feature map.

[0039] In one example, the fusion unit includes:

[0040] The exchange module is used to perform inter-frame information exchange on the feature map based on the early warning model to obtain the exchanged inter-frame feature map.

[0041] The fusion module is used to perform inter-frame feature fusion between the exchanged inter-frame feature map and the clustered image frame based on the early warning model to obtain a fused feature map.

[0042] In one example, the warning model is obtained by training multiple video streams of facial pose features labeled with classification results.

[0043] In one example, the second determining unit includes:

[0044] The second determining module is used to determine to issue the corresponding early warning message if the classification result is a fatigue state.

[0045] The third determining module is used to determine not to issue the corresponding warning message if the classification result is a non-fatigue state.

[0046] In one example, the type of fatigue state representation includes: eye features, mouth features, or head features; the second determining module includes:

[0047] The first determining submodule is used to determine and issue a warning message corresponding to the eye feature if the fatigue state is characterized by an eye feature.

[0048] The second determining submodule is used to determine and issue a warning message corresponding to the mouth feature if the fatigue state is characterized by a mouth feature.

[0049] The third determining submodule is used to determine and issue a warning message corresponding to the head feature if the fatigue state is characterized by a head feature.

[0050] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0051] The memory stores computer-executed instructions;

[0052] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect.

[0053] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0054] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0055] This application provides a method for determining cockpit warning information, which involves acquiring a video stream of the driver's facial pose features. The video stream includes a first preset number of image frames. Based on the first preset number of image frames and a warning model, a corresponding feature map is determined. Based on the warning model, inter-frame feature fusion is performed on the feature map to obtain a fused feature map. Based on the fused feature map, a classification result of the facial pose feature video stream is determined, and based on the classification result, a corresponding warning information is determined. This technical solution can replace manual feature design, has a fast calculation speed, and can achieve real-time warnings, making warning reminders more rapid. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0057] Figure 1 This is a flowchart illustrating a method for determining cockpit warning information according to Embodiment 1 of this application;

[0058] Figure 2 This is a flowchart illustrating a method for determining cockpit warning information according to Embodiment 2 of this application;

[0059] Figure 3 This is a schematic diagram illustrating the process of inter-frame information exchange feature maps according to Embodiment 2 of this application;

[0060] Figure 4 This is a partial structural schematic diagram of an early warning model provided according to Embodiment 2 of this application;

[0061] Figure 5 This is a schematic diagram of a cockpit warning information determination device according to Embodiment 3 of this application;

[0062] Figure 6 This is a schematic diagram of a cockpit warning information determination device according to Embodiment 4 of this application;

[0063] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0064] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0065] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0066] The method for determining cockpit warning information provided in this application aims to solve the above-mentioned technical problems in the prior art.

[0067] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0068] Figure 1 This is a flowchart illustrating a method for determining cockpit warning information according to Embodiment 1 of this application.

[0069] Example 1 includes the following steps:

[0070] S101. Obtain the driver's facial pose feature video stream; wherein, the facial pose feature video stream includes a first preset number of image frames.

[0071] In this embodiment, facial pose features are acquired through a camera device inside the cockpit. Specifically, facial pose features refer to features related to the driver's facial movements, including the driver's facial features, eye features, and head features. The facial pose feature video stream comprises multiple segments, each segment including a first preset number of image frames, where the first preset number can be 30 frames.

[0072] S102. Determine the corresponding feature map based on the first preset number of image frames and the early warning model.

[0073] In this embodiment, the early warning model is used to alert drivers to their fatigue state. The backbone network of this model is ResNet-50, and a CBAM (Convolutional Block Attention Module, a channel and spatial attention mechanism) is added to this ResNet-50 backbone network. CBAM is an attention mechanism module that combines spatial and channel attention, achieving better results compared to attention mechanisms that only focus on channels. In this embodiment, feature maps are images used to represent the characteristics of image frames; specifically, there is more than one feature map.

[0074] S103. Based on the early warning model, perform inter-frame feature fusion on the feature map to obtain a fused feature map.

[0075] In this embodiment, inter-frame feature fusion refers to the fusion of features between adjacent frame feature maps. Specifically, the inter-frame feature maps can be fused between the first and second frame feature maps, or between the third and fourth frame feature maps. In this embodiment, inter-frame feature fusion can be the fusion of multiple channels of the feature map. Specifically, if the number of channels is 256, it can be the fusion of random channels. For example, the fusion of the third channel of the first frame feature map and the third channel of the second frame feature map, the fusion of the fourth channel of the first frame feature map and the fourth channel of the second frame feature map, and so on. In this embodiment, the number of random channels can be one-quarter of the total number of channels. For example, if the total number of channels is 256, the number of random channels can be 64, and 64 channels from adjacent frame feature maps can be fused to obtain a fused feature map. The advantage of this setup is that it avoids the inefficiency and inaccuracy problems caused by moving every channel. Channel movement introduces latency, and when moving channels temporally, it shifts information from many adjacent frames to the current frame, disrupting the spatial structure of the current frame and affecting the spatial modeling capability of the warning model, thus reducing accuracy. Therefore, to address the inefficiency issue, a partial channel movement approach is used, achieving a trade-off between latency and accuracy.

[0076] S104. Based on the fused feature map, determine the classification result of the facial pose feature video stream, and based on the classification result, determine the corresponding early warning information.

[0077] In this embodiment, the classification results include fatigue state and non-fatigue state. If the driver is in a fatigue state, a warning message needs to be sent to inform the driver so that the driver can stop the vehicle and take a timely rest. If the driver is not in a fatigue state, no warning message is needed to avoid interfering with the driver.

[0078] In this embodiment, the classification result of the fused feature map is determined based on the fused feature map, and the classification result of the fused feature map is determined as the classification result of the face pose feature video stream corresponding to the fused feature map.

[0079] This application provides a method for determining cockpit warning information, which involves acquiring a video stream of the driver's facial pose features. The video stream includes a first preset number of image frames. Based on the first preset number of image frames and a warning model, corresponding feature maps are determined. Inter-frame feature fusion is performed on the feature maps based on the warning model to obtain a fused feature map. The classification result of the facial pose feature video stream is determined based on the fused feature map, and the corresponding warning information is determined based on the classification result. This technical solution can replace manual feature design, has a fast calculation speed, and can achieve real-time warnings, making warning reminders more rapid.

[0080] Figure 2 This is a flowchart illustrating a method for determining cockpit warning information according to Embodiment 2 of this application.

[0081] Example 2 includes the following steps:

[0082] S201. Obtain the driver's facial pose feature video stream; wherein, the facial pose feature video stream includes a first preset number of image frames.

[0083] For example, this step can refer to step S101 above, and will not be repeated here.

[0084] S202. Perform cluster analysis on the first preset number of image frames to obtain a second preset number of clustered image frame sets.

[0085] In this embodiment, clustering analysis refers to clustering a first preset number of image frames using the k-means clustering algorithm. k-means is an iterative clustering algorithm that divides the first preset number of image frames into K groups, where the value of each K group represents a second preset number of frames. Then, the first preset number of image frames is clustered. Since the first preset number of frames is greater than the second preset number, the clustering process is not equally applied, resulting in different numbers of image frames in the resulting image frame sets. In this embodiment, the first preset number can be 30 frames, and the second preset number can be 8 groups. The clustered image frame set consists of RGB image frames with a size of 224*224.

[0086] S203. Based on the early warning model, extract features from the image frame set and generate corresponding feature maps; wherein, the number of feature maps is the second preset number.

[0087] In one example, the early warning model is trained from multiple video streams of facial pose features labeled with classification results. Specifically, the loss function used in this early warning model is Focal Loss, and the formula is as follows:

[0088]

[0089] in, γ = 4, p t This is a probability prediction value.

[0090] In this embodiment, the number of facial pose feature video streams labeled with classification results can be 20,000 segments, which is a collection of a certain number of driver facial pose feature video streams, with the first number being 50 bits. Of these, 16,000 segments can be used to train the warning model, and 2,000 segments can be used as the test set and 2,000 segments as the validation set. In this embodiment, the classification result is either a fatigue state or a non-fatigue state. The representation types of fatigue state include: eye features, mouth features, or head features. Specifically, eye features are represented by frequent eye closing, mouth features by frequent yawning, and head features by frequent nodding, which can then be labeled in the facial pose feature video streams.

[0091] In this embodiment, the feature map is obtained by recognizing the RGB images in the image frame set.

[0092] In one example, features are extracted from a set of image frames based on an early warning model to generate corresponding feature maps, including:

[0093] One clustered image frame is acquired in each image frame set; the number of clustered image frames is equal to the second preset number; features are extracted from each clustered image frame based on the early warning model to generate the corresponding feature map.

[0094] In this embodiment, if the second preset number is 8 classes, then one clustering image frame can be obtained in the clustering image frame of each class, resulting in 8 clustering image frames. Then, based on the early warning model, feature extraction is performed on the 8 clustering image frames to obtain 8 feature maps.

[0095] S204. Based on the early warning model, perform inter-frame information exchange on the feature maps to obtain the exchanged inter-frame feature maps.

[0096] In this embodiment, please refer to Figure 3The diagram illustrates a process of inter-frame information exchange feature maps. Each row represents a feature map at a different time point. It can be seen that by exchanging frame feature maps corresponding to different time points, each row of feature maps contains image information from the feature maps of other rows. In this embodiment, the inter-frame information exchange is specifically implemented through the Residual TSM module in the early warning model. For details, please refer to... Figure 4 The diagram shows a partial structure of the early warning model. It can be seen that there are two branches. One branch is for inter-frame feature fusion, and the other branch is for feature maps to preserve the original spatial information of a certain frame. This can avoid damaging the spatial feature learning ability of the original 2DCNN model. This is why the Residual TSM structure can improve accuracy, thereby realizing information exchange between adjacent frames and obtaining temporal information relationships.

[0097] In this embodiment, by exchanging inter-frame information between the first and second frame feature maps, the third and fourth frame feature maps, the fifth and sixth frame feature maps, and the seventh and eighth frame feature maps, four frames of exchanged inter-frame feature maps can be obtained.

[0098] S205. Based on the early warning model, perform inter-frame feature fusion on the exchanged inter-frame feature map and the feature map to obtain the fused feature map.

[0099] In this embodiment, the inter-frame feature maps after the four frames are swapped are fused with the feature maps of eight frames. Specifically, the eight feature maps are combined into a first matrix, and the four swapped inter-frame feature maps are combined into a second matrix, wherein the first matrix and the second matrix have the same dimension. The swapped inter-frame feature maps are then convolved and fused with the eight feature maps to obtain a fused feature map. This fused feature map possesses both spatial and temporal features from different time points, achieving the performance of a three-dimensional CNN while maintaining the relatively low computational cost of a two-dimensional CNN.

[0100] S206. Based on the fused feature map, determine the classification result of the face pose feature video stream.

[0101] In this embodiment, the fused feature map is input into the early warning model, and normalized through a fully connected layer and a softmax layer to finally determine the classification result of the face pose feature video stream.

[0102] S207. If the classification result is fatigue state, then determine to issue the corresponding early warning message.

[0103] In one example, the representation type of fatigue state includes: eye features, mouth features, or head features. If the classification result is fatigue state, then a corresponding warning message will be issued, including: if the representation type of fatigue state is eye features, then a warning message corresponding to the eye features will be issued; if the representation type of fatigue state is mouth features, then a warning message corresponding to the mouth features will be issued; if the representation type of fatigue state is head features, then a warning message corresponding to the head features will be issued.

[0104] In this embodiment, the warning messages can be pre-set, and different fatigue states can correspond to different warning messages. For example, the warning message corresponding to eye features could be a voice message saying, "Your eyes are showing signs of fatigue; please take a rest." The warning message corresponding to mouth features could be a voice message saying, "Your mouth is showing signs of fatigue; please take a rest." The warning message corresponding to head features could be a voice message saying, "Your head is showing signs of fatigue; please take a rest."

[0105] S208. If the classification result is a non-fatigue state, then it is determined not to issue a corresponding warning message.

[0106] For example, this step can refer to step S104 above, and will not be repeated here.

[0107] This application provides a method for determining cockpit warning information. Based on a warning model, inter-frame information exchange is performed on feature maps to obtain exchanged inter-frame feature maps. Then, based on the warning model, the exchanged inter-frame feature maps and the original feature maps are fused to obtain a fused feature map. By employing this technical solution and fusing features from the warning model, more feature information can be obtained, thereby improving the accuracy of driver fatigue detection.

[0108] Figure 5 This is a schematic diagram of a cockpit warning information determining device according to Embodiment 3 of this application. The device 50 in Embodiment 3 includes:

[0109] The acquisition unit 501 is used to acquire a video stream of the driver's facial pose features; wherein the video stream of facial pose features includes a first preset number of image frames.

[0110] The first determining unit 502 is used to determine the corresponding feature map based on a first preset number of image frames and an early warning model.

[0111] The fusion unit 503 is used to perform inter-frame feature fusion on the feature map based on the early warning model to obtain a fused feature map.

[0112] The second determining unit 504 is used to determine the classification result of the face pose feature video stream based on the fused feature map, and to determine the corresponding early warning information based on the classification result.

[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0114] Figure 6 This is a schematic diagram of a cockpit warning information determining device according to Embodiment 4 of this application. The device 60 in Embodiment 4 includes:

[0115] The acquisition unit 601 is used to acquire a video stream of the driver's facial pose features; wherein the video stream of facial pose features includes a first preset number of image frames.

[0116] The first determining unit 602 is used to determine the corresponding feature map based on a first preset number of image frames and an early warning model.

[0117] The fusion unit 603 is used to perform inter-frame feature fusion on the feature map based on the early warning model to obtain a fused feature map.

[0118] The second determining unit 604 is used to determine the classification result of the face pose feature video stream based on the fused feature map, and to determine the corresponding early warning information based on the classification result.

[0119] In one example, the first determining unit 602 includes:

[0120] The first determining module 6021 is used to perform cluster analysis on a first preset number of image frames to obtain a second preset number of clustered image frame sets.

[0121] The generation module 6022 is used to extract features from the image frame set based on the early warning model and generate corresponding feature maps; wherein the number of feature maps is a second preset number.

[0122] In one example, module 6022 is generated, including:

[0123] The acquisition submodule 60221 is used to acquire one clustered image frame in each image frame set; wherein the number of clustered image frames is equal to the second preset number.

[0124] The extraction submodule 60222 is used to extract features from each clustered image frame based on the early warning model and generate the corresponding feature map.

[0125] In one example, the fusion unit 603 includes:

[0126] The switching module 6031 is used to perform inter-frame information exchange on feature maps based on the early warning model to obtain the exchanged inter-frame feature maps.

[0127] The fusion module 6032 is used to perform inter-frame feature fusion with the clustered image frames based on the early warning model to obtain a fused feature map.

[0128] In one example, the early warning model was trained from multiple video streams of facial pose features labeled with classification results.

[0129] In one example, the second determining unit 604 includes:

[0130] The second determining module 6041 is used to determine and issue corresponding early warning information if the classification result is fatigue state.

[0131] The third determining module 6042 is used to determine not to issue a corresponding warning message if the classification result is a non-fatigue state.

[0132] In one example, the representation types of fatigue state include: eye features, mouth features, or head features. The second determining module 6041 includes:

[0133] The first determining submodule 60411 is used to determine and issue a warning message corresponding to the eye feature if the fatigue state is characterized by an eye feature.

[0134] The second determining submodule 60412 is used to determine and issue a warning message corresponding to the mouth feature if the fatigue state is characterized by a mouth feature.

[0135] The third determining submodule 60413 is used to determine and issue a warning message corresponding to the head feature if the fatigue state is characterized by head features.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. The device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0138] The device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output (I / O) interface 712, sensor component 714, and communication component 716.

[0139] Processing component 702 typically controls the overall operation of device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0140] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0141] Power supply assembly 706 provides power to various components of device 700. Power supply assembly 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 700.

[0142] Multimedia component 708 includes a screen that provides an output interface between device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0143] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0144] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0145] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of device 700. For example, sensor assembly 714 may detect the on / off state of device 700, the relative positioning of components such as the display and keypad of device 700, changes in the position of device 700 or a component of device 700, the presence or absence of user contact with device 700, the orientation or acceleration / deceleration of device 700, and temperature changes of device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0146] Communication component 716 is configured to facilitate wired or wireless communication between device 700 and other devices. Device 700 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0147] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0148] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of the device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0149] A non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to perform the aforementioned cockpit warning information determination method.

[0150] This application also discloses a computer program product, including a computer program that, when executed by a processor, implements the method described in this embodiment.

[0151] Various embodiments of the systems and technologies described above in this application can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0153] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0155] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data electronic devices), or computing systems that include middleware components (e.g., application electronic devices), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0156] Computer systems can include client and electronic devices. Clients and electronic devices are generally geographically separated and typically interact via communication networks. The client-electronic device relationship is created by computer programs running on the respective computers and having a client-electronic device relationship with each other. The electronic device can be a cloud electronic device, also known as a cloud computing electronic device or cloud host, a host product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server," or simply "VPS") in terms of management difficulty and weak business scalability. The electronic device can also be an electronic device in a distributed system or an electronic device incorporating blockchain technology. It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application is achieved, and this is not limited herein.

[0157] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0158] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for determining cockpit warning information, characterized in that, The method includes: Acquire a video stream of the driver's facial pose features; wherein the video stream of facial pose features includes a first preset number of consecutive image frames; The first preset number of consecutive image frames are input into the early warning model. The feature map corresponding to each frame image is extracted through the two-dimensional convolutional neural network backbone network and channel and spatial attention mechanism module of the early warning model. The early warning model is obtained by training multiple face pose feature video streams labeled with classification results using the Focal Loss loss function. Based on the residual time displacement module in the early warning model, inter-frame feature fusion is performed on the feature map to obtain a fused feature map. The inter-frame feature fusion includes: shifting a predetermined proportion of channels of the feature maps of adjacent frames along the time dimension to the previous and next frames respectively, generating swapped inter-frame feature maps; and adding the swapped inter-frame feature maps to the original feature maps before the shifting operations via residual connections to construct a fused feature map containing temporal dynamic information. The temporal dynamic information includes at least one of the following features that change over time: eye closure, mouth opening, or head posture angle. Based on the fused feature map, the classification result of the facial pose feature video stream is determined, and based on the classification result, corresponding early warning information is determined.

2. The method according to claim 1, characterized in that, The step of determining the corresponding early warning information based on the classification results includes: If the classification result indicates a state of fatigue, then the corresponding early warning message will be issued. If the classification result indicates a non-fatigue state, then it is determined that the corresponding warning message will not be issued.

3. The method according to claim 2, characterized in that, The fatigue state is characterized by features such as eye features, mouth features, or head features. If the classification result indicates a fatigue state, then the corresponding warning message is issued, including: If the fatigue state is characterized by eye features, then a warning message corresponding to the eye features will be issued. If the fatigue state is characterized by a mouth feature, then a warning message corresponding to the mouth feature will be issued. If the fatigue state is characterized by head features, then a warning message corresponding to the head features will be issued.

4. A cockpit early warning information determination device, characterized in that, The device includes: An acquisition unit is used to acquire a video stream of the driver's facial pose features; wherein the video stream of facial pose features includes a first preset number of consecutive image frames; The first determining unit is used to input the first preset number of consecutive image frames into the early warning model, and extract the feature map corresponding to each frame image through the two-dimensional convolutional neural network backbone network and channel and spatial attention mechanism module of the early warning model; wherein, the early warning model is obtained by training multiple face pose feature video streams labeled with classification results using the Focal Loss loss function. The fusion unit is used to perform inter-frame feature fusion on the feature map based on the residual time displacement module in the early warning model to obtain a fused feature map. The inter-frame feature fusion includes: shifting a predetermined proportion of channels of the feature maps of adjacent frames along the time dimension to the previous and next frames respectively, generating swapped inter-frame feature maps; and adding the swapped inter-frame feature map to the original feature map before the shifting operation through a residual connection to construct a fused feature map containing temporal dynamic information. The temporal dynamic information includes at least one of the following features that change over time: eye closure degree, mouth opening degree, or head posture angle. The second determining unit is used to determine the classification result of the facial pose feature video stream based on the fused feature map, and to determine the corresponding warning information based on the classification result.

5. The apparatus according to claim 4, characterized in that, The second determining unit includes: The first determining module is used to determine to issue the corresponding early warning message if the classification result is a state of fatigue. The second determining module is used to determine not to issue the corresponding warning message if the classification result is a non-fatigue state.

6. The apparatus according to claim 5, characterized in that, The fatigue state characterization types include: eye features, mouth features, or head features. The first determining module includes: The first determining submodule is used to determine and issue a warning message corresponding to the eye feature if the fatigue state is characterized by an eye feature. The second determining submodule is used to determine and issue a warning message corresponding to the mouth feature if the fatigue state is characterized by a mouth feature. The third determining submodule is used to determine and issue a warning message corresponding to the head feature if the fatigue state is characterized by a head feature.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-3.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-3.

Images