Vehicle-mounted device and detection method of vehicle-mounted device

Abstract

The invention provides a vehicle-mounted device and a detection method of the vehicle-mounted device, the vehicle-mounted device comprises a face recognition unit and a fatigue detection unit, the face recognition unit is used for obtaining first image information, and the first image information comprises face information; the fatigue detection unit is used for judging whether the driver is in a first state or not according to the first image information; the fatigue detection unit comprises a pre-training model and an optimization module, and if the pre-training model cannot directly judge whether the driver is in the first state according to the first image information, the pre-training model judges whether the driver is in the first state based on the input information; the optimization module updates the pre-training model based on the first image information and the input information; according to the vehicle-mounted device provided by the invention, the accuracy of fatigue detection is improved by arranging the optimization module, and meanwhile, the pre-training model is further updated.

Description

Technical Field

[0001] The present disclosure relates to the field of display technology, and in particular to a vehicle-mounted device and a detection method for the vehicle-mounted device. Background Art

[0002] With the development of science and technology, in order to improve driving safety, the user fatigue warning system is an important part of the user status monitoring system. The user fatigue warning system can monitor and remind the user of his own first state, reducing the potential harm of user fatigue driving.

[0003] However, the comfort and effect of user fatigue detection of the vehicle-mounted device in the prior art are not good enough. Summary of the invention

[0004] In view of this, the purpose of the present disclosure is to provide a vehicle-mounted device and a detection method for the vehicle-mounted device, wherein the vehicle-mounted device can solve at least one of the above-mentioned technical problems.

[0005] Based on the above objectives, the present disclosure discloses a vehicle-mounted device, the display device comprising:

[0006] A face recognition unit, used to obtain first image information, where the first image information includes face information;

[0007] a fatigue detection unit, configured to determine whether the driver is in a first state according to the first image information;

[0008] The fatigue detection unit includes a pre-trained model and an optimization module. If the pre-trained model cannot directly determine whether the driver is in the first state based on the first image information, the pre-trained model determines whether the driver is in the first state based on input information, and the optimization module updates the pre-trained model based on the first image information.

[0009] In one embodiment, it further comprises:

[0010] The display panel comprises a first display area and a second display area, the light transmittance of the first display area is greater than the light transmittance of the second display area, and the display panel comprises a light emitting side and a backlight side which are arranged opposite to each other;

[0011] an image acquisition unit, located at a backlight side of the display panel and at least a portion of a positive projection of the image acquisition unit on the display panel is located in the first display area, the image acquisition unit being used to collect video information, the video information including a driving area image;

[0012] Preferably, the image acquisition unit comprises an infrared sensor, a filter and a lens group, wherein the infrared sensor is located on a side of the filter away from the display panel, and the lens group is located on a side of the filter close to the display panel;

[0013] Preferably, the vehicle-mounted device further comprises a driving monitoring unit for acquiring a vehicle state, wherein the vehicle state comprises a second state;

[0014] Preferably, the driving monitoring unit further comprises a timing module, which is used to send a start signal to the image acquisition unit according to a preset period, and the image acquisition unit collects the video information according to the start signal;

[0015] Preferably, the vehicle-mounted device further comprises an alarm unit, which is used to output alarm information when the detection unit determines that the driver is in the first state;

[0016] Preferably, the first state is a fatigue state;

[0017] Preferably, the second state is a state where the vehicle is started.

[0018] In one embodiment, the face recognition unit includes:

[0019] A grayscale processing module is used to process the video information in grayscale to obtain grayscale video information;

[0020] The face detection module is used to identify the grayscale video information and obtain the first image information.

[0021] In one embodiment, it further includes an image processing unit, which is used to reconstruct the first image information to obtain second image information;

[0022] Preferably, the image processing unit performs super-resolution reconstruction on the first image information to obtain the second image information;

[0023] Preferably, the fatigue detection unit is used to determine whether the driver is in a first state based on the first image information.

[0024] In one embodiment, the pre-trained model includes an extraction layer and a feature information library, the extraction layer extracts feature information according to the first image information and / or the second image information, and the feature information library determines whether the driver is in the first state according to the feature information;

[0025] If the feature information matches the feature information library, the pre-trained model directly determines whether the driver is in the first state;

[0026] If the feature information does not match the feature information library, the optimization module issues a selection instruction, and the feature information library determines whether the driver is in the first state based on the input information corresponding to the selection instruction; and updates the feature information library in the pre-trained model based on the feature information and the input information;

[0027] Preferably, the input method of the input information includes the optimization module sending a selection instruction to the display panel, the display panel generating a selection instruction interface based on the selection instruction, the selection instruction interface instructing the driver to select whether to be in the first state based on the current state;

[0028] Preferably, the extraction layer extracts feature information of the first image and / or the second image based on a ResNet-18 architecture.

[0029] Based on the same inventive concept, the present disclosure also discloses a detection method for a vehicle-mounted device, the detection method comprising the following steps:

[0030] Acquire first image information, where the first image information includes face information;

[0031] Determining whether the driver is in a first state according to the first image information and the pre-trained model;

[0032] If it is not possible to directly determine whether the driver is in the first state based on the first image information, determine whether the driver is in the first state based on the input information, and update the pre-trained model based on the first image information and the input information.

[0033] In one embodiment, it further includes: if the driver is in the first state, sending an alarm signal to control the vehicle alarm;

[0034] Preferably, if the driver is in the first state, detecting whether the vehicle is in the second state;

[0035] Preferably, if the vehicle is in the second state, an alarm signal is sent to control the vehicle to alarm;

[0036] Preferably, the first state is a fatigue state;

[0037] Preferably, the second state is the vehicle startup state;

[0038] Preferably, if it is not possible to directly determine whether the driver is in the first state according to the first image information and the pre-trained model, determining whether the driver is in the first state based on the selection input information includes:

[0039] If it is not possible to directly determine whether the driver is in the first state based on the first image information and the pre-trained model, a selection instruction is issued, and whether the driver is in the first state is determined based on selection input information corresponding to the selection instruction.

[0040] In one embodiment, the step of acquiring the first image information includes:

[0041] Acquiring video information, wherein the video information includes an image of a driving area;

[0042] Acquire first image information according to the video information;

[0043] Preferably, when the vehicle is in the second state, video information is acquired according to a preset period;

[0044] Preferably, the acquiring the first image information according to the video information comprises:

[0045] grayscale processing the video information to obtain grayscale video information,

[0046] First image information is acquired according to the grayscale video information.

[0047] In one embodiment, before the step of determining whether the driver is in the first state according to the first image information, the step further includes:

[0048] Reconstructing the first image information to obtain second image information;

[0049] Preferably, super-resolution reconstruction is performed on the first image information to obtain second image information;

[0050] Preferably, whether the driver is in the first state is determined based on the first image information and / or the second image information.

[0051] In one embodiment, the step of determining whether the driver is in the first state according to the first image information and the pre-trained model includes:

[0052] Extracting features from the first image information to obtain feature information,

[0053] determining whether the driver is in a first state according to the characteristic information;

[0054] If the feature information matches the feature information library in the pre-trained model, directly determining whether the driver is in the first state;

[0055] If the feature information does not match the feature information library in the pre-trained model, issuing a selection instruction;

[0056] It is determined whether the driver is in the first state according to the input information corresponding to the selection indication, and the feature information library in the pre-trained model is updated based on the feature information and the input information.

[0057] Compared with the prior art, the vehicle-mounted device provided by the present invention improves the accuracy of fatigue detection by setting an optimization module, and further updates the pre-trained model. During the next fatigue detection process, the feature information obtained by the detection unit is matched with the new pre-trained model, and the detection unit can directly determine whether the current user is fatigued. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 A schematic cross-sectional view of a display panel provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of the structure of a vehicle-mounted device provided in an embodiment of the present application;

[0061] Figure 3 A flow chart of a detection method for a vehicle-mounted device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0063] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0064] The present application provides a vehicle-mounted device and a detection method for the vehicle-mounted device, the vehicle-mounted device includes a face recognition unit 300 and a fatigue detection unit 500, the face recognition unit 300 is used to obtain first image information, the first image information includes face information; the fatigue detection unit 500 is used to determine whether the driver is in a first state based on the first image information; the fatigue detection unit 500 includes a pre-trained model and an optimization module, if the pre-trained model cannot directly determine whether the driver is in the first state (the first state here can be a fatigue state) based on the first image information, the pre-trained model determines whether the driver is in the first state (fatigue state) based on the input information, and the optimization module updates the pre-trained model based on the first image information and the input information. The vehicle-mounted device provided by the present disclosure improves the accuracy of fatigue detection by setting an optimization module, and further updates the pre-trained model. In the next fatigue detection process, the feature information obtained by the detection unit matches the new pre-trained model, and the detection unit can directly determine whether the current user is fatigued.

[0065] Next, combine Figure 1-Figure 2 To describe the vehicle-mounted device provided by the present application.

[0066] In some embodiments, the vehicle-mounted device includes a display panel 100. The display panel 100 in this embodiment includes a first display area A1 and a second display area A2. The transmittance of the first display area A1 is greater than that of the second display area A2. The first display area A1 can be transparent. The first display area A1 is the UDIR (Under Display Infrared) area. The specific shapes, sizes, relative positions, etc. of the first display area A1 and the second display area A2 are not limited. For example, the second display area A2 can be located on at least one side of the first display area A1 in a certain direction, or it can be arranged around the second display area A2 on multiple sides of the first display area A1.

[0067] In some embodiments, the vehicle-mounted device includes an image acquisition unit 200, which is located on the backlight side of the display panel 100 and at least part of the orthographic projection of the image acquisition unit on the display panel 100 is located in the first display area A1. The image acquisition unit 200 is used to collect video information, and the video information includes a driving area image. It should be noted that the driving area image in this embodiment includes a face image, an upper body image, a cockpit image, etc. of the driver when driving. The image acquisition unit 200 in this embodiment can be a 3D infrared camera, and the outgoing light emitted by the image acquisition unit 200 (3D infrared camera) is irradiated on the driver's face through the first display area A1 of the display panel 100. Among them, the irradiated area can be understood as the area where the outgoing light emitted by the image acquisition unit 200 is irradiated on the display panel 100. Exemplarily, at least part of the orthographic projection of the image acquisition unit 200 on the display panel 100 is located in the first display area A1. For example, the orthographic projection of the image acquisition unit 200 on the display panel 100 is located in the middle of the first display area A1. In the prior art, most cameras for detecting driver fatigue driving are external cameras, and the user's driving comfort is not high during driving. Therefore, the present application sets the image acquisition unit 200 on the backlight side B2 of the display panel 100, which is beneficial to improving the driver's comfort while realizing fatigue detection. At the same time, the present application can recognize the face in the video information obtained by the image acquisition unit 200 in real time and determine whether the driver is in a fatigue state through the face recognition unit 300 and the fatigue detection unit 500.

[0068] In some embodiments, the display panel 100 may further include a non-display area, and the non-display area is disposed around the first display area A1 and the second display area A2.

[0069] In some embodiments, the display panel 100 includes a substrate 10 and a driving layer 20 , wherein the driving layer 20 is located on a side of the substrate 10 away from the image acquisition unit 200 , and the driving layer 20 includes a first thin film transistor T1 located in the first display area A1 , wherein the first thin film transistor T1 includes an active portion 21 located on the substrate 10 .

[0070] In some embodiments, the display panel 100 also includes a first light-shielding portion 30, wherein the first light-shielding portion 30 is located between the substrate 10 and the active portion 21, and is used to block the outgoing light (infrared light) emitted by the image acquisition unit 200 on the backlight side B2 of the display panel 100 from irradiating the active portion 21. Furthermore, the orthographic projection of the active portion on the substrate 10 is located within the orthographic projection range of the first light-shielding portion on the substrate 10, that is, the orthographic projection of the first light-shielding portion 30 on the substrate 10 must cover the orthographic projection of the active portion on the substrate 10, further preventing the outgoing light (red light) emitted by the image acquisition unit 200 on the backlight side B2 from irradiating the active portion, preventing the active portion from generating carriers, which may cause the display panel 100 corresponding to the driving substrate to produce display performance degradation, infrared irradiation flicker and other adverse phenomena.

[0071] In some embodiments, the display panel 100 also includes a plurality of light-emitting elements and an isolation structure 40, wherein the isolation structure 40 is located on a side of the driving layer 20 away from the substrate 10, and the isolation structure 40 is enclosed to form an isolation opening, wherein at least a portion of the light-emitting element is located within the isolation opening, and the light-emitting element includes a first light-emitting element 50 located in the first display area A1, wherein the first light-emitting element 50 is electrically connected to the first transistor. Exemplarily, the first light-emitting element 50 includes a first electrode 51 located on a side of the driving layer 20 away from the substrate 10, a light-emitting functional layer 52 located on a side of the first electrode 51 away from the substrate 10, and a second electrode 53 located on a side of the light-emitting functional layer 52 away from the substrate 10, wherein the second electrode 53 is electrically connected to the isolation structure 40, and the first electrode 51 is electrically connected to the first thin film transistor T1. Exemplarily, in this embodiment, the first electrode 51 may be an anode of the first light-emitting element 50, and the second electrode 53 may be a cathode of the first light-emitting element 50.

[0072] In some embodiments, the image acquisition unit 200 includes an infrared sensor, a filter, and a lens group. The infrared sensor is located on the side of the filter away from the display panel 100, and the lens group is located on the side of the filter close to the display panel 100. It should be noted that the image acquisition unit 200 is used to perform biometric identification such as face and fingerprint. For example, the image acquisition unit 200 in the present application includes a 3D infrared camera for face recognition. In the application, the image acquisition unit 200 emits outgoing light to the object to be identified located on the light-emitting side B1 of the display panel 100. Part of the outgoing light passes through the first display area A1 and is reflected by the surface of the object to be identified to form reflected light. Part of the reflected light passes through the first display area A1 and is received by the image acquisition unit 200, thereby realizing the detection and identification of the biometric features of the object to be identified.

[0073] In one embodiment, the vehicle-mounted device includes a face recognition unit 300, which is used to obtain first image information. The first image information includes face information, that is, the face recognition unit 300 is used to recognize video information and obtain a first image. In this embodiment, recognizing video information specifically refers to detecting a human figure in an image or a frame of video information and separating the face from the background to obtain face information. Specifically, the face recognition unit 300 includes a grayscale processing module and a face detection module. The grayscale processing module converts video information into grayscale video information. The grayscale processing module in this embodiment can be the cv2.cvtColor function in OpenCV to convert video information into grayscale video information; the face detection module is used to recognize the grayscale video stream and obtain the first image. Specifically, the first image is a face image recognized by the face detection module. The face detection module detects the face and its coordinates of the grayscale video information and automatically saves it. The face detection module in this embodiment can be a Dlib module, which is a C++ open source toolkit containing machine learning algorithms. The Dlib module can help create a large number of complex machine learning software to solve practical problems. Currently, the Dlib module is widely used in industrial and academic fields, including robots, embedded devices, mobile phones, and large high-performance computing environments. The predetermined ratio of the margin is to add a 40% margin to frame the entire head, and the Dlib module can work effectively in a variety of environments and adapt to different lighting conditions.

[0074] Of course, it should be noted that due to the difference in grayscale, the grayscale of the eyes, mouth, hair, contour, etc. of the face in the first image is relatively low, and there is a large gradient between the grayscale and the surrounding parts, and the grayscale features are very obvious. Based on this, this embodiment will pre-process the image before converting it into a grayscale image, and the pre-processing includes: smoothing to remove noise in the image, sharpening to enhance the edge of the image, binarization, etc.

[0075] In some embodiments, the vehicle-mounted device further includes a driving monitoring unit, which is used to obtain the vehicle state. The exemplary vehicle state is in the second state (the second state here is the vehicle start state) or in a stationary state. If the detection unit detects that the driver is in the first state (fatigue state), and the driving monitoring unit detects that the vehicle is in the second state (vehicle start state), the alarm unit outputs an alarm message. In this way, the designer achieves a double protection effect to avoid false alarms. Further, the driving monitoring unit also includes a timing module, which sends a start signal to the image acquisition unit according to a preset period, and the image acquisition unit collects video information according to the start signal, such as performing a fatigue detection on the driver every preset time (such as 30-60 minutes). Preferably, the image acquisition unit collects video information according to a period of 30 minutes, that is, the detection unit performs a fatigue detection on the driver every 30 minutes.

[0076] In some embodiments, the vehicle-mounted device also includes an image processing unit 400, which is used to reconstruct the first image and obtain the second image. Furthermore, the image processing unit 400 performs super-resolution reconstruction on the first image to obtain the second image; for example, the face recognition unit 300 inputs the obtained first image into the generative model of the Real-ESRGAN network, and uses the generative model of the Real-ESRGAN network to perform super-resolution reconstruction on the low-resolution first image. Then, the fatigue detection unit 500 can extract feature information of the second image, and judge whether the user is tired based on the feature information. Super-resolution reconstruction is performed on the first image before the fatigue detection unit 500. The super-resolution reconstruction in this embodiment specifically refers to an image processing technology that processes a low-resolution image (low resolution, LR) to restore a high-resolution image (highresolution, HR). By adopting super-resolution reconstruction, the clarity and details of the first image can be improved, thereby improving the detection accuracy.

[0077] In some embodiments, the pre-trained model in the fatigue detection unit 500 is used to extract feature information of the first image information and / or the second image information. Preferably, the pre-trained model in the fatigue detection unit 500 is used to extract feature information of the second image information, and judge whether the user is fatigued based on the feature information, and the feature information specifically includes eye features and mouth features, wherein the eye features may include at least one of pupil features, sclera features, eye saccade features, blink features, and gaze features.

[0078] Furthermore, the fatigue detection unit 500 includes a pre-trained model, which includes a pre-trained ResNet-18 model. It should be noted that the ResNet (Residual Network) in this embodiment is a deep neural network structure proposed by Microsoft Research Asia, which aims to solve the problems of gradient disappearance and gradient explosion in deep network training, so that the network can be trained deeper and have stronger performance. The core idea of ​​ResNet is to pass the input signal directly to the subsequent layers through residual connections, so that the network can learn residuals instead of global features. ResNet has two main block structures: BasicBlock and Bottleneck Block. BasicBlock is suitable for shallow networks (such as ResNet18 and ResNet34), while Bottleneck Block is suitable for deep networks (such as ResNet50, ResNet101 and ResNet152).

[0079] The pre-trained model includes an extraction layer and a feature information library, the extraction layer in the pre-trained model extracts feature information according to the first image information and / or the second image information, the feature information library determines whether the driver is in the first state according to the feature information, and the extraction layer in the pre-trained model extracts feature information in the first image and / or the second image based on the ResNet-18 architecture; preferably, the extraction layer in the pre-trained model extracts feature information in the second image based on the ResNet-18 architecture, and the feature information library determines whether the driver is in the first state according to the feature information; wherein the pre-trained model is a convolutional neural network model established based on the ResNet-18 architecture, wherein ResNet-18 is actually built in PyTorch, which is much wider than other architectures. It can be characterized by a small convolution filter of 3x3 pixels. Therefore, each filter in ResNet-18 can capture simpler geometric structures, and more complex reasoning can be performed by increasing the width compared to other algorithms, reducing learning loss and improving performance. If the feature information matches the feature information library in the pre-trained model, the feature information library in the pre-trained model directly determines whether the driver is in the first state, wherein the feature information library includes a first feature information sub-library and a second feature information sub-library, the first feature information sub-library may also be called a fatigue feature information sub-library, and the second feature information sub-library may also be called a non-fatigue feature information sub-library. If the feature information matches the first feature information sub-library, the driver is in the first state (fatigue state); if the feature information matches the second feature information sub-library, the driver is in a non-fatigue state.

[0080] The above-mentioned pre-trained model specifically refers to collecting feature information of a large number of users in different environments when they are tired and when they are not tired, and using the collected feature information as training data for the pre-trained module to obtain a model of eye features and / or mouth features and visual fatigue degree. The currently extracted feature information is compared with the pre-trained model to determine whether the driver is in the first state (fatigue state).

[0081] Furthermore, before extracting the feature information, the second image is preprocessed, and the preprocessing includes adjusting the size of the second image, converting the second image into a tensor and standardizing it, and disabling gradient calculation during the comparison process of the pre-trained model, which can improve the inference efficiency.

[0082] Furthermore, the fatigue detection unit 500 also includes an optimization module. Since the feature information library in the pre-trained model has learned (collected) feature information of a large number of users, but there are differences among individual users during the detection process of the detection unit or the feature information obtained by the fatigue detection unit 500 is not in the pre-trained model, the detection unit will not be able to determine whether the driver is in the first state. Therefore, the pre-trained model needs to be continuously optimized (updated) during the real-time detection process, that is, the optimization module includes feature information that is not in the feature information library of the pre-trained model into the feature information library.

[0083] If the pre-trained model cannot directly determine whether the driver is in the first state (fatigue state) based on the feature information extracted from the first image information and / or the second image information, that is, the feature information does not match the feature information library, the optimization module will issue a selection instruction to the driver. The feature information library in the pre-trained model determines whether the driver is in the first state based on the selection input information corresponding to the selection instruction, and the optimization module updates the feature information library based on the feature information extracted from the first image information and / or the second image information and the input information, that is, updates the feature information in this state to the feature information library. Furthermore, the selection instruction in this embodiment includes instructing the driver to select whether to be in the first state. It should be noted that if the detection unit cannot identify whether the driver is in the first state, the optimization module in the detection unit determines whether the driver is in the first state based on the input information. Specifically, the input method of the input information in this embodiment includes the optimization module in the detection unit sending a selection instruction to the display panel 100, the display panel 100 generates a selection instruction interface based on the selection instruction, and the selection instruction interface instructs the driver to select whether to be in the first state based on the current state; the optimization module can also send a selection instruction to the driver based on the voice broadcast, that is, the optimization module sends a voice selection signal to the alarm unit, the alarm unit broadcasts the selection instruction based on the interface voice selection signal, and the driver sends a voice reply. If the driver selects the first state, the optimization module will enter the user's feature information at this time into the fatigue feature information sub-library, and enter the feature information in this state into the first state. If the user chooses the non-fatigue state, the optimization module will enter the user's feature information at this time into the non-fatigue feature information sub-library, and enter the feature information in this state into the non-fatigue state. This design is conducive to improving the accuracy of fatigue detection, and further optimizes the pre-trained model. In the next fatigue detection process, the feature information obtained by the detection unit is matched with the feature information library in the new pre-trained model, and the detection unit can directly determine whether the current user is fatigued.

[0084] This device achieves real-time face recognition, super-resolution enhancement, and fatigue detection by integrating multiple open source models (such as OpenCV, Dlib, Real-ESRGAN, and PyTorch). Each step uses existing high-performance models to ensure that the system reaches a high level of accuracy and efficiency, and the models can be replaced or adjusted to suit specific application scenarios.

[0085] Furthermore, in one embodiment, the vehicle-mounted device also includes an alarm unit 600, which is used to output alarm information for alerting the user that the driver is in the first state when the detection unit determines that the driver is in the first state. When the detection unit detects that the user is in the first state, the detection unit will output an alarm signal to the alarm unit 600 on the one hand, and will also output a page reminder signal to the display panel 100 on the other hand.

[0086] Based on the same inventive concept, the present application also provides a detection method for a vehicle-mounted device. Figure 2-Figure 3 To describe the detection method of the vehicle-mounted device provided by the present application, the detection method includes:

[0087] S10: Acquire first image information, where the first image information includes face information;

[0088] S20: determining whether the driver is in the first state according to the first image information;

[0089] If it is not possible to directly determine whether the driver is in the first state based on the first image information, determine whether the driver is in the first state based on the input information, and update the pre-trained model based on the first image and the input information. This detection method is beneficial to improving the accuracy of fatigue detection. At the same time, the pre-trained model is further optimized. During the next fatigue detection process, the detection unit can directly determine whether the driver is in the first state (fatigue state). It should be noted that the input information input method in this embodiment includes an optimization module sending a selection indication to the display panel, and the display panel generates a selection indication interface based on the selection indication. The selection indication interface instructs the driver to select whether to be in the first state based on the current state, that is, the input information in this embodiment is based on the driver's selection result for the current state.

[0090] In one embodiment, if it is not possible to directly determine whether the driver is in the first state based on the first image information, determining whether the driver is in the first state based on the input information specifically means that if it is not possible to directly determine whether the driver is in the first state based on the first image information and the pre-trained model, a selection indication is issued, and whether the driver is in the first state is determined based on the selection input information corresponding to the selection indication.

[0091] In one embodiment, the detection method further comprises:

[0092] If the driver is in the first state, an alarm signal is sent to control the vehicle alarm;

[0093] Preferably, if the driver is in the first state, detecting whether the vehicle is in the second state (vehicle start state);

[0094] If the vehicle is in the second state, an alarm signal is sent to control the vehicle to alarm.

[0095] In one embodiment, acquiring the first image in step S10 includes:

[0096] S11: Obtaining video information, the video information includes a driving area image. It should be noted that the driving area image in this embodiment includes a face image, an upper body image, a cockpit image, etc. of the driver while driving.

[0097] In this embodiment, video information is acquired based on the image acquisition unit 200, and the image acquisition unit is located on the backlight side of the display panel 100; since the image acquisition unit 200 is located on the backlight side B2 of the display panel 100, no discomfort will be caused to human vision when acquiring video information, thereby improving the user's comfort.

[0098] Further, when the vehicle is in the second state (vehicle startup state), video information is acquired according to a preset period, that is, the image acquisition unit 200 acquires video information according to a preset period when the vehicle is in the startup state, such as acquiring video information every 30 minutes or every 60 minutes.

[0099] S12: Acquire first image information according to the video information;

[0100] In this implementation, face recognition is performed on the video information based on the face recognition unit 300 to obtain a first image;

[0101] Specifically, step S12 includes:

[0102] S121: grayscale processing video information to obtain grayscale video information;

[0103] In this step, the grayscale processing of the video information is mainly performed through the grayscale processing module, wherein the grayscale processing module can be the cv2.cvtColor function in OpenCV to convert the video information into grayscale video information;

[0104] S122: Obtain first image information according to the grayscale video information.

[0105] In this step, the face detection module detects the face and its coordinates in the grayscale video information. The face detection module in this implementation manner may be a Dlib module.

[0106] In one embodiment, determining whether the driver is in the first state (fatigue state) according to the first image information in step S20 specifically includes:

[0107] If the driver is in the first state, sending an alarm signal to control the vehicle to output an alarm message;

[0108] Further, if the driver is in the first state, detecting whether the vehicle is in the second state (vehicle start state);

[0109] If the vehicle is in the second state (vehicle start state), an alarm signal is sent to control the vehicle alarm. This design plays a double protection role to avoid false alarms.

[0110] In some embodiments, before step S20, the step further includes

[0111] S200: Reconstruct the first image information to obtain second image information;

[0112] Reconstructing the first image information based on the image processing unit 400 to obtain second image information;

[0113] Furthermore, in this step, the first image information is super-reconstructed at a high resolution to obtain the second image information, that is, the first image information is super-reconstructed at a high resolution based on the image processing unit 400 to obtain the second image information. The image processing unit 400 in this embodiment can be a generative model of a Real-ESRGAN network, and the generative model of the Real-ESRGAN network is used to perform super-resolution reconstruction on the image of the low-resolution first image to obtain the second image. The super-resolution reconstruction of the first image before the fatigue detection unit 500 can improve the clarity and details of the first image, thereby improving the detection accuracy.

[0114] In one embodiment, step S20 includes:

[0115] Extract features from the first image information to obtain feature information.

[0116] That is, feature extraction is performed on the first image information and / or the second image information based on the extraction layer in the pre-trained model (preferably, feature extraction is performed on the second image information based on the pre-trained model) to obtain feature information;

[0117] determining whether the driver is in a first state according to the characteristic information;

[0118] That is, the characteristic information determines whether the driver is in the first state (fatigue state) based on the characteristic information library;

[0119] If the feature information matches the feature information library in the pre-trained model, it is directly determined whether the driver is in the first state; if the feature information does not match the feature information library in the pre-trained model, a selection instruction is issued;

[0120] determining whether the driver is in the first state according to the selection input information corresponding to the selection indication, and updating the pre-trained model based on the feature information and the input information;

[0121] That is, if the feature information does not match the feature information library, the feature information library in the pre-trained model transmits instructions to the optimization module, and the optimization module issues a selection instruction. The feature information library determines whether the driver is in the first state (fatigue state) based on the input information received by the optimization module. At the same time, the optimization module updates the feature information library in the pre-trained model based on the feature information and the input information received.

[0122] Based on the same inventive concept, the present application also provides a terminal device, which includes the vehicle-mounted device in any of the above embodiments and has the beneficial effects of the vehicle-mounted device proposed in any of the above embodiments, which will not be repeated here. By way of example, the smart terminal can be a new energy vehicle.

[0123] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A vehicle-mounted device, characterized in that: include: A face recognition unit, used to obtain first image information, where the first image information includes face information; a fatigue detection unit, configured to determine whether the driver is in a first state according to the first image information; The fatigue detection unit includes a pre-trained model and an optimization module. If the pre-trained model cannot directly determine whether the driver is in the first state based on the first image information, the pre-trained model determines whether the driver is in the first state based on input information, and the optimization module updates the pre-trained model based on the first image information and the input information.

2. The vehicle-mounted device according to claim 1, characterized in that: Also includes: The display panel comprises a first display area and a second display area, the light transmittance of the first display area is greater than the light transmittance of the second display area, and the display panel comprises a light emitting side and a backlight side which are arranged opposite to each other; an image acquisition unit, located at a backlight side of the display panel and at least a portion of a positive projection of the image acquisition unit on the display panel is located in the first display area, the image acquisition unit being used to collect video information, the video information including a driving area image; Preferably, the image acquisition unit comprises an infrared sensor, a filter and a lens group, wherein the infrared sensor is located on a side of the filter away from the display panel, and the lens group is located on a side of the filter close to the display panel; Preferably, the vehicle-mounted device further comprises a driving monitoring unit for acquiring a vehicle state, wherein the vehicle state comprises a second state; Preferably, the driving monitoring unit further comprises a timing module, which is used to send a start signal to the image acquisition unit according to a preset period, and the image acquisition unit collects the video information according to the start signal; Preferably, the vehicle-mounted device further comprises an alarm unit, which is used to output alarm information when the detection unit determines that the driver is in the first state; Preferably, the first state is a fatigue state; Preferably, the second state is a state where the vehicle is started.

3. The vehicle-mounted device according to claim 2, wherein: The face recognition unit comprises: A grayscale processing module is used to process the video information in grayscale to obtain grayscale video information; The face detection module is used to identify the grayscale video information and obtain the first image information.

4. The vehicle-mounted device according to claim 2, wherein: It also includes an image processing unit, which is used to reconstruct the first image information and obtain second image information; Preferably, the image processing unit performs super-resolution reconstruction on the first image information to obtain second image information; Preferably, the fatigue detection unit is used to determine whether the driver is in the first state according to the second image information.

5. The vehicle-mounted device according to claim 4, characterized in that: The pre-trained model includes an extraction layer and a feature information library, the extraction layer extracts feature information according to the first image information and / or the second image information, and the feature information library determines whether the driver is in the first state according to the feature information; If the feature information matches the feature information library, the feature information library in the pre-trained model directly determines whether the driver is in the first state; If the feature information does not match the feature information library, the optimization module issues a selection instruction, and the feature information library determines whether the driver is in the first state based on the input information corresponding to the selection instruction; and updates the feature information library in the pre-trained model based on the feature information and the input information; Preferably, the input method of the input information includes the optimization module sending a selection instruction to the display panel, the display panel generating a selection instruction interface based on the selection instruction, the selection instruction interface instructing the driver to select whether to be in the first state based on the current state; Preferably, the extraction layer extracts feature information of the first image and / or the second image based on a ResNet-18 architecture.

6. A detection method for a vehicle-mounted device, characterized in that: The steps include: Acquire first image information, where the first image information includes face information; Determining whether the driver is in a first state according to the first image information and the pre-trained model; If it is not possible to directly determine whether the driver is in the first state based on the first image information and the pre-trained model, determine whether the driver is in the first state based on the input information, and update the pre-trained model based on the first image information and the input information.

7. The detection method according to claim 6, characterized in that Also includes: If the driver is in the first state, sending an alarm signal to control the vehicle alarm; Preferably, if the driver is in the first state, detecting whether the vehicle is in the second state; If the vehicle is in the second state, sending an alarm signal to control the vehicle to alarm; Preferably, the first state is a fatigue state; Preferably, the second state is the vehicle startup state; Preferably, if it is not possible to directly determine whether the driver is in the first state according to the first image information and the pre-trained model, determining whether the driver is in the first state based on the selection input information includes: If it is not possible to directly determine whether the driver is in the first state based on the first image information and the pre-trained model, a selection instruction is issued, and whether the driver is in the first state is determined based on input information corresponding to the selection instruction.

8. The detection method according to claim 7, characterized in that The acquiring of the first image information comprises: Acquiring video information, wherein the video information includes an image of a driving area; Acquire first image information according to the video information; Preferably, when the vehicle is in the second state, video information is acquired according to a preset period; Preferably, the acquiring the first image information according to the video information comprises: grayscale processing the video information to obtain grayscale video information, First image information is acquired according to the grayscale video information.

9. The detection method according to claim 6, characterized in that: Before the step of determining whether the driver is in the first state according to the first image information, the step further includes: Reconstructing the first image information to obtain second image information; Preferably, super-resolution reconstruction is performed on the first image information to obtain second image information; Preferably, whether the driver is in the first state is determined based on the first image information and / or the second image information.

10. The detection method according to claim 6, characterized in that: The step of determining whether the driver is in the first state according to the first image information and the pre-trained model comprises: Extracting features from the first image information to obtain feature information, determining whether the driver is in a first state according to the characteristic information; If the feature information matches the feature information library in the pre-trained model, directly determining whether the driver is in the first state; If the feature information does not match the feature information library in the pre-trained model, issuing a selection instruction; It is determined whether the driver is in the first state according to the input information corresponding to the selection indication, and the feature information library in the pre-trained model is updated based on the feature information and the input information.

Images