A driver state monitoring method, device and electronic equipment

Abstract

The application discloses a driver state monitoring method and device and electronic equipment, wherein the driver state monitoring method can be applied to the automatic driving field, a camera for monitoring the driver state is installed above a steering wheel steering column, and the method comprises the following steps: acquiring a first monitoring image for monitoring the driver state; when a face in the first monitoring image is incomplete, determining an adjusted height value of the steering wheel and an expected height based on a missing part of the face in the first monitoring image; adjusting the height of the steering wheel based on the adjusted height value; acquiring a second monitoring image for monitoring the driver state, and determining an adjusted height of the steering wheel based on the second monitoring image; determining a difference between the adjusted height and the expected height, and when the difference meets a preset error requirement, detecting whether the face in the second monitoring image is complete, and until the face in the second monitoring image is complete, the monitoring adjustment state is exited. The application realizes the camera position adjustment suitable for the driving sitting posture of various people.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a driver state monitoring method, device, and electronic device. Background Technology

[0002] The driver status monitoring system is part of the human-machine interaction of autonomous driving. The system uses images acquired by cameras and data input from other vehicle sensors to monitor the driver's driving behavior and physiological state through technologies such as visual tracking and motion recognition. When it determines that the driver is not present or is in an abnormal driving state (fatigue, distraction, etc.), the autonomous driving system issues an alarm to the driver or executes other safety strategies to ensure the safe operation of the vehicle.

[0003] The current driver status monitoring camera layout has issues where the steering wheel airbag cover obstructs the camera's field of view. This increases the likelihood of the driver's face being outside the camera's field of view due to varying seating positions, causing the driver status monitoring system to fail to provide accurate driving status detection for different drivers. This results in false alarms and missed alarms for abnormal driving conditions, affecting safe driving. Summary of the Invention

[0004] To address the problems of existing technologies, this application provides a driver status monitoring method, device, and electronic device. The technical solution is as follows:

[0005] On one hand, a driver state monitoring method is provided, wherein a camera for monitoring the driver's state is mounted above the steering column of the steering wheel, the method comprising:

[0006] Acquire the first monitoring image used to monitor the driver's condition;

[0007] When the face in the first monitoring image is incomplete, the adjustment height value and expected height of the steering wheel are determined based on the missing part of the face in the first monitoring image;

[0008] Adjust the height of the steering wheel based on the aforementioned height adjustment value;

[0009] Acquire a second monitoring image for monitoring the driver's state, and determine the adjusted height of the steering wheel based on the second monitoring image;

[0010] The difference between the adjusted height and the expected height is determined. When the difference meets the preset error requirement, the face in the second monitoring image is checked for completeness. The monitoring and adjustment state is exited when the face in the second monitoring image is complete.

[0011] On the other hand, a driver state monitoring device is provided, wherein a camera for monitoring the driver's state is mounted above the steering column of the steering wheel, the device comprising:

[0012] The image acquisition module is used to acquire the first monitoring image for monitoring the driver's condition;

[0013] The adjustment determination module is used to determine the adjustment height value and expected height of the steering wheel based on the missing part of the face in the first monitoring image when the face in the first monitoring image is incomplete.

[0014] A height adjustment module is used to adjust the height of the steering wheel based on the adjusted height value;

[0015] A height determination module is used to acquire a second monitoring image for monitoring the driver's state, and to determine the adjusted height of the steering wheel based on the second monitoring image.

[0016] The adjustment detection module is used to determine the difference between the adjusted height and the expected height. When the difference meets the preset error requirement, it detects whether the face in the second monitoring image is complete. The monitoring adjustment state is exited when the face in the second monitoring image is complete.

[0017] In one exemplary embodiment, the adjustment determination module includes:

[0018] The feature extraction module is used to extract facial feature points of the driver based on the first monitoring image;

[0019] The quantity statistics module is used to determine the number of facial feature points of the driver;

[0020] The missing feature point determination module is used to determine the missing feature points in the driver's facial feature points when the number does not meet the preset number of feature points;

[0021] A part determination module is used to determine the missing facial parts based on the missing feature points;

[0022] The adjustment value determination module is used to determine the adjustment height value of the steering wheel based on the missing facial features.

[0023] In one exemplary embodiment, the adjustment determination module includes:

[0024] The first selection module is used to select a first key point based on the first monitoring image; the first key point is the edge point of the fixed interior trim of the vehicle.

[0025] The first acquisition module is used to acquire the first original coordinates and the original view coordinates of the first key point; the first original coordinates are obtained by transforming the coordinates of the first key point in the original image coordinate system to the vehicle coordinate system; the original image coordinate system is the coordinate system of the image acquired by the camera when it is in the factory position.

[0026] The first conversion module is used to convert the coordinates of the first key point in the coordinate system of the first monitoring image to the coordinate system of the whole vehicle to obtain the first current coordinates;

[0027] The first change module is used to determine the amount of change between the first original coordinates and the first current coordinates, and to obtain the first change amount;

[0028] The first coordinate module is used to determine the coordinates of the viewpoint when acquiring the first monitoring image based on the first change amount and the original viewpoint coordinates, and obtain the first viewpoint coordinates;

[0029] The expected determination module is used to determine the expected coordinates of the viewpoint based on the first viewpoint coordinates and the missing facial portion; the expected coordinates indicate the expected height.

[0030] In one exemplary implementation, the height determination module includes:

[0031] The second selection module is used to select a second key point based on the second monitoring image; the second key point is the edge point of the fixed interior trim of the vehicle.

[0032] The second acquisition module is used to acquire the second original coordinates of the second key point; the second original coordinates are obtained by transforming the coordinates of the second key point in the original image coordinate system to the vehicle coordinate system.

[0033] The second conversion module is used to convert the coordinates of the second key point in the coordinate system of the second monitoring image to the coordinate system of the whole vehicle to obtain the second current coordinates;

[0034] The second change module is used to determine the amount of change between the second original coordinates and the second current coordinates, and to obtain the second change amount;

[0035] The second coordinate module is used to determine the coordinates of the viewpoint when acquiring the second monitoring image based on the second change amount and the original viewpoint coordinates, and obtain the second viewpoint coordinates; the second viewpoint coordinates indicate the adjusted height.

[0036] In one exemplary embodiment, the adjustment detection module includes:

[0037] The difference determination module is used to determine the difference between the second viewpoint coordinates and the expected coordinates;

[0038] The image detection module is used to detect whether the face in the second monitoring image is complete when the difference meets the preset error requirement, and exit the monitoring adjustment state when the face in the second monitoring image is complete.

[0039] In one exemplary embodiment, the device further includes a screen display module for displaying an adjustment state on an in-vehicle screen, the screen display module comprising:

[0040] A line-of-sight determination module is used to determine the driver's line of sight based on the second monitoring image;

[0041] The display module is completed to display monitoring and adjustment when the line of sight intersects with the vehicle screen.

[0042] In one exemplary embodiment, the device further includes an adjustment exit module for exiting the monitoring adjustment state without needing to adjust the steering wheel height, the adjustment exit module comprising:

[0043] The adjustment exit module is used to exit the monitoring adjustment state when the face in the first monitoring image is complete.

[0044] In one exemplary embodiment, the device further includes a readjustment module for readjusting the steering wheel height when the difference does not meet a preset error requirement, the readjustment module comprising:

[0045] The update adjustment module is used to update the adjustment height value based on the difference when the difference does not meet the preset error requirement, and to update the difference based on the second monitoring image corresponding to the updated adjustment height value, until the updated difference meets the preset error requirement, and then stop the update.

[0046] The update detection module is used to detect whether the face in the updated second monitoring image is complete, and exits the monitoring adjustment state when the face in the updated second monitoring image is complete.

[0047] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the driver state monitoring method of any of the above aspects.

[0048] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the driver state monitoring method as described above.

[0049] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the driver state monitoring method of any of the above aspects.

[0050] This application embodiment automatically detects whether the driver's face is complete by capturing images from a driver status monitoring camera. Based on the missing parts, it infers the adjustment direction and height, and adjusts the camera by adjusting the steering wheel. This achieves camera position adjustment that adapts to the driving posture of various people, taking into account intelligence, accuracy, and adaptability. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic flowchart of a driver status monitoring method provided in an embodiment of this application;

[0053] Figure 2 This is a schematic block diagram of a driver status monitoring system provided in an embodiment of this application;

[0054] Figure 3 This is a schematic flowchart of a face detection method provided in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram illustrating the determination of adjustable height values ​​based on facial feature points provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of the self-calibration process of a driver status monitoring camera provided in an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the self-calibration process of a driver status monitoring camera provided in an embodiment of this application;

[0058] Figure 7This is a structural block diagram of a driver status monitoring device provided in an embodiment of this application;

[0059] Figure 8 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0061] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0062] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0063] Please see Figure 1 The diagram shown is a flowchart illustrating a driver status monitoring method provided in an embodiment of this application. This method can be applied to... Figure 1 Driver status monitoring in [the context of the document]. It should be noted that this specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many, and does not represent the only execution order. In actual system or product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 1 As shown, the method may include:

[0064] S101, acquire the first monitoring image for monitoring the driver's condition.

[0065] The first monitoring image is an image captured by the driver status monitoring camera in the driver status monitoring system at the current time. The driver status monitoring system can monitor the driver's status through this image. When it is determined that the driver is not present or is in an abnormal driving state (fatigue, distraction, etc.), the system will issue an alarm to the driver or implement other safety strategies to ensure the safe operation of the vehicle.

[0066] In this embodiment, the driver status monitoring camera is mounted above the steering column. The height of the camera is adjusted by adjusting the steering wheel height. Specifically, as follows: Figure 2 The diagram shows a principle block diagram of a driver state monitoring system. The driver turns on the adjustment switch of the DMS (Driver Monitor System) via the DHU (Display Head Unit). The DMS first self-calibrates the position of the DMC (Driver Monitor Camera), then extracts facial contour feature points from the image currently captured by the DMC. If the number of facial contour feature points is insufficient, it is determined that the driver's face in the image captured by the DMC is incomplete and cannot accurately monitor the driver's state. The DMS sends a position adjustment request signal to the ASWM (Adjustable Steering Wheel Module). The ASWM adjusts the position of the DMC by adjusting the steering wheel height and sends a position adjustment completion status signal back to the DMS. If the number of facial contour feature points meets the normal requirement, it is determined that the driver's face captured by the DMC is complete, and no adjustment of the DMC position is needed. The DMS monitors the driver's line of sight relative to the DIM / CSD (Driver Information Module / Center Stack). When there is an intersection with the display, the Eyedetected signal is output to the DHU via the flexray (vehicle network standard) bus. The DHU displays "Adjustment OK" on the DIM / CSD. The driver can then click to exit the adjustment interface or wait for the system to automatically exit the adjustment state.

[0067] S103, Detect whether the face in the first monitoring image is complete.

[0068] Specifically, if the detection result is yes, step S105 can be executed; otherwise, if the detection result is no, step S107 can be executed.

[0069] Specifically, the current detection image is used to detect the driver's state. Whether the current detection image can completely cover the driver's face is related to whether the driver state monitoring system can accurately detect the driver's state.

[0070] Specifically, when the face in the first monitoring image is complete, the driver state monitoring system exits the adjustment state; when the driver's face in the first monitoring image is incomplete, the steering wheel height is adjusted based on the missing facial parts until the face in the monitoring image acquired by the driver state monitoring camera is complete, at which point the driver state monitoring system exits the adjustment state.

[0071] S105, Exit monitoring and adjustment mode.

[0072] Specifically, when the driver's face is detected as intact in the monitoring footage captured by the driver status monitoring camera, there is no need to adjust the camera position, i.e., no need to adjust the steering wheel height. At this point, the adjustment state of the driver status monitoring system is exited.

[0073] S107, determine the adjustment height value and expected height of the steering wheel based on the missing facial portion in the first monitoring image.

[0074] The missing facial portion could be a part of the driver's face that is outside the field of view of the driver status monitoring camera, or it could be a part that is obscured by the vehicle's interior.

[0075] The adjustment height value includes the adjustment direction and the adjustment distance. For example, when the driver's lips and below are missing in the first monitoring image, the adjustment direction is downward. Since the distance from the highest point of the lips to the chin on a normal human face is 10 to 12 centimeters, and based on the camera parameters, the adjustment distance of the steering wheel in this case is set to 10 millimeters, then the adjustment height value is -10 millimeters.

[0076] The expected height is determined based on the current height of the steering wheel and the adjusted height value, serving as a reference value for the height of the steering wheel after adjustment. Only when the difference between the two is negligible can the accuracy of the steering wheel motor be disregarded.

[0077] In one exemplary implementation, such as Figure 3 The diagram shown is a flowchart of a face detection method. Steps S103 to S107 may include:

[0078] S301, extract facial feature points of the driver based on the first monitoring image.

[0079] Among them, facial feature points are the special points of the driver's face, including the facial contours of the eyebrows, eyes, nose, mouth, and chin.

[0080] S303, determine the number of facial feature points of the driver.

[0081] Specifically, the number of complete driver facial feature points is pre-set in the driver status monitoring system, and the number of feature points extracted in the first monitoring image is used to characterize whether the driver's facial information is complete.

[0082] S305, determine whether the quantity meets the preset number of feature points.

[0083] Specifically, if the result of the judgment is yes, then step S307 can be executed; otherwise, if the result of the judgment is no, then step S309 can be executed.

[0084] The preset number of feature points is the number of facial feature points that the driver's face can be extracted by the driver's state monitoring system when the driver's face is complete in the detection image captured by the driver's state monitoring camera. This value is stored in advance in the driver's state monitoring system.

[0085] Specifically, when the number of feature points meets the preset number of feature points, the driver's face in the current detection image is complete, and the driver state monitoring system exits the adjustment state; when the number of feature points does not meet the preset number of feature points, the driver's face in the first monitoring image is incomplete, and the steering wheel adjustment height value is determined based on the missing facial feature points to adjust the steering wheel height until the number of feature points meets the preset number of feature points, at which point the driver state monitoring system exits the adjustment state.

[0086] S307, Exit monitoring and adjustment mode.

[0087] Specifically, when the driver status monitoring system detects that the driver's face is intact in the monitoring image captured by the driver status monitoring camera, there is no need to adjust the position of the driver status monitoring camera, that is, there is no need to adjust the steering wheel height. Therefore, the driver status monitoring system exits the adjustment state.

[0088] S309, determine the missing feature points in the driver's facial feature points.

[0089] The missing feature points refer to the driver's facial feature points extracted by the driver's state monitoring system in the current state detection image, compared to the driver's facial feature points that the system could extract when the driver's face was complete in the detection image captured by the driver's state monitoring camera. Specifically, the driver's state monitoring system extracts the driver's facial feature points based on the first monitoring image transmitted by the driver's state monitoring camera, such as... Figure 4As shown, each feature point has a corresponding number and its position in the coordinate system of the first monitoring image. Feature points that were not captured by the driver status monitoring camera and therefore cannot be extracted are output with a default value of 0. The driver status monitoring system uses this to determine which specific feature points were not captured.

[0090] S3011, Determine the missing facial features based on the missing feature points.

[0091] Specifically, the missing facial features are determined by the number corresponding to the feature point with a coordinate value of 0.

[0092] S3013, determine the adjustment height value of the steering wheel based on the missing facial portion.

[0093] The adjustment height value includes both the adjustment direction and the adjustment distance. For example, when the driver's nose tip and below are missing in the first monitoring image, the adjustment direction is downward. Based on the normal distance from the nose tip to the chin of a human face being 15-18 cm, combined with camera parameters and large-sample real-vehicle test results, the steering wheel adjustment distance is set to 15 mm in this case, resulting in an adjustment height value of -15 mm. When the missing part of the driver's face in the current detection image is the chin, such as... Figure 4 As shown, considering that the impact of the chin on the driver status monitoring system is negligible, the camera position does not need to be adjusted in this case.

[0094] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application determine the missing parts of the face by extracting facial feature points in the current detection image, and then determine the adjustment height value of the steering wheel. This makes the driver state monitoring system have a concrete standard for detecting whether the driver's face in the monitoring image captured by the driver state monitoring camera is complete, and the determined adjustment height value is more accurate.

[0095] In one exemplary implementation, such as Figure 5 As shown, the process of determining the expected steering wheel height in step S107 above may include:

[0096] S501, Select a first key point based on the first monitoring image; the first key point is the edge point of the fixed interior trim of the vehicle.

[0097] Specifically, based on the coordinates of the first key point of the camera when it is in the factory position, which are transformed from the image coordinate system to the vehicle coordinate system, and the coordinates of the first key point of the camera when it is in the current position, which are transformed from the image coordinate system to the vehicle coordinate system, the change in the position of the camera can be deduced from the change in the coordinates of the first key point in the vehicle coordinate system. In order to control variables, the first key point must be a fixed point.

[0098] S503, obtain the first original coordinates and original view coordinates of the first key point; the first original coordinates are obtained by transforming the coordinates of the first key point in the original image coordinate system to the vehicle coordinate system.

[0099] The original image coordinate system is the coordinate system of the image acquired by the camera when it was in its factory position.

[0100] Among them, the original view coordinates are the coordinates of the origin of the camera coordinate system in the vehicle coordinate system when the driver status monitoring camera is in the factory position.

[0101] Specifically, since the first key point is a fixed point and the camera's factory position is a fixed position, the relationship between the first key point and the camera's factory position is fixed. The coordinates of the first key point in the original image coordinate system, when transformed to the vehicle coordinate system, have a fixed relationship with the coordinates of the camera's origin in the vehicle coordinate system when the camera is in its factory position. This is the premise for determining the current position of the camera.

[0102] S505, transform the coordinates of the first key point in the coordinate system of the first monitoring image to the coordinate system of the whole vehicle to obtain the first current coordinates.

[0103] Wherein, the first current coordinate is the coordinate obtained by transforming the first key point from the image coordinate system to the vehicle coordinate system when the camera is at its current position.

[0104] Specifically, the coordinates of the first key point in the pixel coordinate system are transformed into the coordinates of the first monitoring image through a quadratic transformation, then transformed into the coordinates of the camera coordinate system through perspective projection transformation, and finally transformed into the coordinates of the whole vehicle coordinate system through rigid transformation.

[0105] S507, determine the change in the first original coordinates and the first current coordinates to obtain the first change.

[0106] Specifically, since the position of the first key point is fixed, both the original coordinates and the current coordinates of the first key point are obtained by transforming from the pixel coordinate system to the image coordinate system, and then to the vehicle coordinate system. The only variable in this process is the image coordinate system, and the change in the image coordinate system is caused by the change in the camera position, and there is a fixed relationship between them. Therefore, in the vehicle coordinate system, the change in the coordinates of the first key point is directly and obviously related to the change in the camera position, so that the change in the current position of the camera relative to the factory position can be determined based on the first change.

[0107] S509, based on the first change amount and the original viewpoint coordinates, determine the coordinates of the viewpoint when acquiring the first monitoring image, and obtain the first viewpoint coordinates.

[0108] The first-person perspective coordinates are the coordinates of the origin of the camera coordinate system in the vehicle coordinate system when the camera is in its current position.

[0109] Specifically, the change in the camera's current position relative to its factory position is determined based on the first change. By combining this change with the camera's factory position, the camera's current position can be determined.

[0110] S5011, Based on the first viewpoint coordinates and the missing facial portion, determine the expected coordinates of the viewpoint; the expected coordinates indicate the expected height.

[0111] The expected coordinates are the coordinates of the origin of the camera coordinate system after the camera adjustment in the vehicle coordinate system, where the camera is expected to capture the driver's face completely.

[0112] Specifically, the expected location of the camera is determined based on its current location and the missing parts of the driver's face in the monitoring image captured by the camera at that current location.

[0113] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application infer the current position of the driver status monitoring camera by the coordinate change of a fixed point in the vehicle coordinate system, and then determine the expected coordinates of the origin of the camera coordinate system, providing a standard reference for the adjusted position of the camera, so as to prevent the camera position from not being adjusted accurately according to the adjusted height value due to motor accuracy issues.

[0114] S109, Adjust the height of the steering wheel based on the adjusted height value.

[0115] The height adjustment includes both the adjustment direction and the adjustment distance. Specifically, the steering wheel position is adjusted according to the adjustment direction and distance. Since the driver status detection camera is installed above the steering column, the change in steering wheel position and the change in camera position are in a 1:1 ratio. Therefore, adjusting the steering wheel position achieves the adjustment of the camera position.

[0116] S1011, acquire a second monitoring image for monitoring the driver's state, and determine the adjusted height of the steering wheel based on the second monitoring image.

[0117] The second monitoring image is the image re-acquired by the driver status monitoring camera after the steering wheel position is adjusted.

[0118] Specifically, based on the second monitoring image, the driver status monitoring camera installed above the steering column of the steering wheel performs self-calibration to determine the current position of the camera, which indicates the adjusted height of the steering wheel.

[0119] In one exemplary implementation, such as Figure 6 As shown, after acquiring the second monitoring image used to monitor the driver's state, step S1011 above may include:

[0120] S601, Select a second key point based on the second monitoring image; the second key point is the edge point of the fixed interior trim of the vehicle.

[0121] Specifically, based on the coordinates of the second key point when the camera is in its factory position, which are transformed from the image coordinate system to the vehicle coordinate system, and the coordinates of the second key point when the camera is in its current position, which are transformed from the image coordinate system to the vehicle coordinate system, the change in the camera's position can be deduced from the change in the coordinates of the second key point in the vehicle coordinate system. In order to control variables, the second key point must be a fixed point.

[0122] S603, obtain the second original coordinates of the second key point; the second original coordinates are obtained by transforming the coordinates of the second key point in the original image coordinate system to the vehicle coordinate system.

[0123] The original image coordinate system is the coordinate system of the image acquired by the camera when it was in its factory position.

[0124] Specifically, since the second key point is a fixed point and the camera's factory position is a fixed position, the relationship between the second key point and the camera's factory position is fixed. The coordinates of the second key point in the original image coordinate system are transformed to the coordinates in the vehicle coordinate system, and there is a fixed relationship between the coordinates of the camera's origin in the vehicle coordinate system when the camera is in its factory position. This is the premise for determining the current position of the camera.

[0125] S605, transform the coordinates of the second key point in the coordinate system of the second monitoring image to the coordinate system of the whole vehicle to obtain the second current coordinates.

[0126] The second current coordinates are the coordinates of the second key point obtained by transforming the image coordinate system to the vehicle coordinate system when the camera is in its current position.

[0127] Specifically, the coordinates of the second key point in the pixel coordinate system are transformed into the coordinates of the second monitoring image through a quadratic transformation, then transformed into the coordinates of the camera coordinate system through perspective projection transformation, and finally transformed into the coordinates of the whole vehicle coordinate system through rigid transformation.

[0128] S607, determine the change in the second original coordinate and the second current coordinate, and obtain the second change.

[0129] Specifically, since the position of the second key point is fixed, both the original coordinates and the current coordinates of the second key point are obtained by transforming from the pixel coordinate system to the image coordinate system, and then to the vehicle coordinate system. The only variable in this process is the image coordinate system, and the change in the image coordinate system is caused by the change in the camera position, and there is a fixed relationship between them. Therefore, in the vehicle coordinate system, the change in the coordinates of the second key point is directly and obviously related to the change in the camera position, so that the change in the current position of the camera relative to the factory position can be determined based on the second change.

[0130] S609, based on the second change amount and the original viewpoint coordinates, determine the coordinates of the viewpoint when acquiring the second monitoring image, and obtain the second viewpoint coordinates; the second viewpoint coordinates indicate the adjusted height.

[0131] Among them, the original view coordinates are the coordinates of the origin of the camera coordinate system in the vehicle coordinate system when the driver status monitoring camera is in the factory position.

[0132] The second-view coordinates are the coordinates of the origin of the camera coordinate system in the vehicle coordinate system when the camera is in its current position.

[0133] Specifically, the change in the camera's current position relative to its factory position is determined based on the second change value. By combining this change value with the camera's factory position, the camera's current position can be determined.

[0134] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application can infer the current position of the driver status monitoring camera by the coordinate change of a fixed point in the vehicle coordinate system, thereby determining the adjusted height of the camera coordinate system origin.

[0135] S1013, determine the difference between the adjusted height and the expected height.

[0136] Specifically, due to the precision issues of the motor used to adjust the steering wheel, there may be errors in the adjusted height. Therefore, the difference between the adjusted height and the expected height is calculated to characterize the error, and the steering wheel height is adjusted further based on this difference.

[0137] In one exemplary embodiment, step S1013 above may include the following steps:

[0138] Determine the difference between the second viewpoint coordinates and the expected coordinates.

[0139] The second-view coordinates are the coordinates of the origin of the rear camera coordinate system in the vehicle coordinate system.

[0140] The expected coordinates are the coordinates of the origin of the camera coordinate system after the camera adjustment in the vehicle coordinate system, where the camera is expected to capture the driver's face completely.

[0141] Specifically, the difference between the second-view coordinates and the expected coordinates is used to obtain the error of this adjustment.

[0142] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application use coordinate calculation to visualize the steering wheel height adjustment error caused by the motor precision problem, so as to facilitate the subsequent adjustment of the steering wheel height based on the error and continuously reduce the error.

[0143] S1015, determine whether the difference meets the preset error requirements.

[0144] Specifically, if the result of the judgment is yes, then step S1017 can be executed; otherwise, if the result of the judgment is no, then step S1019 can be executed.

[0145] The preset error requirement is that the difference is less than or equal to the difference between the preset adjusted height and the expected height. If this requirement is met, the adjustment error caused by the motor accuracy can be ignored.

[0146] Specifically, when the difference meets the preset error requirement, the system checks whether the driver's face in the second monitoring image is complete. Once the driver's face in the second monitoring image is complete, the driver state monitoring system exits the adjustment state. When the difference does not meet the preset error requirement, the system redetermines the adjustment height value based on the difference and readjusts the steering wheel height. Once the difference meets the preset error requirement, the system checks whether the driver's face in the second monitoring image is complete. Once the driver's face in the second monitoring image is complete, the driver state monitoring system exits the adjustment state.

[0147] S1017, when the face in the second monitoring image is complete, exit the monitoring adjustment state.

[0148] Specifically, when the driver's face is detected as intact in the monitoring image captured by the driver status monitoring camera, there is no need to continue adjusting the camera position, i.e., there is no need to continue adjusting the steering wheel height. At this point, the adjustment state of the driver status monitoring system is exited.

[0149] In one exemplary embodiment, after step S1017 described above, the following steps may be included:

[0150] The driver's line of sight is determined based on the second monitoring image;

[0151] When the line of sight intersects with the vehicle screen, the monitoring and adjustment are completed.

[0152] Specifically, when the driver's gaze intersects with the in-vehicle screen, the driver's gaze monitoring system outputs an Eyedetected signal to the display host via the FlexRay bus, and displays relevant information such as "Adjustment complete" on the in-vehicle screen. In practice, even after exiting the monitoring state, if the face is complete in the first monitoring image, the in-vehicle screen can still display "Adjustment complete."

[0153] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application monitor the driver's line of sight and display the adjustment status information of the driver status monitoring system on the vehicle screen in a timely manner, thereby increasing the intelligent interaction between the driver status monitoring system and the driver and improving the intelligence of the driver status monitoring system.

[0154] S1019, Update the adjustment height value based on the difference.

[0155] Specifically, after executing step S1017, step S109 is executed.

[0156] Specifically, the updated height adjustment value is based on the difference between the current steering wheel height and the expected height, and the steering wheel height is readjusted based on this adjustment value.

[0157] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application automatically detect whether the driver's face is complete by capturing images with a driver status monitoring camera, infer the adjustment direction and adjustment height based on the missing parts, and adjust the camera by adjusting the steering wheel, thereby realizing camera position adjustment that adapts to the driving posture of various groups of people, taking into account intelligence, accuracy and adaptability.

[0158] Corresponding to the driver state monitoring methods provided in the above embodiments, this application also provides a driver state monitoring device. Since the driver state monitoring device provided in this application corresponds to the driver state monitoring methods provided in the above embodiments, the implementation methods of the aforementioned driver state monitoring methods are also applicable to the driver state monitoring device provided in this embodiment, and will not be described in detail in this embodiment.

[0159] Please see Figure 7 The diagram shown is a structural schematic of a driver state monitoring device provided in an embodiment of this application. This device has the function of implementing the driver state monitoring method described in the above-described method embodiments. This function can be implemented by hardware or by hardware executing corresponding software. Figure 7 As shown, the device may include:

[0160] Image acquisition module 710 is used to acquire a first monitoring image for monitoring the driver's condition;

[0161] The adjustment determination module 720 is used to determine the adjustment height value and expected height of the steering wheel based on the missing part of the face in the first monitoring image when the face in the first monitoring image is incomplete.

[0162] The height adjustment module 730 is used to adjust the height of the steering wheel based on the adjusted height value;

[0163] The height determination module 740 is used to acquire a second monitoring image for monitoring the driver's state, and to determine the adjusted height of the steering wheel based on the second monitoring image.

[0164] The adjustment detection module 750 is used to determine the difference between the adjusted height and the expected height. When the difference meets the preset error requirement, it detects whether the face in the second monitoring image is complete. The monitoring adjustment state is exited when the face in the second monitoring image is complete.

[0165] In one exemplary embodiment, the adjustment determination module includes:

[0166] The feature extraction module is used to extract facial feature points of the driver based on the first monitoring image;

[0167] The quantity statistics module is used to determine the number of facial feature points of the driver;

[0168] The missing feature point determination module is used to determine the missing feature points in the driver's facial feature points when the number does not meet the preset number of feature points;

[0169] A part determination module is used to determine the missing facial parts based on the missing feature points;

[0170] The adjustment value determination module is used to determine the adjustment height value of the steering wheel based on the missing facial features.

[0171] In one exemplary embodiment, the adjustment determination module includes:

[0172] The first selection module is used to select a first key point based on the first monitoring image; the first key point is the edge point of the fixed interior trim of the vehicle.

[0173] The first acquisition module is used to acquire the first original coordinates and the original view coordinates of the first key point; the first original coordinates are obtained by transforming the coordinates of the first key point in the original image coordinate system to the vehicle coordinate system; the original image coordinate system is the coordinate system of the image acquired by the camera when it is in the factory position.

[0174] The first conversion module is used to convert the coordinates of the first key point in the coordinate system of the first monitoring image to the coordinate system of the whole vehicle to obtain the first current coordinates;

[0175] The first change module is used to determine the amount of change between the first original coordinates and the first current coordinates, and to obtain the first change amount;

[0176] The first coordinate module is used to determine the coordinates of the viewpoint when acquiring the first monitoring image based on the first change amount and the original viewpoint coordinates, and obtain the first viewpoint coordinates;

[0177] The expected determination module is used to determine the expected coordinates of the viewpoint based on the first viewpoint coordinates and the missing facial portion; the expected coordinates indicate the expected height.

[0178] In one exemplary implementation, the height determination module includes:

[0179] The second selection module is used to select a second key point based on the second monitoring image; the second key point is the edge point of the fixed interior trim of the vehicle.

[0180] The second acquisition module is used to acquire the second original coordinates of the second key point; the second original coordinates are obtained by transforming the coordinates of the second key point in the original image coordinate system to the vehicle coordinate system.

[0181] The second conversion module is used to convert the coordinates of the second key point in the coordinate system of the second monitoring image to the coordinate system of the whole vehicle to obtain the second current coordinates;

[0182] The second change module is used to determine the amount of change between the second original coordinates and the second current coordinates, and to obtain the second change amount;

[0183] The second coordinate module is used to determine the coordinates of the viewpoint when acquiring the second monitoring image based on the second change amount and the original viewpoint coordinates, and obtain the second viewpoint coordinates; the second viewpoint coordinates indicate the adjusted height.

[0184] In one exemplary embodiment, the adjustment detection module includes:

[0185] The difference determination module is used to determine the difference between the second viewpoint coordinates and the expected coordinates;

[0186] The image detection module is used to detect whether the face in the second monitoring image is complete when the difference meets the preset error requirement, and exit the monitoring adjustment state when the face in the second monitoring image is complete.

[0187] In one exemplary embodiment, the device further includes a screen display module for displaying an adjustment state on an in-vehicle screen, the screen display module comprising:

[0188] A line-of-sight determination module is used to determine the driver's line of sight based on the second monitoring image;

[0189] The display module is completed to display monitoring and adjustment when the line of sight intersects with the vehicle screen.

[0190] In one exemplary embodiment, the device further includes an adjustment exit module for exiting the monitoring adjustment state without needing to adjust the steering wheel height, the adjustment exit module comprising:

[0191] The adjustment exit module is used to exit the monitoring adjustment state when the face in the first monitoring image is complete.

[0192] In one exemplary embodiment, the device further includes a readjustment module for readjusting the steering wheel height when the difference does not meet a preset error requirement, the readjustment module comprising:

[0193] The update adjustment module is used to update the adjustment height value based on the difference when the difference does not meet the preset error requirement, and to update the difference based on the second monitoring image corresponding to the updated adjustment height value, until the updated difference meets the preset error requirement, and then stop the update.

[0194] The update detection module is used to detect whether the face in the updated second monitoring image is complete, and exits the monitoring adjustment state when the face in the updated second monitoring image is complete.

[0195] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0196] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement any of the driver state monitoring methods provided in the above method embodiments.

[0197] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0198] The method embodiments provided in this application can be executed in a computer terminal, server or similar computing device, that is, the above-mentioned electronic device may include a computer terminal, server or similar computing device. Figure 8 This is a hardware structure block diagram of a computer device for running a driver status monitoring method provided in an embodiment of the present invention, such as... Figure 8 As shown, the internal structure of this computer device may include, but is not limited to, a processor, a network interface, and a memory. The processor, network interface, and memory within the computer device can be connected via a bus or other means, as illustrated in the embodiments of this specification. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0199] The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device. The network interface may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.). Memory is the storage device in the computer device used to store programs and data. It is understood that the memory here can be a high-speed RAM storage device, or a non-volatile storage device, such as at least one disk storage device; optionally, it can also be at least one storage device located remotely from the aforementioned processor. The memory provides storage space, which stores the operating system of the electronic device, including but not limited to: Windows (an operating system), Linux (an operating system), Android (a mobile operating system), iOS (a mobile operating system), etc., which are not limited in this invention; and the storage space also stores one or more instructions suitable for being loaded and executed by the processor, which can be one or more computer programs (including program code). In the embodiments of this specification, the processor loads and executes one or more instructions stored in the memory to implement the driver state monitoring method provided in the above method embodiments. Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a driver state monitoring method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the driver state monitoring methods provided in the above-described method embodiments.

[0200] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0201] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0202] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0203] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0204] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A driver state monitoring method characterized by, A camera for monitoring the state of the driver is mounted above the steering wheel steering column, and the method comprises: acquiring a first monitoring image for monitoring the state of the driver; detecting whether the face in the first monitoring image is complete; when the face in the first monitoring image is not complete, determining the adjusted height value of the steering wheel and the expected height based on the missing part of the face in the first monitoring image; adjusting the height of the steering wheel based on the adjusted height value; acquiring a second monitoring image for monitoring the state of the driver, and determining the adjusted height of the steering wheel based on the second monitoring image; determining the difference between the adjusted height and the expected height, and when the difference meets the preset error requirement, detecting whether the face in the second monitoring image is complete, and exiting the monitoring adjustment state when the face in the second monitoring image is complete.

2. The driver state monitoring method according to claim 1, characterized by, When the face in the first monitoring image is not complete, the method comprises: extracting driver facial feature points based on the first monitoring image; determining the number of driver facial feature points; when the number does not meet the preset number of feature points, determining the missing feature points in the driver facial feature points; determining the missing part of the face based on the missing feature points; determining the adjusted height value of the steering wheel based on the missing part of the face.

3. The driver state monitoring method according to claim 1, characterized by, The method comprises: selecting a first key point based on the first monitoring image; the first key point is an edge point of the fixed interior of the vehicle; acquiring a first original coordinate and an original perspective coordinate of the first key point; the first original coordinate is obtained by converting the coordinate of the first key point in the original image coordinate system to the vehicle coordinate system; the original image coordinate system is the coordinate system in which the image obtained by the camera in the factory position is located; converting the coordinate of the first key point in the coordinate system of the first monitoring image to the vehicle coordinate system to obtain a first current coordinate; determining the change amount of the first original coordinate and the first current coordinate to obtain a first change amount; based on the first change amount and the original perspective coordinate, determining the coordinate of the perspective when the first monitoring image is acquired to obtain a first perspective coordinate; based on the first perspective coordinate and the missing part of the face, determining the expected coordinate of the perspective; the expected coordinate indicates the expected height.

4. The driver state monitoring method according to claim 3, characterized by, The method comprises: selecting a second key point based on the second monitoring image; the second key point is an edge point of the fixed interior of the vehicle; acquiring a second original coordinate of the second key point; the second original coordinate is obtained by converting the coordinate of the second key point in the original image coordinate system to the vehicle coordinate system; converting the coordinate of the second key point in the coordinate system of the second monitoring image to the vehicle coordinate system to obtain a second current coordinate; determining the change amount of the second original coordinate and the second current coordinate to obtain a second change amount; determine a coordinate of the view angle when the second monitoring image is acquired based on the second change amount and the original view angle coordinate, to obtain a second view angle coordinate; the second view angle coordinate indicates the adjusted height.

5. The driver state monitoring method according to claim 4, characterized by, The method further comprises: determine a difference between the second view angle coordinate and the expected coordinate; detect whether the face in the second monitoring image is complete when the difference meets the preset error requirement, and exit the monitoring adjustment state when the face in the second monitoring image is complete.

6. The driver state monitoring method according to any one of claims 1 to 5, characterized by, After the monitoring adjustment state is exited, the method further comprises: determine a line of sight of the driver based on the second monitoring image; display a monitoring adjustment completion when the line of sight and the vehicle-mounted screen have an intersection. The method further comprises:

7. The driver state monitoring method according to claim 1, characterized by, exit the monitoring adjustment state when the face in the first monitoring image is complete. The method further comprises:

8. The driver state monitoring method according to claim 1, characterized by, when the difference does not meet the preset error requirement, update the adjusted height value based on the difference, and update the difference based on a second monitoring image corresponding to the updated adjusted height value until the updated difference meets the preset error requirement, and stop the updating; detect whether the face in the updated second monitoring image is complete, and exit the monitoring adjustment state when the face in the updated second monitoring image is complete. A camera for monitoring a driver state is mounted above a steering wheel steering column, and the device comprises:

9. A driver state monitoring apparatus characterized by comprising: an image acquisition module configured to acquire a first monitoring image for monitoring a driver state; an adjustment determination module configured to, when the face in the first monitoring image is not complete, determine an adjusted height value and an expected height of the steering wheel based on a missing part of the face in the first monitoring image; a height adjustment module configured to adjust the height of the steering wheel based on the adjusted height value; a height determination module configured to acquire a second monitoring image for monitoring a driver state, and determine an adjusted height of the steering wheel based on the second monitoring image; an adjustment detection module configured to determine a difference between the adjusted height and the expected height, detect whether the face in the second monitoring image is complete when the difference meets a preset error requirement, and exit a monitoring adjustment state when the face in the second monitoring image is complete. The device comprises a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to implement the driver state monitoring method according to any one of claims 1-8.

10. An electronic device, comprising: ​

Images