Vehicle rearview mirror adjusting method and device, vehicle and storage medium
Through image data acquisition and processing, the rearview mirror is automatically recognized and adjusted, which solves the problems of low manual adjustment efficiency and inconvenient voice control, and realizes efficient and safe intelligent adjustment of the rearview mirror, suitable for language barrier drivers.
Patent Information
- Application Number
- CN202411236985.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The existing rearview mirror adjustment method requires the driver to manually turn the button, which is inefficient, increases the operating burden, affects driving safety, and is difficult to recognize the driver's intentions, especially for drivers with language barriers.
By collecting multi-frame image data of the driver, head recognition and eye-catching area detection, identify gesture changes, match corresponding adjustment instructions, and automatically adjust the rearview mirror to achieve manual operation without manual operation. Intelligent adjustment.
It improves the efficiency of rearview mirror adjustment, reduces the operating burden of drivers, enhances safety, and is especially suitable for drivers with language barriers, and improves intelligence level.
Smart Images

Figure CN119018052A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of rearview mirror control, and in particular to a rearview mirror adjustment method, device, vehicle and storage medium for a vehicle. Background Art
[0002] In the related art, the rearview mirror adjustment method requires the driver to manually push a button, which not only has low adjustment efficiency and increases the driver's operating burden, but may also distract their attention and reduce driving safety. The rearview mirror adjustment method through voice control commands is difficult to accurately identify the driver's actual adjustment intention, and is not conducive to some drivers with language barriers, and needs to be improved. Summary of the invention
[0003] The present application provides a vehicle rearview mirror adjustment method, device, vehicle and storage medium to solve technical problems in related technologies, such as low efficiency of manual adjustment by the driver pushing buttons, heavy operating burden on the driver, high impact on the driver's attention, certain safety hazards, low intelligence level, and voice control method is not conducive to some drivers with language barriers.
[0004] The first aspect of the present application provides a method for adjusting a rearview mirror of a vehicle, comprising the following steps: acquiring multi-frame image data of a driver; performing head recognition of the driver based on the multi-frame image data to obtain a recognition result, and detecting the driver's gaze area according to the recognition result; using the multi-frame image data to recognize the driver's gesture change trend, matching corresponding gesture adjustment instructions based on the gesture change trend, determining target adjustment parameters of the vehicle's rearview mirror according to the gesture adjustment instructions, determining theoretical adjustment parameters of the vehicle's rearview mirror according to the vehicle's surrounding traffic environment parameters, and obtaining actual adjustment parameters of the vehicle's rearview mirror in combination with the target adjustment parameters, the theoretical adjustment parameters and preset logical constraints, so as to adjust the vehicle's rearview mirror using the actual adjustment parameters.
[0005] Optionally, in one embodiment of the present application, before using the multi-frame image data to identify the driver's gesture change trend, it also includes: determining whether the vehicle rearview mirror exists in the gaze area; if the vehicle rearview mirror exists, controlling the vehicle to enter a rearview mirror adjustment mode to identify the gesture change trend, otherwise, prohibiting the vehicle from entering the rearview mirror adjustment mode.
[0006] Optionally, in one embodiment of the present application, the driver's head is recognized based on the multiple frames of image data to obtain a recognition result, and the driver's gaze area is detected based on the recognition result, including: identifying the driver's facial area from the multiple frames of image data; identifying multiple key points in the facial area to determine the positions of the driver's eyes and eyebrows based on the multiple key points; estimating the driver's head posture based on the positions of the driver's eyes and eyebrows, and determining the driver's initial gaze direction based on the head posture; based on the positions of the driver's eyes and eyebrows, performing pupil recognition and iris edge tracking recognition of the driver to obtain eye tracking data of the driver, and determining the driver's actual gaze direction based on the head posture, the eye tracking data and the initial gaze direction, so as to obtain the gaze area based on the actual gaze direction.
[0007] Optionally, in one embodiment of the present application, obtaining the gaze area based on the actual gaze direction includes: constructing an in-vehicle coordinate system for the vehicle; obtaining eye coordinates in the in-vehicle coordinate system based on the head posture and the positions of the eyes and eyebrows; and converting the actual gaze direction into a gaze area in the in-vehicle coordinate system based on the eye coordinates.
[0008] Optionally, in one embodiment of the present application, the use of the multiple frames of image data to identify the driver's gesture change trend includes: extracting the driver's hand features from each frame of image data, and sorting the hand features in the time sequence of each frame of image data to obtain a sorting result; matching the corresponding hand features in a preset gesture database in sequence according to the time sequence to obtain a matching result; if the matching result is a successful match, taking the hand features corresponding to the current frame of image data as the gesture change starting frame, and determining the gesture change trend based on the gesture change starting frame and the hand features, otherwise, matching the hand features corresponding to the next frame of image data.
[0009] Optionally, in one embodiment of the present application, after using the multi-frame image data to identify the driver's gesture change trend, it also includes: based on the gesture change trend, identifying the driver's actual control intention; if the actual control intention is the control intention of the vehicle rearview mirror, matching the corresponding adjustment instructions based on the gesture change trend, otherwise, controlling the vehicle to execute other control instructions except the vehicle rearview mirror control based on the actual control intention.
[0010] Optionally, in one embodiment of the present application, the preset logical constraints include: the danger level posed by the vehicle's surrounding traffic environment to the vehicle, wherein the danger level is obtained by the surrounding traffic environment parameters and the vehicle's current driving parameters; the driver's emotional characteristics, wherein the driver's emotional characteristics are obtained by the multiple frames of image data and the driver's voice data.
[0011] The second aspect of the present application provides a rearview mirror adjustment device for a vehicle, comprising: an acquisition module for acquiring multiple frames of image data of a driver; a first recognition module for performing head recognition of the driver based on the multiple frames of image data to obtain a recognition result, and detecting the driver's gaze area based on the recognition result; an adjustment module for using the multiple frames of image data to identify the driver's gesture change trend, matching corresponding gesture adjustment instructions based on the gesture change trend, determining target adjustment parameters of the vehicle's rearview mirror according to the gesture adjustment instructions, determining theoretical adjustment parameters of the vehicle's rearview mirror according to the vehicle's surrounding traffic environment parameters, and obtaining actual adjustment parameters of the vehicle's rearview mirror in combination with the target adjustment parameters, the theoretical adjustment parameters and preset logical constraints, so as to adjust the vehicle's rearview mirror using the actual adjustment parameters.
[0012] Optionally, in one embodiment of the present application, it also includes: a judgment module, used to determine whether the vehicle rearview mirror exists in the gaze area; a first control module, used to control the vehicle to enter the rearview mirror adjustment mode and identify the gesture change trend when the vehicle rearview mirror exists, otherwise, prohibit the vehicle from entering the rearview mirror adjustment mode.
[0013] Optionally, in one embodiment of the present application, the recognition module includes: a first recognition unit, used to identify the driver's facial area from the multiple frames of image data; a second recognition unit, used to identify multiple key points within the facial area, so as to determine the positions of the driver's eyes and eyebrows based on the multiple key points; an estimation unit, used to estimate the driver's head posture based on the positions of the driver's eyes and eyebrows, and determine the driver's initial gaze direction according to the head posture; a third recognition unit, used to perform pupil recognition and iris edge tracking recognition of the driver based on the positions of the driver's eyes and eyebrows, obtain the driver's eye tracking data, and determine the driver's actual gaze direction in combination with the head posture, the eye tracking data and the initial gaze direction, so as to obtain the gaze area based on the actual gaze direction.
[0014] Optionally, in one embodiment of the present application, the third recognition unit includes: a construction subunit, used to construct an in-vehicle coordinate system of the vehicle; a calculation subunit, used to obtain eye coordinates in the in-vehicle coordinate system based on the head posture and the positions of the eyes and eyebrows; and a conversion subunit, used to convert the actual line of sight direction into a gaze area in the in-vehicle coordinate system based on the eye coordinates.
[0015] Optionally, in one embodiment of the present application, the adjustment module includes: an extraction unit, used to extract the hand features of the driver from each frame of image data, and sort the hand features in the time sequence of each frame of image data to obtain a sorting result; a first matching unit, used to match the corresponding hand features in a preset gesture database in sequence according to the time sequence to obtain a matching result; a second matching unit, used to use the hand features corresponding to the current frame of image data as the gesture change starting frame when the matching result is a successful match, and determine the gesture change trend based on the gesture change starting frame and the hand features, otherwise, match the hand features corresponding to the next frame of image data.
[0016] Optionally, in one embodiment of the present application, it also includes: a second recognition module, used to identify the actual control intention of the driver based on the gesture change trend; a second control module, used to match the corresponding adjustment instructions based on the gesture change trend when the actual control intention is the control intention of the vehicle rearview mirror, otherwise, control the vehicle to execute other control instructions except the vehicle rearview mirror control based on the actual control intention.
[0017] Optionally, in one embodiment of the present application, the preset logical constraints include: the danger level posed by the vehicle's surrounding traffic environment to the vehicle, wherein the danger level is obtained by the surrounding traffic environment parameters and the vehicle's current driving parameters; the driver's emotional characteristics, wherein the driver's emotional characteristics are obtained by the multiple frames of image data and the driver's voice data.
[0018] A third aspect of the present application provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the rearview mirror adjustment method of the vehicle as described in the above embodiment.
[0019] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the vehicle rearview mirror adjustment method as described in the above embodiment.
[0020] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned method for adjusting the rearview mirror of a vehicle.
[0021] The embodiment of the present application can collect multiple frames of image data of the driver, so as to perform head recognition of the driver, detect the driver's gaze area according to the recognition result, and at the same time, recognize the change trend of the driver's gesture, match the corresponding gesture adjustment instruction based on the change trend of the gesture, determine the target adjustment parameter of the vehicle rearview mirror according to the gesture adjustment instruction, determine the theoretical adjustment parameter of the vehicle rearview mirror according to the surrounding traffic environment parameters of the vehicle, and combine the target adjustment parameter, the theoretical adjustment parameter and the preset logical constraint to obtain the actual adjustment parameter of the vehicle rearview mirror, so as to adjust the vehicle rearview mirror with the actual adjustment parameter, realize the automatic adjustment of the vehicle rearview mirror, and do not need manual operation by the driver, and the adjustment efficiency is higher. By combining the surrounding traffic environment factors, the accidental touch is avoided, and the safety of the vehicle rearview mirror adjustment is further improved. Therefore, the technical problems in the related art that the driver manually pulls the button to adjust the low efficiency, the driver's operation burden is large, the impact on the driver's attention is high, there are certain safety hazards, the intelligence level is low, and the voice control method is not conducive to some drivers with language barriers are solved.
[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0024] Figure 1 A flowchart of a vehicle rearview mirror adjustment method provided according to an embodiment of the present application;
[0025] Figure 2 is a flow chart of a rearview mirror adjustment method according to an embodiment of the present application;
[0026] Figure 3 It is a structural schematic diagram of a rearview mirror adjustment device for a vehicle provided according to an embodiment of the present application;
[0027] Figure 4 It is a schematic diagram of the structure of a vehicle provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0029] The following describes the rearview mirror adjustment method, device, vehicle and storage medium of the vehicle of the embodiment of the present application with reference to the accompanying drawings. In view of the technical problems that the driver manually pulls the button to adjust the low efficiency, the driver's operation burden is large, the driver's attention is affected, there are certain safety hazards, the intelligence level is low, and the voice control method is not conducive to some drivers with language barriers, the present application provides a vehicle rearview mirror adjustment method, in which multiple frames of image data of the driver can be collected to perform driver head recognition, so as to detect the driver's gaze area according to the recognition result, and at the same time, identify the driver's gesture change trend, match the corresponding gesture adjustment instruction based on the gesture change trend, determine the target adjustment parameter of the vehicle rearview mirror according to the gesture adjustment instruction, determine the theoretical adjustment parameter of the vehicle rearview mirror according to the vehicle's surrounding traffic environment parameters, and combine the target adjustment parameter, the theoretical adjustment parameter and the preset logical constraint to obtain the actual adjustment parameter of the vehicle rearview mirror, so as to adjust the vehicle rearview mirror using the actual adjustment parameter, realize the automatic adjustment of the vehicle rearview mirror, without the need for manual operation by the driver, and achieve higher adjustment efficiency, and by combining the surrounding traffic environment factors, avoid the situation of accidental touch, and further improve the safety of the vehicle rearview mirror adjustment. Thereby, technical problems in related technologies such as low efficiency of manual adjustment by the driver by pushing buttons, heavy operating burden on the driver, high impact on the driver's attention, certain safety hazards, low intelligence level, and voice control method that is not conducive to some drivers with language barriers are solved.
[0030] Specifically, Figure 1 A schematic flow chart of a vehicle rearview mirror adjustment method provided in an embodiment of the present application.
[0031] like Figure 1 As shown, the rearview mirror adjustment method of the vehicle includes the following steps:
[0032] In step S101 , multiple frames of image data of the driver are collected.
[0033] During the actual implementation process, the embodiment of the present application can utilize the vehicle's internal camera, or reuse the camera of the vehicle's driver status monitoring system, to obtain multiple frames of image data of the driver within a certain period of time, so as to determine the driver's movement trend based on the acquisition time of the multiple frames of image data.
[0034] In step S102, the driver's head is recognized based on the multi-frame image data to obtain a recognition result, and the driver's gaze area is detected according to the recognition result.
[0035] Furthermore, the embodiments of the present application can capture the driver's facial and eye information in real time based on multiple frames of image data, and determine the driver's gaze area through algorithm analysis and other methods to determine whether the driver is looking at the left, right or center.
[0036] Optionally, in one embodiment of the present application, the driver's head is recognized based on multiple frames of image data to obtain a recognition result, and the driver's gaze area is detected based on the recognition result, including: identifying the driver's facial area from multiple frames of image data; identifying multiple key points in the facial area to determine the positions of the driver's eyes and eyebrows based on the multiple key points; estimating the driver's head posture based on the positions of the driver's eyes and eyebrows, and determining the driver's initial gaze direction based on the head posture; based on the position of the driver's eyes and eyebrows, performing pupil recognition and iris edge tracking recognition of the driver to obtain eye tracking data of the driver, and determining the driver's actual gaze direction based on the head posture, eye tracking data and initial gaze direction, so as to obtain the gaze area based on the actual gaze direction.
[0037] For example, when performing head recognition, the embodiment of the present application may include the following steps:
[0038] Step S1: Face detection and positioning. The embodiment of the present application can use computer vision technology, such as Haar features, HOG+SVM, deep learning models, etc., to pre-process the image, identify and frame the driver's facial area.
[0039] Step S2: Key point detection. The embodiment of the present application can detect key points in the facial region, such as the positions of eyes and eyebrows, to understand the head posture and eye direction.
[0040] Step S3: Head posture estimation. The embodiment of the present application can use key point information to estimate the head posture angle (pitch, yaw and roll) through a 3D model or machine learning method, thereby inferring the driver's general gaze direction, that is, the initial gaze direction.
[0041] Step S4: Eye tracking. The embodiment of the present application can be further refined to the eye area, using pupil detection and iris edge tracking technology, combined with head posture information, to accurately calculate the direction of sight, that is, the actual gaze direction.
[0042] Optionally, in one embodiment of the present application, obtaining a gaze area based on the actual gaze direction includes: constructing an in-vehicle coordinate system of the vehicle; obtaining eye coordinates in the in-vehicle coordinate system based on the head posture and the positions of the eyes and eyebrows; and converting the actual line of sight direction into a gaze area in the in-vehicle coordinate system based on the eye coordinates.
[0043] Furthermore, the embodiments of the present application can combine the three-dimensional coordinate system inside the vehicle to convert the actual gaze direction into a specific gaze area inside or outside the vehicle.
[0044] In the actual implementation process, the embodiment of the present application can establish a model of the in-vehicle environment and construct a coordinate system with a certain point as the origin to facilitate the conversion of the line of sight direction into the actual gaze area.
[0045] In step S103, the driver's gesture change trend is identified using multi-frame image data, and corresponding gesture adjustment instructions are matched based on the gesture change trend. The target adjustment parameters of the vehicle rearview mirror are determined according to the gesture adjustment instructions, and the theoretical adjustment parameters of the vehicle rearview mirror are determined according to the vehicle's surrounding traffic environment parameters. The actual adjustment parameters of the vehicle rearview mirror are obtained by combining the target adjustment parameters, the theoretical adjustment parameters and the preset logical constraints, so as to adjust the vehicle rearview mirror using the actual adjustment parameters.
[0046] As a possible implementation method, the embodiment of the present application can use multi-frame image data to identify the driver's gesture change trend, thereby obtaining the driver's gesture control action, and then use the driver's gesture adjustment instructions to adjust the vehicle's rearview mirror, and eliminate the risk of accidental touch or adjustment according to the traffic environment. Intelligent and automatic rearview mirror adjustment can be achieved without the driver's manual operation or voice control.
[0047] The embodiment of the present application can determine the target adjustment parameters according to the gesture adjustment instruction and the driver's gaze area, so that the rearview mirror can display the reflection picture required by the driver.
[0048] Among them, the embodiment of the present application can determine the driver's target vehicle rearview mirror based on the gaze area. For example, when the driver looks at the left vehicle rearview mirror and makes a gesture, it means that the driver wants to adjust the left vehicle rearview mirror.
[0049] For another example, when the driver turns his head to the right by more than a certain value and his gaze area is to the right rear of the driver's seat, the driver's gaze area is the right side or right rear side of the vehicle. This means that the driver wants to obtain the situation on the right side or right rear side of the vehicle. The embodiment of the present application can adjust all the rearview mirrors of the vehicle to ensure that the rearview mirrors can reflect the image on the right side or right rear side of the vehicle.
[0050] Optionally, in one embodiment of the present application, the preset logical constraints include: the danger level posed to the vehicle by the vehicle's surrounding traffic environment, wherein the danger level is obtained by the surrounding traffic environment parameters and the vehicle's current driving parameters; the driver's emotional characteristics, wherein the driver's emotional characteristics are obtained by multiple frames of image data and the driver's voice data.
[0051] For example, the embodiment of the present application can combine the target adjustment parameters, theoretical adjustment parameters and preset logical constraints to obtain the actual adjustment parameters of the vehicle rearview mirror.
[0052] The target adjustment parameter is the adjustment parameter of the rearview mirror determined according to the driver's gesture.
[0053] The theoretical adjustment parameters are adjustment parameters of the rearview mirrors that are determined based on the current driving environment of the vehicle and surrounding traffic flow data so that the driver can ensure the maximum observation field of view by observing the rearview mirrors.
[0054] The preset logical constraint may include the danger level of the vehicle's surrounding traffic environment. For example, when there are pedestrians or animals close to the rear of the vehicle, the danger level is high; when there are other vehicles traveling in the same direction but at a uniform speed behind the vehicle, the danger level is medium, etc. Therefore, when the danger level is high, the embodiment of the present application may increase the weight of the logical constraint, so that the driver can observe the pedestrians or animals behind by adjusting the vehicle rearview mirror.
[0055] Among them, the danger level can be obtained by using the vehicle's radar, camera and other perception parameters to obtain the surrounding traffic environment parameters, and combined with the vehicle's current driving parameters.
[0056] The preset logical constraints may also include the driver's emotional characteristics. For example, when the driver's emotions are positive, the embodiment of the present application may increase the weight of the target adjustment parameters. When the driver's emotions are negative, the embodiment of the present application may increase the weight of the theoretical adjustment parameters to avoid the driver's erroneous operations under negative emotions.
[0057] Among them, emotional features can be matched through image data analysis of the driver, and combined with the tone and timbre of the voice to determine the driver's actual emotions.
[0058] Optionally, in one embodiment of the present application, before using multiple frames of image data to identify the driver's gesture change trend, it also includes: determining whether there is a vehicle rearview mirror in the gaze area; if there is a vehicle rearview mirror, controlling the vehicle to enter a rearview mirror adjustment mode to identify the gesture change trend, otherwise, prohibiting the vehicle from entering a rearview mirror adjustment mode.
[0059] In some embodiments, the embodiments of the present application first determine whether there is a vehicle rearview mirror in the gaze area. When there is no vehicle rearview mirror, it means that the driver does not want to control the vehicle rearview mirror, and the vehicle is prohibited from entering the rearview mirror adjustment mode.
[0060] When there is a vehicle rearview mirror, it can be determined that the driver's intention is to control the vehicle rearview mirror, and gesture recognition is performed to control the vehicle rearview mirror.
[0061] In addition, the gaze area can also be expressed as the driver's area of interest, that is, the area the driver wants to see. At this time, the embodiment of the present application can control the vehicle to enter the rearview mirror adjustment mode according to the area of interest and gesture instructions, so that the vehicle's rearview mirror can reflect the driver's area of interest after adjustment.
[0062] Optionally, in one embodiment of the present application, after using multiple frames of image data to identify the driver's gesture change trend, it also includes: based on the gesture change trend, identifying the driver's actual control intention; if the actual control intention is the control intention of the vehicle's rearview mirror, matching the corresponding adjustment instructions based on the gesture change trend, otherwise, controlling the vehicle to execute other control instructions except the vehicle rearview mirror control based on the actual control intention.
[0063] In other embodiments, when the actual control intention corresponding to the gesture change trend is to control the vehicle rearview mirror, the embodiments of the present application can match the corresponding adjustment instructions according to the gesture change trend.
[0064] When the actual control intention corresponding to the gesture change trend is not to control the vehicle's rearview mirror, it means that the driver does not want to control the vehicle's rearview mirror, and the driver's gesture action may be used to control other functions of the vehicle. At this time, other functions of the vehicle can be controlled.
[0065] Optionally, in one embodiment of the present application, multiple frames of image data are used to identify the trend of changes in the driver's gestures, including: extracting the driver's hand features from each frame of image data, and sorting the hand features in time sequence of each frame of image data to obtain a sorting result; matching the corresponding hand features in a preset gesture database in time sequence to obtain a matching result; if the matching result is a successful match, taking the hand features corresponding to the current frame of image data as the gesture change starting frame, and determining the gesture change trend based on the gesture change starting frame and the hand features, otherwise, matching the hand features corresponding to the next frame of image data.
[0066] It is understandable that when the driver performs gesture control, the first frame image is not necessarily the starting point of the driver's gesture action. In this case, gesture recognition is likely to produce erroneous recognition results.
[0067] Therefore, the embodiment of the present application can extract the driver's hand features in each frame of image data, and compare them in sequence in the gesture database. Those that fail to match are marked as meaningless actions. The first image frame corresponding to the hand features that can be matched is the starting point of the gesture, and then the subsequent image frames are compared in sequence.
[0068] In addition, if the hand features of the next frame of image data after the image frame corresponding to the gesture starting point cannot match the data in the gesture database, it can also be determined that the gesture starting point is wrong, and the starting point can be reconfirmed at this time.
[0069] Combination Figure 2 As shown, the working principle of the rearview mirror adjustment method of the vehicle of the embodiment of the present application is described in detail with an embodiment.
[0070] First, the embodiment of the present application may be composed of a driver status monitoring system, a gesture recognition module, a rearview mirror drive unit and an electric rearview mirror. The camera in the driver status monitoring system is used to capture the driver's facial and eye information, and the gesture recognition module captures the driver's gesture changes through the camera. The rearview mirror drive unit receives instructions from the gesture recognition module and drives the electric rearview mirror to adjust.
[0071] The camera of the driver status monitoring system can be installed in the cab, and the driver's facial and eye information can be captured in real time through image processing technology. The embodiment of the present application can analyze the driver's gaze direction through an algorithm to determine whether the gaze area belongs to the left, right or center.
[0072] The gesture recognition module can capture the driver's gesture changes through the camera, and can accurately identify the driver's gesture actions such as sliding up and down, waving left and right, etc.
[0073] The rearview mirror drive unit can receive instructions from the gesture recognition module and control the electric rearview mirror to adjust up, down, left, and right. The electric rearview mirror is controlled by a high-precision motor to ensure the accuracy and speed of adjustment.
[0074] like Figure 2 As shown, the embodiment of the present application may include the following steps:
[0075] Step S1: fixation area determination.
[0076] The driver status monitoring system captures the driver's gaze direction in real time and determines the area of his gaze. Based on the area of his gaze, the system determines whether the driver intends to control the left or right side, or whether there is no need to adjust the rearview mirror.
[0077] Step S2: Gesture recognition.
[0078] After determining the driver's intention, the gesture recognition module begins to capture the driver's gesture changes. Based on the recognized gestures, such as sliding up and down, waving left and right, the driver's specific adjustment needs for the rearview mirror are determined.
[0079] Step S3: Rearview mirror adjustment.
[0080] The rearview mirror drive unit drives the electric rearview mirror to make corresponding adjustments according to the instructions sent by the gesture recognition module. During the adjustment process, the adjustment effect is monitored in real time to ensure that the rearview mirror can be accurately adjusted to the angle required by the driver.
[0081] In addition, the embodiments of the present application can also eliminate the possibility of driver's erroneous operation by combining factors such as the external traffic environment and the driver's emotional characteristics, so as to improve the safety of rearview mirror adjustment.
[0082] According to the vehicle rearview mirror adjustment method proposed in the embodiment of the present application, multiple frames of image data of the driver can be collected to perform driver head recognition, so as to detect the driver's gaze area according to the recognition result, and at the same time, identify the driver's gesture change trend, match the corresponding gesture adjustment instruction based on the gesture change trend, determine the target adjustment parameter of the vehicle rearview mirror according to the gesture adjustment instruction, determine the theoretical adjustment parameter of the vehicle rearview mirror according to the vehicle's surrounding traffic environment parameters, and combine the target adjustment parameter, the theoretical adjustment parameter and the preset logical constraint to obtain the actual adjustment parameter of the vehicle rearview mirror, so as to adjust the vehicle rearview mirror using the actual adjustment parameter, realize automatic adjustment of the vehicle rearview mirror, and do not require manual operation by the driver, and the adjustment efficiency is higher. By combining the surrounding traffic environment factors, the accidental touch is avoided, and the safety of the vehicle rearview mirror adjustment is further improved. Therefore, the technical problems in the related art that the driver manually pulls the button to adjust the low efficiency, the driver's operation burden is large, the impact on the driver's attention is high, there are certain safety hazards, the intelligence level is low, and the voice control method is not conducive to some drivers with language barriers are solved.
[0083] Next, the rearview mirror adjustment device for a vehicle proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.
[0084] Figure 3 It is a block diagram of a rearview mirror adjustment device for a vehicle according to an embodiment of the present application.
[0085] like Figure 3 As shown, the rearview mirror adjustment device 10 of the vehicle includes: a collection module 100 , a first recognition module 200 and an adjustment module 300 .
[0086] Specifically, the acquisition module 100 is used to acquire multiple frames of image data of the driver.
[0087] The first recognition module 200 is used to recognize the driver's head based on multiple frames of image data, obtain a recognition result, and detect the driver's gaze area according to the recognition result.
[0088] The adjustment module 300 is used to use multi-frame image data to identify the driver's gesture change trend, match the corresponding gesture adjustment instruction based on the gesture change trend, determine the target adjustment parameters of the vehicle rearview mirror according to the gesture adjustment instruction, determine the theoretical adjustment parameters of the vehicle rearview mirror according to the vehicle's surrounding traffic environment parameters, and obtain the actual adjustment parameters of the vehicle rearview mirror in combination with the target adjustment parameters, the theoretical adjustment parameters and preset logical constraints, so as to adjust the vehicle rearview mirror using the actual adjustment parameters.
[0089] Optionally, in one embodiment of the present application, the vehicle rearview mirror adjustment device 10 further includes: a judgment module and a first control module.
[0090] The judgment module is used to judge whether there is a vehicle rearview mirror in the gaze area.
[0091] The first control module is used to control the vehicle to enter a rearview mirror adjustment mode and identify a gesture change trend when there is a vehicle rearview mirror. Otherwise, the vehicle is prohibited from entering the rearview mirror adjustment mode.
[0092] Optionally, in one embodiment of the present application, the recognition module includes: a first recognition unit, a second recognition unit, an estimation unit and a third recognition unit.
[0093] Wherein, the first recognition unit is used to recognize the driver's facial area from multiple frames of image data.
[0094] The second recognition unit is used to recognize a plurality of key points in the facial area to determine the positions of the driver's eyes and eyebrows based on the plurality of key points.
[0095] The estimation unit is used to estimate the driver's head posture based on the positions of the driver's eyes and eyebrows, and determine the driver's initial gaze direction according to the head posture.
[0096] The third recognition unit is used to perform pupil recognition and iris edge tracking recognition of the driver based on the positions of the driver's eyes and eyebrows, obtain the driver's eye tracking data, and determine the driver's actual gaze direction in combination with the head posture, eye tracking data and initial gaze direction, so as to obtain a gaze area based on the actual gaze direction.
[0097] Optionally, in one embodiment of the present application, the third recognition unit includes: a construction subunit, a calculation subunit and a conversion subunit.
[0098] Among them, the construction subunit is used to construct the vehicle's in-vehicle coordinate system.
[0099] The calculation subunit is used to obtain the eye coordinates in the vehicle's interior coordinate system based on the head posture and the positions of the eyes and eyebrows.
[0100] The conversion subunit is used to convert the actual sight direction into the gaze area in the vehicle's coordinate system based on the eye coordinates.
[0101] Optionally, in one embodiment of the present application, the adjustment module 300 includes: an extraction unit, a first matching unit and a second matching unit.
[0102] The extraction unit is used to extract the driver's hand features from each frame of image data, and sort the hand features according to the time sequence of each frame of image data to obtain a sorting result.
[0103] The first matching unit is used to match the corresponding hand features in a preset gesture database in chronological order to obtain a matching result.
[0104] The second matching unit is used to, when the matching result is a successful match, use the hand features corresponding to the current frame image data as the gesture change starting frame, and determine the gesture change trend based on the gesture change starting frame and the hand features; otherwise, match the hand features corresponding to the next frame image data.
[0105] Optionally, in one embodiment of the present application, the vehicle rearview mirror adjustment device 10 further includes: a second identification module and a second control module.
[0106] The second recognition module is used to recognize the driver's actual control intention based on the gesture change trend.
[0107] The second control module is used to match the corresponding adjustment instructions based on the gesture change trend when the actual control intention is the control intention of the vehicle rearview mirror, otherwise, control the vehicle to execute other control instructions except the vehicle rearview mirror control based on the actual control intention.
[0108] Optionally, in one embodiment of the present application, the preset logical constraints include: the danger level of the vehicle's surrounding traffic environment to the vehicle, wherein the danger level is obtained from the surrounding traffic environment parameters and the vehicle's current driving parameters; the driver's emotional characteristics, wherein the driver's emotional characteristics are obtained from multiple frames of image data and the driver's voice data. It should be noted that the aforementioned explanation of the embodiment of the vehicle's rearview mirror adjustment method is also applicable to the vehicle's rearview mirror adjustment device of this embodiment, and will not be repeated here.
[0109] According to the rearview mirror adjustment device of the vehicle proposed in the embodiment of the present application, multiple frames of image data of the driver can be collected to perform head recognition of the driver, so as to detect the driver's gaze area according to the recognition result, and at the same time, the driver's gesture change trend can be identified, and the corresponding gesture adjustment instruction can be matched based on the gesture change trend, and the target adjustment parameter of the vehicle rearview mirror can be determined according to the gesture adjustment instruction, and the theoretical adjustment parameter of the vehicle rearview mirror can be determined according to the surrounding traffic environment parameters of the vehicle, and the actual adjustment parameter of the vehicle rearview mirror can be obtained by combining the target adjustment parameter, the theoretical adjustment parameter and the preset logical constraint, so as to adjust the vehicle rearview mirror by using the actual adjustment parameter, realize the automatic adjustment of the vehicle rearview mirror, and do not require manual operation by the driver, and the adjustment efficiency is higher, and by combining the surrounding traffic environment factors, the accidental touch can be avoided, and the safety of the vehicle rearview mirror adjustment can be further improved. Therefore, the technical problems in the related art that the adjustment efficiency of the driver manually turning the button is low, the driver's operation burden is large, the impact on the driver's attention is high, there are certain safety hazards, the intelligence level is low, and the voice control method is not conducive to some drivers with language barriers are solved.
[0110] Figure 4 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle may include:
[0111] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .
[0112] When the processor 402 executes the program, the vehicle rearview mirror adjustment method provided in the above embodiment is implemented.
[0113] Furthermore, the vehicle also includes:
[0114] The communication interface 403 is used for communication between the memory 401 and the processor 402 .
[0115] The memory 401 is used to store computer programs that can be executed on the processor 402 .
[0116] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0117] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0118] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.
[0119] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0120] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned rearview mirror adjustment method of the vehicle is implemented.
[0121] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the vehicle rearview mirror adjustment method provided by an embodiment of the present invention.
[0122] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0123] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0124] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0126] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0127] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0128] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0129] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for adjusting a rearview mirror of a vehicle, characterized in that: The following steps are involved: Collecting multiple frames of image data of the driver; Performing head recognition of the driver based on the multiple frames of image data to obtain a recognition result, and detecting a gaze area of the driver according to the recognition result; The multi-frame image data is used to identify the driver's gesture change trend, and a corresponding gesture adjustment instruction is matched based on the gesture change trend. The target adjustment parameters of the vehicle rearview mirror are determined according to the gesture adjustment instruction. The theoretical adjustment parameters of the vehicle rearview mirror are determined according to the surrounding traffic environment parameters of the vehicle. The actual adjustment parameters of the vehicle rearview mirror are obtained by combining the target adjustment parameters, the theoretical adjustment parameters and preset logical constraints, so as to adjust the vehicle rearview mirror using the actual adjustment parameters.
2. The method according to claim 1, characterized in that Before using the multiple frames of image data to identify the driver's gesture change trend, the method further includes: Determining whether the vehicle rearview mirror exists in the gaze area; If the vehicle rearview mirror exists, the vehicle is controlled to enter a rearview mirror adjustment mode and the gesture change trend is identified; otherwise, the vehicle is prohibited from entering the rearview mirror adjustment mode.
3. The method according to claim 1, characterized in that The step of performing head recognition of the driver based on the multiple frames of image data to obtain a recognition result, and detecting the driver's gaze area according to the recognition result, comprises: Identifying a driver's facial region from the multiple frames of image data; identifying a plurality of key points within the facial region to determine the positions of the driver's eyes and eyebrows based on the plurality of key points; estimating the driver's head posture based on the positions of the driver's eyes and eyebrows, and determining the driver's initial gaze direction according to the head posture; Based on the positions of the driver's eyes and eyebrows, the driver's pupil recognition and iris edge tracking recognition are performed to obtain the driver's eye tracking data, and the driver's actual gaze direction is determined in combination with the head posture, the eye tracking data and the initial gaze direction to obtain the gaze area based on the actual gaze direction.
4. The method according to claim 3, characterized in that The obtaining the gaze area based on the actual gaze direction includes: Constructing an in-vehicle coordinate system of the vehicle; Based on the head posture and the positions of the eyes and eyebrows, obtaining eye coordinates in the in-vehicle coordinate system; Based on the eye coordinates, the actual sight line direction is converted into a gaze area in the in-vehicle coordinate system.
5. The method according to claim 1, characterized in that The method of using the multiple frames of image data to identify the driver's gesture change trend includes: Extracting the hand features of the driver from each frame of image data, and sorting the hand features according to the time sequence of each frame of image data to obtain a sorting result; Matching corresponding hand features in a preset gesture database in sequence according to the time sequence to obtain a matching result; If the matching result is a successful match, the hand features corresponding to the current frame image data are used as the gesture change starting frame, and based on the gesture change starting frame and the hand features, the gesture change trend is determined; otherwise, the hand features corresponding to the next frame image data are matched.
6. The method according to claim 5, characterized in that After using the multiple frames of image data to identify the driver's gesture change trend, the method further includes: Based on the gesture change trend, identifying the driver's actual control intention; If the actual control intention is the control intention of the vehicle rearview mirror, the corresponding adjustment instruction is matched based on the gesture change trend; otherwise, the vehicle is controlled to execute other control instructions except the vehicle rearview mirror control based on the actual control intention.
7. The method according to claim 1, characterized in that The preset logical constraints include: The danger level posed by the surrounding traffic environment of the vehicle to the vehicle, wherein the danger level is obtained by the surrounding traffic environment parameters and the current driving parameters of the vehicle; The emotional characteristics of the driver, wherein the emotional characteristics of the driver are obtained from the multiple frames of image data and the voice data of the driver.
8. A rearview mirror adjustment device for a vehicle, characterized in that: include: An acquisition module, used for acquiring multiple frames of image data of the driver; A recognition module, configured to recognize the driver's head based on the multiple frames of image data, obtain a recognition result, and detect a gaze area of the driver according to the recognition result; An adjustment module is used to use the multi-frame image data to identify the driver's gesture change trend, match corresponding gesture adjustment instructions based on the gesture change trend, determine the target adjustment parameters of the vehicle rearview mirror according to the gesture adjustment instructions, determine the theoretical adjustment parameters of the vehicle rearview mirror according to the surrounding traffic environment parameters of the vehicle, and obtain the actual adjustment parameters of the vehicle rearview mirror in combination with the target adjustment parameters, the theoretical adjustment parameters and preset logical constraints, so as to adjust the vehicle rearview mirror using the actual adjustment parameters.
9. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the rearview mirror adjustment method for a vehicle as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the rearview mirror adjustment method for a vehicle as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Facial movement and gesture sensing side-view mirror
CN109153354A
Intelligent left and right rearview mirrors of automobile
CN114379460A
Exterior rear-view mirror adjusting method and device, electronic equipment and storage medium
CN116252709A
Safety control method and device for vehicle
CN117885522A
Intelligent rearview mirror adjusting technical method and device and vehicle
CN118219984A
Cited By
Multi-mode rearview mirror automatic adjusting method and system
CN120116850A
