Intelligent rearview mirror control method based on artificial intelligence and device and system used for intelligent rearview mirror control method

Abstract

A smart mirror control method installed in a smart mirror control device of a vehicle according to an embodiment of the present invention may include: performing a startup step in accordance with application of power; recognizing a user gesture in a state in which the touch display screen is always closed; activating the closed touch display picture according to the recognized user gesture, outputting a user interface picture for controlling the intelligent rearview mirror, and waiting for a user to input voice; performing machine learning on a user voice signal input through a microphone to perform user authentication and recognize a voice command for controlling the smart rearview mirror; and when the user authentication is successful, configuring a view screen on the basis of the recognized voice command and displaying the view screen on the smart mirror, the user authentication including: speaker-independent authentication for performing authentication only with a specific speaker registered in advance as a target; and speaker-dependent authentication in which authentication is performed for all speakers registered in the database.

Description

Technical Field

[0001] The present invention relates to an intelligent rearview mirror control technology, and in detail, to an intelligent rearview mirror control method that can dynamically control the view of the intelligent rearview mirror based on artificial intelligence, and a device and system used therefor. Background Art

[0002] Recently, cars can provide safety and convenience to drivers through various advanced driver assistance systems (ADAS) such as lane departure warning (LDW), adaptive cruise control (ACC), forward collision warning (FCW), blind spot detection (BSD) systems, etc.

[0003] The most important technology in this intelligent system is external object recognition technology, including cameras, radars, ultrasonic, lidar and other sensors. These sensors provide safety and convenience in driving and parking conditions by detecting lanes, vehicles in the front, back and blind spots, etc.

[0004] Cameras are necessary devices used in autonomous driving environments above Level 2, such as collision prevention, lane keeping, and parking assistance. With mandatory installation in many countries, their proportion is gradually increasing.

[0005] In addition, these sensors are linked with the Vehicle to Everything (V2X) communication system, which can effectively avoid accidents even in situations that are difficult to detect with environmental sensors, such as intersections.

[0006] In order to overcome the weaknesses of individual sensors, sensor fusion technology that combines sensor information is being actively developed, and the development of next-generation sensor technologies such as stereo cameras and three-dimensional image sensors is being actively carried out.

[0007] In the global image sensor market, automobiles play an important role along with smartphones and digital cameras. In particular, the recent promotion of driver assistance systems (ADAS) and the development of autonomous vehicles are creating a better environment for the adoption of image sensors in automobiles.

[0008] To ensure the safety of drivers and pedestrians, automakers are replacing mirrors in digital rearview mirrors and panoramic monitoring imaging systems, which detect blind spots, with displays. In fact, vehicles such as the Infiniti Q80, Volkswagen XL1, and Toyota NS4 have completely removed side mirrors and installed panoramic monitoring imaging systems and displays.

[0009] A panoramic monitor or surround view monitor is a monitoring system that uses four cameras to provide an image that looks like it is viewed from the sky. This is also called a top view or a bird's eye view.

[0010] However, a disadvantage of currently available digital rearview mirrors is that they can only provide a simple panoramic view and cannot provide multiple views as the user intends. Summary of the invention

[0011] The object of the present invention is to provide an intelligent rearview mirror control method based on speech recognition artificial intelligence and a device and system therefor.

[0012] Another object of the present invention is to provide the following artificial intelligence-based smart rearview mirror control method and device and system therefor, which obtains template data of each authentication unit by pre-learning sensing information collected from multiple authentication sensors, determines the authentication weighted value of each authentication unit by comparing the characteristic data of each authentication unit acquired in real time with the corresponding template data, and sets the optimal authentication level of each user based on the determined authentication weighted value, so that customized user authentication can be performed.

[0013] Another object of the present invention is to provide the following intelligent rearview mirror control method based on voice recognition artificial intelligence and the device and system used therefor, which can control the view of the intelligent rearview mirror through voice recognition during driving, thereby providing user convenience and driving safety.

[0014] The technical problems of the present invention are not limited to the technical problems mentioned above, and ordinary technicians in the technical field to which the present invention belongs can clearly understand other technical problems not mentioned through the following description.

[0015] A smart rearview mirror control method in a smart rearview mirror control device installed in a vehicle according to an embodiment of the present invention may include: executing a startup step as power is applied; recognizing a user gesture when an always-on display (AOD) touch display screen is turned off; activating the turned-off touch display screen according to the recognized user gesture, outputting a user interface screen for controlling the smart rearview mirror, and then waiting for user voice input; performing machine learning on a user voice signal input through a microphone to perform user authentication and recognize voice commands for controlling the smart rearview mirror; when the user authentication is successful, constructing a view screen based on the recognized voice command and displaying it on the smart rearview mirror, wherein the user authentication includes: speaker-independent authentication, in which authentication is performed only on a specific speaker registered in advance; and speaker-dependent authentication, in which authentication is performed on all speakers registered in a database.

[0016] In an embodiment, the startup step may include: initializing a main controller that controls the overall input and output of the smart rearview mirror control device; establishing a communication connection with multiple cameras; initializing a display panel to control the output of the touch display screen; calibrating at least one sensor, and initializing a sensor panel configured to collect sensing information from the at least one sensor, wherein the sensor may include: at least one of an illumination sensor, a proximity sensor, a touch sensor, a fingerprint sensor, and an iris sensor.

[0017] In an embodiment, it may also include: based on the above-mentioned smart rearview mirror control mode being a manual control mode, detecting user touch work on the above-mentioned user interface screen, and when the above-mentioned touch work is detected, identifying the smart camera control instruction corresponding to the above-mentioned touch work, and acquiring the corresponding camera shooting image according to the identified above-mentioned smart camera control instruction to form the above-mentioned view screen.

[0018] In an embodiment, the method may further include: determining a view type corresponding to the recognized voice command based on that the smart rearview mirror control mode is a voice control mode; determining at least one camera to receive a captured image corresponding to the determined view type; and constructing the view screen based on the captured image obtained from the determined at least one camera, wherein the view type includes at least one of a front view, a left view, a right view, a rear view, a left / right view, a front surround view, a rear surround view and a bird's-eye view.

[0019] In an embodiment, the method may include: initiating a user registration step according to a predetermined user menu selection on the user interface screen; extracting characteristic data of the authentication unit of each user; extracting first characteristic data by recognizing the user's voice from the voice signal input through the microphone; extracting second characteristic data by recognizing the user's iris from the image captured by the camera; extracting third characteristic data by recognizing the user's fingerprint through a fingerprint sensor; determining the authentication weight value of each characteristic data based on pre-learned first template data, second template data and third template data, the pre-learned first template data, second template data and third template data corresponding to the first characteristic data, second characteristic data and third characteristic data respectively; determining the authentication level of the corresponding user based on the determined authentication weight value of each characteristic data, storing the extracted characteristic data and information related to the determined authentication level of each user in an internal memory and managing them.

[0020] In an embodiment, the method may further include detecting approaching obstacles using a Smart Parking Assistance System (SPAS) sensor equipped to the vehicle during driving, and displaying information of the detected approaching obstacles on one side of the view screen based on the detected approaching obstacles.

[0021] Another embodiment of the present invention is a smart rearview mirror control system arranged in a vehicle and may include: a microphone, providing a voice signal input by a user; a smart rearview mirror, provided with a digital display device; a camera system, photographing the front, left / right sides and rear of the above-mentioned vehicle; a smart rearview mirror control device, based on the above-mentioned smart rearview mirror control mode being a voice control mode, using the onboard voice recognition engine to perform voice recognition on the above-mentioned voice signal, receiving an image captured by at least one camera from the above-mentioned camera system according to the user voice command recognized by the above-mentioned voice recognition, constructing a dynamic view picture based on the at least one received image and providing the constructed above-mentioned view picture to the above-mentioned smart rearview mirror.

[0022] In an embodiment, the above-mentioned intelligent rearview mirror control device can also perform user authentication based on the above-mentioned voice signal, and the above-mentioned user authentication includes: speaker-independent authentication, which performs authentication only on a specific pre-registered speaker; and speaker-dependent authentication, which performs authentication on all speakers registered in the database.

[0023] In an embodiment, the above-mentioned smart rearview mirror control device can use the smart parking assistance system (SPAS) sensor equipped to detect approaching obstacles while the above-mentioned vehicle is driving, and based on the detected approaching obstacles, display the detected approaching obstacle information on one side of the above-mentioned view screen.

[0024] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and ordinary technicians in the technical field to which the present invention belongs can clearly understand other technical problems not mentioned through the following description.

[0025] The advantage of the present invention is that it can provide an intelligent rearview mirror control method based on voice recognition artificial intelligence and a device and system therefor.

[0026] Furthermore, the present invention has the advantage of performing minimal authentication steps according to the authentication level determined by each user based on artificial intelligence, thereby providing a user-customized smart rearview mirror control technology that can minimize user inconvenience caused by complicated user authentication.

[0027] Furthermore, the advantage of the present invention is that even during driving, the voice recognition artificial intelligence can be used to conveniently control the smart rearview mirror image, thereby providing driving safety.

[0028] Furthermore, the present invention has the advantage of providing a smart rearview mirror control device that can effectively cut off unnecessary power waste.

[0029] Furthermore, the advantage of the present invention is that it provides a variety of user authentication engines and voice recognition engines through an artificial intelligence platform, thereby providing a safer and more convenient voice recognition interaction system.

[0030] Furthermore, various effects directly or indirectly grasped by this text can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention is a state diagram of an intelligent rearview mirror control device based on artificial intelligence speech recognition according to an embodiment.

[0032] Figure 2 The flowchart is used to illustrate the initial startup steps of the intelligent rearview mirror control device of the embodiment.

[0033] Figure 3 The flowchart is used to illustrate the intelligent rearview mirror control method of the intelligent rearview mirror control device in the parking / parking state according to the embodiment.

[0034] Figure 4 FIG. 1 is a flowchart for explaining a user registration procedure in the smart rearview mirror control device of the embodiment.

[0035] Figure 5 FIG. 1 is a flowchart for explaining a user authentication step in the smart rearview mirror control device according to the embodiment.

[0036] Figure 6 A flowchart for explaining a user authentication procedure of the authentication application speaker mode according to the embodiment.

[0037] Figure 7 The invention is a block diagram for explaining the structure of an intelligent rearview mirror system based on speech recognition artificial intelligence (AI) according to an embodiment.

[0038] Figure 8 An example of a view image provided by the smart rearview mirror system of an embodiment.

[0039] Fig. 9 FIG. 4 is a block diagram for illustrating a detailed structure of an intelligent rearview mirror system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. When assigning reference numerals to structural elements in various drawings, the same reference numerals are assigned to the same structural elements as much as possible even if they are shown in different drawings. In addition, in the process of describing the present invention, if it is determined that the specific description of the relevant known structure or function hinders the understanding of the embodiments of the present invention, the detailed description thereof will be omitted.

[0041] When describing the structural elements of the embodiments of the present invention, terms such as first, second, A, B, (a), (b) and the like may be used. These terms are only used to distinguish the structural element from other structural elements, and the nature, order or sequence of the structural element are not limited to these terms. Furthermore, unless otherwise defined, the meanings of all terms used herein, including technical or scientific terms, are the same as those commonly understood by those of ordinary skill in the art to which the present invention belongs. The meanings of terms defined in commonly used dictionaries are the same as those in the context of the relevant technology, and shall not be interpreted as ideal or overly formalized meanings unless clearly defined in this application.

[0042] Figure 1 The present invention is a state diagram of an intelligent rearview mirror control device based on artificial intelligence speech recognition according to an embodiment.

[0043] Reference Figure 1The intelligent rearview mirror control status (or intelligent reversing mirror control device) can generally include a standby status 10 (Standby Status), a wakeup status 20 (Wakeup Status), an analysis status 30 (Analysis Status), a result feedback and control status 40 (Result Feedback and Control Status) and a lock status 50 (Lock Status).

[0044] When the vehicle is stopped or in motion, when the smart rearview mirror control device (or smart rearview mirror control switch) for controlling the vehicle rearview mirror is in the always on display (AOD) working mode, the standby state 10 may mean a state of waiting for the driver to input a specific gesture. The display screen of the smart rearview mirror control device in the standby state 10, for example, a touch light emitting diode (LED) screen may be in a closed state, that is, in an OFF state.

[0045] In one example, when the driver performs a pre-set or registered gesture in the smart rearview mirror control device in the standby state 10, the smart rearview mirror control device recognizes the driver's gesture through the proximity sensor it is equipped with, and based on the recognition that the gesture is consistent with the pre-set / registered gesture, the display screen in the closed state can be switched to the activated state, that is, the on (ON) state.

[0046] When the display screen is activated, the smart rearview mirror control device may transition from the standby state 10 to the awake state 20 to output a predetermined notification message notifying that the user's voice state can be recognized through the vehicle speaker.

[0047] When a user's voice is input through an equipped microphone, the smart rearview mirror control device may transition from the awakening state 20 to the analyzing state 30 .

[0048] The intelligent rearview mirror control device can analyze the input user voice signal through the artificial intelligence voice recognition engine in the analysis state 20, so as to recognize the user voice command, for example, the wake-up command (Wake-up Word).

[0049] When the voice recognition is completed normally, the smart rearview mirror control device can be changed to the result feedback state 40, and a predetermined control instruction corresponding to the voice recognition result is transmitted to the camera system. According to the control instruction received from the smart rearview mirror control device, the camera system can transmit the image captured by the corresponding camera to the smart rearview mirror control device. The smart rearview mirror control device can output the image received by the camera system through a screen set on the rearview mirror.

[0050] In an embodiment, the smart rearview mirror control device can simultaneously receive images captured by multiple cameras. In this case, the smart rearview mirror control device can adaptively form a picture according to the type of camera receiving the image and output it to the rearview mirror. In one example, the camera may include a front camera, a left / right view camera, and a rear camera.

[0051] In one example, when the smart rearview mirror control device receives images captured by the left / right view camera, the smart rearview mirror control device can control the left half of the rearview mirror screen to display the image captured by the left view camera, and control the right half of the rearview mirror screen to display the image captured by the right view camera.

[0052] In another example, when the intelligent rearview mirror control device receives images taken by the left / right side view camera, the rearview mirror screen can be divided into three areas, namely the first area (left side), the second area (center), and the third area (right side). The intelligent rearview mirror control device controls the first area to display the image taken by the left side view camera, controls the second area to work as a reversing mirror, and controls the third area to display the image taken by the right side view camera.

[0053] In another example, when the smart rearview mirror control device receives images captured by the left / right view cameras and the front camera, the smart rearview mirror control device can construct a front surround view picture based on the received multiple images and output it to the rearview mirror.

[0054] In another example, when the smart rearview mirror control device receives images captured by the left / right view camera and the rear camera, the smart rearview mirror control device can compose a rear surround view screen based on the received multiple images and output it to the rearview mirror.

[0055] In another example, when the smart rearview mirror control device receives images captured by the front camera, left / right view cameras, and rear camera, the smart rearview mirror control device composes a bird's-eye view screen based on the received multiple images and outputs it to the rearview mirror.

[0056] In the analysis state 30 , when the user voice recognition fails, the smart rearview mirror control device may transition to the result feedback state 40 , and output a predetermined notification message notifying the voice recognition failure through the vehicle speaker (or rearview mirror screen).

[0057] According to the pre-set safety and / or locking mode, the smart rearview mirror control device can perform user identification and authentication by performing machine learning (or deep learning) on ​​the input voice signal.

[0058] When user identification and authentication are successful, the smart rearview mirror control device can recognize user voice commands and control the rearview mirror image through voice recognition. When user identification or authentication fails, the smart rearview mirror control device can output a predetermined warning alarm message to notify the user of the identification or authentication failure and transition from the result feedback state 40 to the wake-up state 20.

[0059] When user identification and authentication fails for more than a predetermined number of times, the smart rearview mirror control device may output a predetermined warning alarm message notifying an unregistered user and then transition to the locked state 50. In one example, in the locked state 50, the smart rearview mirror is used only as a rearview mirror.

[0060] In the locked state 50, according to the pre-set safety and / or lock mode level, the smart rearview mirror control device can perform at least one biometric recognition of fingerprint recognition, iris recognition, face recognition and weight recognition to perform user identification and authentication. If the biometric recognition is successful, the smart rearview mirror control device can change from the awake state 20 and wait for the user's voice command.

[0061] According to the menu setting, when the driver speaks a pre-registered or set start command (StartCommand) in the standby state 10, the intelligent rearview mirror control device can also switch the display screen in the closed state to the active state, that is, the open (ON) state.

[0062] In an embodiment, when the result feedback is completed, the intelligent rearview mirror control device may return to the awake state 20 and wait for the next user instruction.

[0063] The smart rearview mirror control device may drive a predefined time wake-up timer whenever entering the wake-up state 20. The smart rearview mirror control device may recognize a voice command for controlling the smart rearview mirror only when the wake-up timer is being driven. If, in the case where the wake-up timer expires, the smart rearview mirror control device may transition to the standby state 10 and turn off the display screen.

[0064] According to menu settings, when the number of user identification and authentication failures in the awake state 20 reaches or exceeds a specified number, the smart rearview mirror control device of the embodiment can also be transformed into the standby state 10 .

[0065] In any of the wake-up state 20, the analysis state 30, and the result feedback state 40, the intelligent rearview mirror control device of the embodiment can also be switched to the standby state 10 when a pre-registered exit command (Quit Command) is detected through voice recognition or a specific control signal is received, for example, a specific button input signal set in the intelligent rearview mirror control device.

[0066] Figure 2The flowchart is used to illustrate the initial startup steps of the intelligent rearview mirror control device of the embodiment.

[0067] When power is applied, the smart rearview mirror control device may initialize a main controller, ie, a processor, by booting / loading (step S210 ).

[0068] The main controller may initialize a communication board for communicating with external devices and systems (step S220 ).

[0069] In one example, when the communication board is initialized, the smart rearview mirror control device can be linked with various electronic control units (ECU, Electric Control Unit) and vehicle terminals provided in the vehicle. The smart rearview mirror control device can be linked with external devices, such as road side units (RSU, Road Side Unit), base stations (Base Station), other vehicle terminals, etc. through the vehicle terminal. In one example, the smart rearview mirror control device can obtain peripheral image signals taken by the roadside unit (RSU), image signals taken by other vehicle terminals, etc. through the vehicle terminal. In this case, the smart rearview mirror control device can display the peripheral images taken by the roadside unit (RSU) and (or) images taken by other vehicle terminals through its own smart rearview mirror.

[0070] Also, the main controller may activate an always on display (AOD) operation mode by initializing the display panel (step S230).

[0071] The main controller can complete the startup of the smart rearview mirror control device by initializing the sensor board (step S240). The main controller can receive the image signal captured by the vehicle camera and the user voice signal input through the microphone through the sensor board. In addition, the main controller can also receive iris recognition signals, fingerprint recognition signals, gesture recognition signals, touch recognition signals, illumination sensing signals, etc. through the sensor board.

[0072] The main controller can maintain the sensing accuracy of the multiple sensors by calibrating the various sensors connected to the sensor board. The sensors connected to the sensor board may include touch sensors, gesture sensors, microphones, cameras, iris recognition sensors, fingerprint recognition sensors, etc., but are not limited thereto.

[0073] The main controller of the embodiment may also send and receive signals with various cameras installed in the vehicle through the communication board.

[0074] Figure 3 The flowchart is used to illustrate the intelligent rearview mirror control method of the intelligent rearview mirror control device in the parking / parking state according to the embodiment.

[0075] Reference Figure 3 In the state where the light-emitting diode (LED) screen in the always-on display (AOD) working mode is turned off, that is, when the screen is turned off, the smart rearview mirror control device can recognize the user gesture through the equipped sensor, such as the proximity sensor (step S310).

[0076] Based on the recognized user gesture, the smart rearview mirror control device may output a smart rearview mirror control interface screen by switching the display screen from an off state to an on state (step S320).

[0077] The smart rearview mirror control device can recognize the user touch operation or user voice command on the smart rearview mirror control interface screen (step S330).

[0078] In one example, the smart rearview mirror control interface screen may include a touch button for selecting the type of camera image displayed in the smart rearview mirror, a touch button for turning on / off the smart rearview mirror, and a touch button for turning on / off the smart rearview mirror.

[0079] A touch button for closing control, a touch button for biometrics, etc., but not limited thereto. The camera image type, i.e., the view type may include a front view, a left view, a right view, a rear view, a left / right view, a front surround view, a rear surround view, a bird's-eye view, etc., but not limited thereto.

[0080] Furthermore, the user voice command may also include a plurality of predetermined voice commands for selecting the camera image type, which is displayed on the rearview mirror. In one example, the voice command may include a front view, a left view, a right view, a rear view, a left / right view, a front surround view, a rear surround view, a bird's eye view, etc., but is not limited thereto.

[0081] The intelligent rearview mirror control device can transmit a predetermined control instruction to the camera system according to the detected touch operation or the recognized voice instruction and obtain the captured images of the corresponding (multiple) cameras (step S340).

[0082] The smart rearview mirror control device can compose a picture based on the acquired camera (or cameras) and display it on the smart rearview mirror (step S350).

[0083] Figure 4 FIG. 1 is a flowchart for explaining a user registration procedure in the smart rearview mirror control device of the embodiment.

[0084] Reference Figure 4 , the smart rearview mirror control device can initiate a user registration step (step S410) according to the user menu selection on the user interface screen.

[0085] The smart rearview mirror control device may recognize the user's voice through a microphone to extract first characteristic data (step S420).

[0086] The smart rearview mirror control device may extract second characteristic data by recognizing the user's iris through the iris sensor (step S430 ).

[0087] The smart rearview mirror control device may extract third characteristic data by identifying the user's fingerprint through the fingerprint sensor (step S440).

[0088] The intelligent rearview mirror control device can determine the authentication weighted value of each characteristic data based on the pre-learned first template data, second template data and third template data, and the above-mentioned pre-learned first template data, second template data and third template data correspond to the first characteristic data, second characteristic data and third characteristic data respectively (step S450).

[0089] The authentication level of the corresponding user may be determined and registered based on the determined authentication weight value of each characteristic data (step S460). The extracted characteristic data and the determined authentication level information of each user may be stored and maintained in a user database and managed.

[0090] In one example, some people may have stronger voice characteristics than others. This may mean that it is easier to identify and authenticate users through voice. Also, some people have weak voice characteristics but stronger iris characteristics. In this case, the accuracy of user identification and authentication can be improved by setting a high iris authentication weighting value compared to voice. As described above, the present application can adaptively determine the authentication weighting value based on the characteristics of each user authentication data, which has the advantage of not only improving the reliability of user identification and authentication by determining the optimal authentication level, but also minimizing user inconvenience by minimizing unnecessary user authentications.

[0091] In the above Figure 4 In the embodiment, voice recognition, iris recognition, fingerprint recognition, etc. have been described as user authentication units, but this is only one embodiment, and other biometric recognition units such as face recognition, weight recognition, etc. can also be added.

[0092] Figure 5 FIG. 1 is a flowchart for explaining a user authentication step in the smart rearview mirror control device according to the embodiment.

[0093] Reference Figure 5 , when the user authentication step is initiated, the smart rearview mirror control device can activate the biometric recognition engine (step S510).

[0094] Among them, the biometric recognition engine may include at least one of a voice recognition engine, an iris recognition engine, a fingerprint recognition engine, a face recognition engine, and a gesture recognition engine.

[0095] The smart rearview mirror control device may obtain at least one characteristic data through at least one biometric recognition engine (step S520).

[0096] The intelligent rearview mirror control device may select a candidate user from the user database based on the acquired characteristic data (step S530 ). There may be multiple candidate users according to the characteristic data.

[0097] The smart rearview mirror control device may identify a pre-registered authentication level corresponding to the selected candidate user (step S540 ).

[0098] The smart rearview mirror control device may determine whether additional authentication is required according to the identified authentication level (step S550 ).

[0099] In the case where additional authentication is required, the smart rearview mirror control device may finally authenticate the candidate user(s) by performing additional authentication steps corresponding to the corresponding authentication levels (step S560 ).

[0100] In the above step S550 , when it is determined that additional authentication is not required, the smart rearview mirror control device may determine the selected candidate user as the final authenticated user.

[0101] As described above, the present application can adaptively determine the biometric recognition unit used to authenticate the user according to the pre-registered authentication level, thereby having the advantage of more effectively identifying and authenticating the user.

[0102] Also, the present application performs a minimum number of authentication steps according to a predetermined authentication level of each user, thereby having an advantage in that user inconvenience can be minimized.

[0103] Figure 6 A flowchart for explaining a user authentication procedure of the authentication application speaker mode according to the embodiment.

[0104] Reference Figure 6 , the smart rearview mirror control device can be set to the authentication application speaker mode through a user's predetermined menu selection (step S610).

[0105] Among them, the speaker mode of the authentication application can be divided into speaker-dependent mode and speaker-independent mode.

[0106] The speaker dependent mode is a mode in which a user authentication procedure is performed only on speakers registered (selected) in advance by the user.

[0107] The speaker-independent mode is a mode in which the user authentication procedure is performed on all speakers registered in the database.

[0108] In the case of speaker-dependent mode, the user authentication step is performed only for a specific speaker. Therefore, the advantage is that user authentication can be performed faster, but the problem is that authentication fails for users who are not registered (or selected) in the speaker-dependent mode.

[0109] In contrast, the speaker-independent mode performs user authentication on all speakers registered in the database. Therefore, its advantage is that the probability of authentication failure is very low, but its disadvantage is that the time required for user authentication increases compared to the speaker-dependent mode.

[0110] When the user authentication step is initiated (step S620 ), the smart rearview mirror control device may determine whether the currently set authentication application speaker mode is speaker-dependent or speaker-independent (step S630 ).

[0111] If the result of the determination is that the mode is speaker dependent, the smart rearview mirror control device may perform user authentication only for a specific speaker registered (or selected) in advance (step S640 ).

[0112] If the result of determination in step S630 is that the mode is speaker-independent, the smart rearview mirror control device may perform user authentication on all speakers registered in the database (step S650 ).

[0113] Figure 7 The invention is a block diagram for explaining the structure of an intelligent rearview mirror system based on speech recognition artificial intelligence (AI) according to an embodiment.

[0114] Reference Figure 7 The smart rearview mirror system 700 may include a microphone 710 , a voice recognition processor 720 , a main controller 730 , a camera system 740 , and a smart rearview mirror 750 .

[0115] Although the camera system 740 of the embodiment may include a left view camera 741, a right view camera 742, a front view camera 743, and a rear view camera 744, this is only an embodiment and may include more or less cameras than these.

[0116] In yet another embodiment, the camera system 740 may further include a surround view monitor (SVM) camera. The surround view monitor (SVM) camera is installed at the front / rear / left / right side of the vehicle, etc., and provides not only a wide view (front camera view) but also a front top view (or front surround view) (front / left / right camera composite view), a left side view (left camera view), a right side view (right camera view), a rear view (rear camera view), a rear top view (or rear surround view) (rear / left / right camera composite view), etc.

[0117] In another embodiment, the camera system 740 may also include a Smart Parking Assistance System (SPAS) sensor. In this case, the main controller 730 may detect short-distance obstacles based on sensing information received from the Smart Parking Assistance System (SPAS) sensor. When a short-distance obstacle is detected, the main controller 730 may display the short-distance obstacle detection information through the smart rearview mirror 750. In one example, the main controller 730 may identify the detected obstacle position based on the sensing information received from the Smart Parking Assistance System (SPAS) sensor. The main controller 730 may match the identified obstacle to the camera view corresponding to the identified obstacle position and display it through the smart rearview mirror 750. In one example, when an obstacle is detected on the left side of the vehicle, the main controller 730 may transmit an image containing the obstacle detection result in the image captured by the left camera to the smart rearview mirror 750 and display it. Therefore, its advantage is that the driver can intuitively confirm where the short-distance obstacle is.

[0118] Hereinafter, the steps of forming the screen of the smart rearview mirror 750 by the smart rearview mirror system 700 based on voice recognition artificial intelligence (AI) will be described in detail.

[0119] A user voice signal input through the microphone 710 may be transmitted to the voice recognition processor 720 .

[0120] The voice recognition processor 720 may extract keywords, ie, user voice commands, by performing natural language processing on the voice signal through a voice recognition engine installed therein after removing noise included in the received voice signal.

[0121] The main controller 730 controls based on the user voice command received from the voice recognition processor 720 to transmit the corresponding camera-captured image to the camera system 740 .

[0122] The main controller 730 may construct a corresponding view image to be displayed on the smart rearview mirror 750 based on the image signal received from the camera system 740, and transmit the constructed view image to the smart rearview mirror 750 for display. Figure 8 As shown, the view screen that can be constructed by the main controller 730 can be variously constructed according to the user's voice command.

[0123] Figure 8 An example of a view image provided by the smart rearview mirror system of an embodiment.

[0124] Reference Figure 8 The view screens that can be displayed on the smart rearview mirror 750 can roughly include a left camera view screen 810, a right camera view screen 820, a side camera view screen 830, a front camera view screen 840, a front (or rear) surround view screen 850 and a bird's-eye view screen 860.

[0125] The smart rearview mirror system 700 can adaptively construct a view image according to the user's voice command and display it on the smart rearview mirror 750.

[0126] When detecting short-distance obstacles by linking with the smart parking assist system (SPAS) sensor, the smart rearview mirror system 700 of the embodiment can further display obstacle detection information on one side of the view screen. The obstacle detection information may include obstacle location information and obstacle type information, but is not limited thereto.

[0127] In an embodiment, as shown in reference numerals 810 to 840 , the smart rearview mirror 750 may further include a partial mirror area according to the kind of the view screen, that is, the view type.

[0128] Fig. 9 FIG. 4 is a block diagram for illustrating a detailed structure of an intelligent rearview mirror system according to an embodiment of the present invention.

[0129] like Fig. 9 As shown, the smart rearview mirror system 900 may generally include a smart rearview mirror control device 910 , a communication board 920 , a display board 930 , and a sensor board 940 .

[0130] The communication board 920 can communicate by connecting with the vehicle electronic control unit (ECU) and / or the vehicle terminal 921. In one example, according to the request of the smart rearview mirror control device 910, the communication board 920 can receive an image captured by another vehicle camera through the vehicle terminal 921 and transmit it to the smart rearview mirror control device 910.

[0131] Also, the communication board 920 may transmit the view screen connected to the smart rearview mirror 750 and formed by the smart rearview mirror control device 910 to the smart rearview mirror 750 .

[0132] The display panel 930 can be connected to the light emitting diode 931 (LED), and the display panel 930 can display the user interface screen and various status information of the smart rearview mirror system 900 on the light emitting diode 931 (LED) according to the control signal of the smart rearview mirror control device 910. The display panel 620 can control the light emitting diode 931 (LED) in the always-on display (AOD) working mode.

[0133] The sensor board 940 can be connected to at least one of the illumination sensor 941, the proximity sensor 942, the touch sensor 943, the fingerprint sensor 944, the iris sensor 945, and the microphone 710, and provide the sensing information obtained from the corresponding sensor to the smart rearview mirror control device 910. The smart rearview mirror control device 910 can perform gesture recognition, user authentication, user voice command recognition, etc. based on the sensing information received from at least one of the illumination sensor 941, the proximity sensor 942, the touch sensor 943, the fingerprint sensor 944, the iris sensor 945, and the microphone 710.

[0134] Furthermore, the sensor board 940 can be connected to the camera system 740. The sensor board 940 can receive a captured image from a corresponding camera according to a control signal of the smart rearview mirror control device 910, and provide the received image to the smart rearview mirror control device 910.

[0135] And, the sensor board 940 can also be connected with a smart parking assist system (SPAS) sensor 946. The sensor board 940 can provide sensing information received from the smart parking assist system (SPAS) sensor 946 to the smart rearview mirror control device 910. In one example, the smart rearview mirror control device 910 can detect a short-distance obstacle based on the sensing information from the smart parking assist system (SPAS) sensor 946.

[0136] When the sensor board 940 is initialized, a calibration operation may be performed on the connected plurality of sensors.

[0137] The illuminance sensor 941 may measure the illuminance around the smart rearview mirror control device 910 and / or the smart rearview mirror 750. The sensor board 940 may transmit the collected illuminance sensing information to the main controller 950.

[0138] Reference Fig. 9 The smart rearview mirror control device 910 may include a main controller 950, a memory 960, and various engines.

[0139] Among them, the engine may include a voice command recognition engine 970, a gesture recognition engine 980, a touch recognition engine 990, a user authentication engine 995, etc.

[0140] The main controller 950 can transmit the picture brightness control instruction to the display panel 930 according to the illumination sensing information and dynamically control the brightness of the picture output by the light emitting diode 931 (LED) and the brightness of the picture output by the smart rearview mirror 750 according to the peripheral illumination.

[0141] When the always-on display (AOD) screen is turned off, the proximity sensor 942 may be used to sense a user gesture within a specified distance. In one example, when the proximity sensor 942 detects a capacitance change above a specified reference value, sensing information related to the detected capacitance change may be transmitted to the main controller 950 through the sensor board 940. In another example, the proximity sensor 942 may include a Hall sensor, and the change in illumination is detected by the Hall sensor. Sensing information related to the illumination change measured by the proximity sensor 942 may be transmitted to the main controller 950 through the sensor board 940. The information sensed by the proximity sensor 942 may be used to recognize a user gesture.

[0142] The main controller 950 may recognize a user gesture corresponding to the proximity sensing information using the gesture recognition engine 980 .

[0143] The main controller 950 may activate a light emitting diode 931 (LED) screen in an always on display (AOD) operating mode based on a recognized user gesture.

[0144] Furthermore, when the light emitting diode 931 (LED) screen in the always-on display (AOD) working mode is activated, the main controller 950 can construct a user interface screen for controlling the smart rearview mirror, transmit the constructed user interface screen to the display panel 930, and display it on the light emitting diode 931 (LED) screen.

[0145] When acquiring touch sensing information from the touch sensor 943 , the main controller 950 may recognize a user touch operation using the touch recognition engine 990 .

[0146] The main controller 950 may construct a view screen according to the recognized touch operation and transmit it to the smart rearview mirror 750 through the communication board 920 .

[0147] According to the recognized touch operation, the main controller 950 can switch the control mode of the smart rearview mirror 750 from the manual control mode to the voice control mode or from the voice control mode to the manual control mode.

[0148] In one example, in manual control mode, the view screen of the smart rearview mirror 750 can be dynamically controlled according to the user menu selection on the user interface screen, and in voice control mode, the view screen of the smart rearview mirror 750 can be dynamically controlled based on user voice commands.

[0149] In an embodiment, the other user authentication engine 995 may include at least one of a voice recognition engine 996 , an iris recognition engine 997 , a fingerprint recognition engine 998 , and an authentication level determination engine 999 .

[0150] The main controller 950 can extract characteristic data based on at least one of the fingerprint sensing information obtained from the fingerprint sensor 944, the image sensing information obtained from the iris sensor 945, the voice sensing information of the user voice input through the microphone 710, and the iris sensing information collected from the iris sensor 945, and dynamically determine the authentication level for the corresponding user by performing machine learning on the extracted characteristic data.

[0151] The speech recognition engine 996 can extract characteristic data of the user's speech based on the speech sensing information, and determine the authentication weight value of the corresponding speech characteristic data by comparing the extracted speech characteristic data with the learned speech template data.

[0152] The speech recognition engine 996 can improve the speech recognition sensitivity by removing noise contained in the speech signal.

[0153] The iris recognition engine 997 may extract characteristic data of the user's iris based on the image sensing information, and determine an authentication weight value corresponding to the iris characteristic data by comparing the extracted iris characteristic data with the learned iris template data.

[0154] The fingerprint recognition engine 998 may extract characteristic data of the user's fingerprint based on the fingerprint sensing information, and determine an authentication weight value of the corresponding fingerprint characteristic data by comparing the extracted fingerprint characteristic data with the learned fingerprint template data.

[0155] The main controller 950 may determine an authentication level of a corresponding user based on the determined authentication weight value of each characteristic data.

[0156] The main controller 950 may sort the characteristic data in order of the authentication weighted values ​​from high to low and sum up the authentication weighted values ​​until the cumulative sum of the authentication weighted values ​​of the sorted characteristic data reaches a predetermined threshold.

[0157] In the event that the cumulative total authentication weight value reaches above a threshold, the main controller 950 may determine an authentication level based on the total characteristic data type and quantity.

[0158] In one example, in the case where the threshold is reached only by the authentication weighted value corresponding to the voice characteristic data, the authentication level may be determined as the first level. On the contrary, in the case where the cumulative sum of the authentication weighted value corresponding to the voice characteristic data and the authentication weighted value corresponding to the iris characteristic data reaches or exceeds the threshold, the authentication level may be determined as the second level. On the contrary, in the case where the cumulative sum of the authentication weighted value corresponding to the voice characteristic data, the authentication weighted value corresponding to the iris characteristic data, and the authentication weighted value corresponding to the fingerprint characteristic data reaches or exceeds the threshold, the authentication level may be determined as the third level.

[0159] In the memory 960, characteristic data and authentication level of each user may be stored.

[0160] Also, pre-learned template data for each authentication type may be maintained in the memory 960 .

[0161] The main controller 950 can control the above Figures 1 to 8 The operation of the intelligent rearview mirror control device described in .

[0162] When user authentication is completed, the voice command recognition engine 940 may extract keywords by performing natural language processing on the input user voice signal through the microphone 710 , and provide the extracted keywords to the main controller 950 .

[0163] The main controller 950 may construct a dynamic view screen based on a keyword, ie, a user voice command, received from the voice command recognition engine 940 , and transmit the constructed view screen to the smart rearview mirror 750 through the communication board 920 .

[0164] The steps of the methods or algorithms related to and described in the embodiments disclosed in this specification may be implemented by hardware, software modules, or a combination of the two, which are executed by a processor. The software modules may reside in a storage medium (i.e., a memory and / or a storage device) such as a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disk (CD-ROM), or the like.

[0165] An exemplary storage medium may be coupled to a processor, which may read information from the storage medium and may write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The application specific integrated circuit (ASIC) may also reside in a user terminal. Alternatively, the processor and the storage medium may also reside in a user terminal as separate components.

[0166] The above description is only an exemplary description of the technical concept of the present invention, and a person skilled in the art can make various modifications and variations without departing from the essential features.

[0167] Therefore, the embodiments disclosed in the present invention are only for illustration, and are not intended to limit the technical ideas of the present invention, and the scope of the technical ideas of the present invention is not limited to the above embodiments. The protection scope of the present invention should be interpreted by the protection scope of the attached invention claims, and all technical ideas within the equivalent scope should be interpreted as included in the protection scope of the present invention.

Claims

1. A smart rearview mirror control method, a smart rearview mirror control method installed in a smart rearview mirror control device of a vehicle, characterized in that: include: performing a startup step as power is applied; Recognizing user gestures when the always-on touch display is turned off; Activate the closed touch display screen according to the recognized user gesture, output a user interface screen for controlling the smart rearview mirror, and then wait for the user to input a voice; Performing machine learning on a user voice signal input through a microphone to perform user authentication and recognize voice commands for controlling the smart rearview mirror; and When the user authentication is successful, a view screen is formed based on the recognized voice command and displayed on the smart rearview mirror. Based on the above-mentioned intelligent rearview mirror control mode being a manual control mode, it also includes detecting a user touch operation on the above-mentioned user interface screen, When the above-mentioned touch operation is detected, the intelligent camera control instruction corresponding to the above-mentioned touch operation is identified, and the corresponding camera shooting image is obtained according to the identified above-mentioned intelligent camera control instruction to form the above-mentioned view screen.

2. The intelligent rearview mirror control method according to claim 1, characterized in that: The above startup steps include: Initialize a main controller that controls the overall input and output of the smart rearview mirror control device; Realize communication connection with multiple cameras; Initializing the display panel to control the output of the touch display screen; and calibrating at least one sensor, initializing a sensor board configured to collect sensing information from the at least one sensor, The sensor includes at least one of an illumination sensor, a proximity sensor, a touch sensor, a fingerprint sensor, and an iris sensor.

3. The intelligent rearview mirror control method according to claim 1, characterized in that: The above user authentication includes: Speaker-independent authentication, which performs authentication only on specific pre-registered speakers; and Speaker-dependent authentication performs authentication on all speakers registered in the database.

4. The intelligent rearview mirror control method according to claim 1, characterized in that: Also includes: Based on the intelligent rearview mirror control mode being a voice control mode, determining a view type corresponding to the recognized voice command; Determining at least one camera to receive a captured image corresponding to the determined view type; and The view image is formed based on the captured image obtained from the at least one camera determined above, The above view types include at least one of a front view, a left view, a right view, a rear view, a left / right view, a front surround view, a rear surround view and a bird's-eye view.

5. The intelligent rearview mirror control method according to claim 1, characterized in that: include: Initiating a user registration step according to a predetermined user menu selection on the above user interface screen; Extracting characteristic data of each user authentication unit; Recognizing a user's voice from a voice signal input through the microphone to extract first characteristic data; Recognizing the user's iris through the iris sensor to extract second characteristic data; Extracting third characteristic data by identifying the user's fingerprint through the fingerprint sensor; determining an authentication weighted value of each characteristic data based on pre-learned first template data, second template data, and third template data, the pre-learned first template data, second template data, and third template data corresponding to the first characteristic data, second characteristic data, and third characteristic data, respectively; and Determine the authentication level of the corresponding user based on the determined authentication weight value of each of the above characteristic data, The extracted characteristic data of each user and information related to the determined authentication level are stored in an internal memory and managed.

6. The intelligent rearview mirror control method according to claim 1, characterized in that: The method also includes: using the equipped intelligent parking assistance system sensor to detect approaching obstacles during the driving process of the above-mentioned vehicle, Based on the detected approaching obstacle, information of the detected approaching obstacle is displayed on one side of the view screen.

7. An intelligent rearview mirror control system, arranged in a vehicle, characterized in that: include: A microphone provides a voice signal input by the user; An intelligent rearview mirror provided with a digital display device; A camera system for capturing images of the front, left / right sides, and rear of the above-mentioned vehicle; as well as The intelligent rearview mirror control device, based on the intelligent rearview mirror control mode being a voice control mode, performs voice recognition on the voice signal by using a voice recognition engine, receives images taken by at least one camera from the camera system according to the user voice command recognized by the voice recognition, constructs a dynamic view screen based on the at least one received image, and provides the constructed view screen to the intelligent rearview mirror, and based on the intelligent rearview mirror control mode being a manual control mode, detects user touch operation on a user interface screen, identifies the camera control command corresponding to the detected touch operation, receives images taken by at least one camera from the camera system according to the recognized camera control command to construct the view screen, and provides the constructed view screen to the intelligent rearview mirror.

8. The intelligent rearview mirror control system according to claim 7, characterized in that: Based on the voice signal, the smart rearview mirror control device also performs user authentication. The above user authentication includes: Speaker-independent authentication, which performs authentication only on specific pre-registered speakers; and Speaker-dependent authentication performs authentication on all speakers registered in the database.

9. The intelligent rearview mirror control system according to claim 7, characterized in that: The intelligent rearview mirror control device detects approaching obstacles using the intelligent parking assistance system sensor when the vehicle is traveling, and displays the detected approaching obstacle information on one side of the view screen based on the detection of the approaching obstacle.

Images