Intelligent vehicle assistance system with artificial intelligence

Abstract

The invention relates to an intelligent automobile auxiliary system with artificial intelligence, the intelligent automobile auxiliary system is designed to be used in an automobile, a VIP vehicle, a commercial vehicle and an intelligent home system, the intelligent automobile auxiliary system recognizes the face of a user, detects the emotional state of the user, provides suggestions according to the detected emotions, and sends the suggestions to the user. The present invention relates to a vehicle-mounted device having a two-dimensional or three-dimensional holographic face to establish an emotional link between the user and the vehicle and produce a feeling of a real human during a conversation, with the ability to chat, acquire information, answer questions and answers, dictate text, read news, provide weather information, send emails, and allow voice control of devices in the vehicle, different languages may be translated, vehicle failures may be informed, range information may be transmitted, enabling it to perceive commands issued by conversations in daily and natural spoken languages, Turkish languages, or a desired language, and to answer questions proposed, and also allowing shopping while within the vehicle.

Description

Technical Field

[0001] The present invention relates to an intelligent car assistance system with artificial intelligence, which is designed for use in automobiles, VIP vehicles, commercial vehicles and smart home systems, and which recognizes the face of a user, detects the emotional state of the user, provides suggestions based on the detected emotions, and has a two-dimensional or three-dimensional holographic face to establish an emotional connection between the user and the vehicle and create a real human feeling when talking, has the ability to chat, obtain information, ask questions, dictate texts, read news, provide weather information, send emails and allow voice control of devices in the vehicle, can translate different languages, can inform vehicle failures, can transmit range information, enable it to perceive commands issued by talking in daily and natural spoken language, Turkish or a desired language, and answer questions asked, and also allows shopping while in the vehicle. Background Art

[0002] Humans form emotional bonds with other people or creatures and go about their lives because of them. It is wonderful to feel connected to someone and to feel that someone understands, cares for, and protects you. However, these feelings only occur between humans and other creatures.

[0003] Cars purchased with a high budget, VIP vehicles and similar vehicles do not have the qualities that allow users to establish an emotional connection and are unable to break through the limitations of their metal shells. There is no way to establish an emotional connection with the vehicles used today, to ensure communication in natural spoken language, these vehicles cannot recognize their owners, cannot detect and learn their owners' behavior and emotions, cannot answer questions asked, and most importantly, these vehicles do not have faces.

[0004] There are voice assistance systems in vehicles produced by some vehicle manufacturers, but these systems can only perform a limited number of operations, such as opening and closing and controlling devices such as multimedia and air conditioning. In addition, these systems only work under specific commands, and when different expressions other than these commands are spoken, the system cannot perceive the words.

[0005] In addition to the above-mentioned voice assistance systems, there are also applications provided by some mobile devices, and these applications only allow the use of functions of the mobile device, such as navigation and multimedia on the vehicle screen.

[0006] There is a need for an intelligent vehicle assistant that will establish an emotional connection with the vehicle being used, recognize and detect the user's face and emotions, make suggestions, converse and answer questions in natural spoken language, Turkish or desired language.

[0007] Therefore, the need to eliminate such shortcomings and deficiencies of embodiments and practices employed in the prior art requires improvements in the corresponding technical field. Summary of the invention

[0008] The present disclosure relates to a smart car assistant with artificial intelligence, which is developed to eliminate the above-mentioned deficiencies and provide new advantages in the corresponding technical field.

[0009] The object of the present invention is to provide an intelligent vehicle assistant having a face.

[0010] Another object of the present invention is to create an intelligent vehicle assistant that recognizes the user's face, detects emotional state, and provides suggestions based on the detected emotions.

[0011] Another object of the present invention is to provide an intelligent vehicle assistant that enables a user to establish an emotional connection with his or her vehicle.

[0012] Another object of the present invention is to provide an intelligent vehicle assistant having features such as reading news, providing weather information, sending emails, creating memos and recording alarms.

[0013] Another object of the present invention is to provide a smart vehicle assistant that allows voice control of devices in the vehicle by detecting what is said in everyday and natural spoken language, Turkish or a desired language, and also has the ability to answer questions asked.

[0014] Another purpose of the invention is to translate what is said in the vehicle into the desired language, thanks to its ability to translate different languages.

[0015] Another object of the present invention is to enable the user to be audibly informed of the remaining fuel or charge distance and possible vehicle malfunctions.

[0016] Another object of the present invention is to provide a structure that enables shopping by voice command while in a vehicle, thereby saving time and creating convenience for the user.

[0017] Upon reading the detailed description given below, the structural and characteristic features of the present disclosure, including all advantages thereof, will be more clearly understood, and thus the present disclosure should be evaluated by considering this detailed description. DETAILED DESCRIPTION

[0018] The preferred embodiments of the intelligent vehicle assistance system with artificial intelligence according to the present disclosure mentioned in this detailed description are merely intended to provide a better understanding of the subject matter and should not be interpreted in any limiting sense.

[0019] The present invention relates to an intelligent car assistance system with artificial intelligence, which is designed for use in automobiles, VIP vehicles, commercial vehicles and smart home systems, and which recognizes the face of a user, detects the emotional state of the user, provides suggestions based on the detected emotions, and has a two-dimensional or three-dimensional holographic face to establish an emotional connection between the user and the vehicle and create a real human feeling when talking, has the ability to chat, obtain information, ask questions, dictate texts, read news, provide weather information, send emails and allow voice control of devices in the vehicle, can translate different languages, can inform vehicle failures, can transmit range information, enable it to perceive commands issued by talking in daily and natural spoken language, Turkish or a desired language, and answer questions asked, and also allows shopping while in the vehicle.

[0020] The system of the present invention generally comprises: at least one camera, the at least one camera being used for visually detecting a user and his / her emotional state; at least one microphone, the at least one microphone being used for voice recognition of what the user says; at least one speaker, the at least one speaker being used for providing audio feedback to the user; at least one speech-to-text module (STT (speech to text)), the at least one speech-to-text module being used for converting what the user says into digital text and making it suitable for use by the system; at least one text-to-speech conversion module (TTS (text to speech)), the at least one text-to-speech conversion module being used for ensuring that digital text data generated by the system is transmitted to the user in voice form; at least one a noise cancellation module (noise cancellation), the at least one noise cancellation module is used to ensure the cancellation of noise in the vehicle interior or environment and a healthy perception of the user's conversation; at least one natural language processing module (NLP (neural linguistic programming)), the at least one natural language processing module is used to interpret the user's voice after it has been translated into digital text; at least one data set module, in which the data used in response to what the user said is loaded; at least one dashboard module (dashboard), the at least one dashboard module is used to ensure instant remote control of all tools and platforms where the system is located, display problems occurring in the system and solve these problems, and remotely update the system; at least one an electronic card with a processor, the at least one electronic card being used to provide awareness of the user and the user's emotional state by: performing image processing based on visual data from the camera; activating the system by detecting a wake-up word; translating user speech into digital text and transmitting it to the natural language processing module; translating text generated by the system into spoken speech and transmitting it to the user through a speaker; receiving and processing fault, range and similar information of the vehicle from the OBD brain in the vehicle via a Canbus line, websocket, Ethernet, RF or Bluetooth; and transmitting the vehicle to the vehicle via a Canbus line, websocket, Ethernet, RF or Bluetooth. sending signals to lighting devices, seats, windows, doors and similar devices in the vehicle; storing data obtained due to repeated routine behaviors of the user in a database or cloud, and using this data when the user enters the vehicle; at least one ChatGPT or similar chat service, the at least one ChatGPT or similar chat service is used to enable the system to chat with the user; at least one question and answer module, the at least one question and answer module is used to enable the system to answer questions raised by the user; at least one translation service, the at least one translation service is used to ensure that the language translation function requested by the user is realized; at least one news reading service, the at least one news reading service is used to provide news reading to the user;at least one email service for ensuring that incoming emails are read to the user and that the user can send emails with voice commands; at least one shopping service for enabling the user to shop within the vehicle and with voice commands; a two-dimensional facial display or a three-dimensional facial display with holographic hardware for enabling the user to emotionally connect with the vehicle and converse as if there were a person in front of them.;

[0021] When a user enters the vehicle, visual data of the user's face is captured by a camera and transmitted to an electronic card with a processor connected to the camera, the visual data from the camera is image processed by the electronic card with the processor and compared with photos defined in a database, and if a person is recognized, the system welcomes the person by calling the person's name through a speaker connected to the electronic card. For example, "Welcome Mr. Serdar".

[0022] The user's face is image processed by an electronic card having a processor, emotional expressions on the face, such as happiness, sadness, anger, etc., are recognized by emotion recognition software, and as a result of this recognition, relevant suggestions in the data set are transmitted to the user through the speaker connected to the electronic card. For example, "You look unhappy today, do you want me to play your favorite music?"

[0023] When the user wants to control the hardware in the vehicle, the user speaks the wake-up word and the action to be performed, the wake-up word and the command of the action to be performed are received by the microphone and transmitted to the electronic card with the processor, the spoken word is converted into text by the speech-to-text conversion module and transmitted to the natural language processing module for interpretation, and according to the operation to be performed, a signal is sent to a relevant point through a Canbus line, websocket, Ethernet, RF or Bluetooth to control the relevant hardware of the vehicle, and in addition, after the interpretation process in the natural language processing module, the voice obtained by the text-to-speech conversion module is transmitted to the user through the speaker connected to the electronic card. For example, in response to the user's request to "turn on the light", the system will say "OK, turning on the light", and the light will turn on.

[0024] Information such as faults of the vehicle, current battery or current fuel status and distance that can be travelled is transmitted from the OBD brain of the vehicle to the electronic card having the processor via a Canbus line, websocket, Ethernet, RF or Bluetooth, and the incoming information is converted into speech via the text-to-speech module and transmitted to the user with the help of the speaker connected to the electronic card.

[0025] When the user wants to chat, the wake-up word and request voice are received by the microphone, transmitted to the electronic card connected to the microphone and transcribed by the voice-to-text module, the text is passed to the natural language processing module for interpretation, and the chat service is started, the spoken content is transmitted to the chat service in text form, the response text from the chat service is passed to the text-to-speech module for conversion into voice, and the generated voice is transmitted to the user through the speaker connected to the electronic card with the processor. For example, "Hey, Dizzy, can I chat with you for a minute?", and the response is "OK, let's chat", and "What do you think about global warming?", and the system will continue the conversation with a logical answer.

[0026] When the user is curious about the answer to a question, the user asks the question after the wake-up word, which is received by the microphone and transmitted to the electronic card with the processor, and converted into text by the speech-to-text module, and the generated text is passed to the natural language processing module for interpretation, and after the interpretation, the question-answering module is activated, and the question-answering module searches for the answer to the question in various search engines or online library platforms, and the found answer is converted into speech by the text-to-speech module and transmitted to the user through the speaker connected to the electronic card with the processor. For example, the answer to the question "Which mountain is the highest in the world?" is transmitted to the user by voice.

[0027] Operations performed by the user in the vehicle or features used in the vehicle are recorded in a database inside the electronic card or in the cloud, and when the recorded data is repeated at certain intervals, the electronic card detects it, and when the user enters the vehicle, various settings are automatically made, and the user's information is used in a dialogue generated by the system.

[0028] When the user requests the system to translate a word or sentence, the user expresses this request with a wake-up word, the wake-up word is received by the microphone and transmitted to the electronic card with a processor, the request is converted into text by the speech-to-text module and passed to the natural language processing module, and an interpretation process occurs, after which a translation service for translating 22 languages ​​is activated, the text to be translated is transmitted to the translation service, and the translated text received from the translation service is converted into speech by the speech conversion module and transmitted to the user through the speaker connected to the electronic card with the processor. For example, if you say "I need you to translate into Japanese", the sentence will be expressed by the system in Japanese.

[0029] When the user wants to access current news, the user expresses this request with the wake-up word, and this request is received by the microphone and transmitted to the electronic card with the processor, the request is converted into text by the speech-to-text conversion module and passed to the natural language processing module, and an interpretation process occurs, after which the news reading service runs, the generated text is passed to the news reading service, and the news text from the news reading service is converted into speech by the text-to-speech conversion module and transmitted to the user through the speaker connected to the electronic card with the processor. For example, when the user says "Can you read the economic news from CNN Türk?", the system reads the current economic news to the user from this source.

[0030] When the user wants to send an email or read an incoming email, the user expresses this request with the wake-up word, and this request is received by the microphone and transmitted to the electronic card with the processor, the request is converted into text by the speech-to-text module and passed to the natural language processing module, and the interpretation process takes place, after which the email service is activated, the email address previously defined in the system is logged in, the voice to be sent expressed by the user is converted into text by the speech-to-text conversion module and sent to the relevant person through the email service, and in the case of an incoming email, the email content is converted into speech by the text-to-speech conversion module and read to the user through the speaker connected to the electronic card with the processor. For example, when you say "I want you to send an email to Mr. Serdar", the system asks the content of the email and forwards the generated email to the relevant person; if an email arrives, the system asks "A new email has arrived, do you want me to read it?" and when you answer "Yes, please read it", the system reads the email content.

[0031] When the user wants to shop, the user expresses this request with the wake-up word, and this request is received by the microphone and transmitted to the electronic card with the processor, the request is converted into text by the voice-to-text module and passed to the natural language processing module, and the interpretation process occurs, after which the shopping service is activated, and the shopping platform predefined in the system is accessed, the user speaks the product he wants to search for, and this voice is received by the microphone and transmitted to the electronic card with the processor, the spoken word is converted into text by the voice-to-text conversion module and transmitted to the shopping service, and the searched product is found in the search engine and added to the shopping cart, and the purchase occurs, the text of the searched product content is converted into speech by the text-to-speech conversion module and audibly transmitted to the user through the speaker connected to the electronic card, and the user is also informed that the product has been purchased. For example, when you say "I want to buy a navy blue jacket", the product you are looking for is found and purchased by adding it to the shopping cart.

[0032] The microphone continuously listens for the wake-up word. Once the microphone captures the wake-up word, the system is activated by an electronic card with a processor and the spoken word is transmitted to the other modules. Without the wake-up word, the system would have to listen to all conversations, which would violate the protection of personal data.

[0033] In the system of the present invention, the auxiliary image on the two-dimensional screen and / or the three-dimensional auxiliary image created by the holographic hardware moves the lips and mouth according to the speech executed by the system to provide the user with a conversation experience as if there is a person in front of the user. This creates an emotional connection between the user and the vehicle and makes the user feel as if he is talking to a real person.

Claims

1. An intelligent vehicle assistance system with artificial intelligence, characterized in that: The intelligent vehicle assistance system comprises: at least one camera for visually detecting a user and his / her emotional state; at least one microphone, the at least one microphone being used for voice recognition of what the user says; at least one speaker, the at least one speaker being configured to provide audio feedback to the user; at least one speech-to-text module for converting what the user says into digital text and making it suitable for use by the system; at least one text-to-speech conversion module for ensuring that digital text data generated by the system is transmitted to the user in speech form; at least one noise cancellation module for ensuring cancellation of noise in the vehicle interior or environment and healthy perception of user conversations; at least one natural language processing module for interpreting user speech after it has been translated into digital text; at least one data set module, in which data used in response to content spoken by the user is loaded; at least one dashboard module for ensuring instant remote control of all tools and platforms where the system resides, displaying problems occurring in the system and resolving them, and remotely updating the system; at least one electronic card with a processor, the at least one electronic card being used to provide awareness of the user and the user's emotional state by: performing image processing based on visual data from the camera; activating the system by detecting a wake-up word; translating user speech into digital text and transmitting it to the natural language processing module; translating text generated by the system into spoken speech and transmitting it to the user through a speaker; receiving fault, range and similar information of the vehicle from an OBD brain in the vehicle and processing it via a Canbus line, websocket, Ethernet, RF or Bluetooth; sending signals to lighting, seats, windows, doors and similar devices in the vehicle via a Canbus line, websocket, Ethernet, RF or Bluetooth; storing data obtained due to repeated routine behavior of the user in a database or cloud and using this data when the user enters the vehicle; at least one ChatGPT or similar chat service, the at least one ChatGPT or similar chat service being used to enable the system to chat with the user; at least one question-answering module, the at least one question-answering module being used to enable the system to answer questions raised by the user; at least one translation service, the at least one translation service being used to ensure that a language translation function requested by the user is implemented; at least one news reading service, the at least one news reading service being used to provide news reading to the user; at least one email service for ensuring that incoming emails are read to the user and that the user is able to send emails using voice commands; at least one shopping service for enabling the user to shop within the vehicle and using voice commands; At least one two-dimensional display and / or at least one holographic assembly for creating auxiliary images so that the user can establish an emotional connection with the vehicle and talk as if there is a person in front of him.

2. A method for operating an intelligent vehicle assistance system with artificial intelligence, characterized in that: The operation method comprises the following process steps: When a user enters a vehicle, visual data of the user's face is captured by a camera and transmitted to an electronic card with a processor connected to the camera. The visual data from the camera is image processed by the electronic card with the processor and compared with photos defined in a database. If a person is recognized, the system welcomes the person by calling the person's name through a speaker connected to the electronic card. performing image processing on the visual data of the user's face by means of an electronic card having a processor, identifying emotional expressions on the face, such as happiness, sadness, anger, etc., by means of emotion recognition software, and transmitting relevant suggestions in the data set to the user via the speaker connected to the electronic card as a result of this recognition, The microphone continuously listens for the wake-up word, and once the microphone captures the wake-up word, an electronic card with a processor activates the system and transmits the spoken word to the other modules, The auxiliary image on the two-dimensional screen and / or the three-dimensional auxiliary image created by the holographic hardware moves the lips and mouth according to the speech performed by the system to provide the user with a conversation experience as if there is a person in front of the user, When the user wants to control the hardware in the vehicle, the user speaks the wake-up word and the action to be performed, the wake-up word and the command of the action to be performed are received by the microphone and transmitted to the electronic card having the processor, the spoken word is converted into text by the speech-to-text conversion module and transmitted to the natural language processing module for interpretation, and according to the operation to be performed, a signal is sent to a relevant point through a Canbus line, websocket, Ethernet, RF or Bluetooth to control the relevant hardware of the vehicle, and further, after the interpretation process in the natural language processing module, the voice obtained by the text-to-speech conversion module is transmitted to the user through the speaker connected to the electronic card, transmitting information such as faults of the vehicle, current battery or current fuel status and distance that can be travelled from the OBD brain of the vehicle to the electronic card having the processor via a Canbus line, websocket, Ethernet, RF or Bluetooth, converting the incoming information into speech via the text-to-speech module and transmitting it to the user with the help of the speaker connected to the electronic card, When the user wants to chat, the wake-up word and request voice are received by the microphone, transmitted to the electronic card connected to the microphone and transcribed by the voice-to-text module, the text is passed to the natural language processing module for interpretation, and the chat service is started, the spoken content is transmitted to the chat service in text form, the response text from the chat service is passed to the text-to-speech module to be converted into voice, and the generated voice is transmitted to the user through the speaker connected to the electronic card having the processor, When the user is curious about the answer to a question, the user asks the question after the wake-up word, the question is received by the microphone and transmitted to the electronic card with the processor, and converted into text by the speech-to-text module, the generated text is passed to the natural language processing module for interpretation, and after the interpretation, the question-answering module is activated, the question-answering module searches for the answer to the question in various search engines or online library platforms, the found answer is converted into speech by the text-to-speech module and transmitted to the user through the speaker connected to the electronic card with the processor, Operations performed by the user in the vehicle or features used in the vehicle are recorded in a database or cloud inside the electronic card, and when the recorded data is repeated at certain intervals, the electronic card detects it, and when the user enters the vehicle, various settings are automatically made, and the user's information is used in a dialog generated by the system, When the user requests the system to translate a word or sentence, the user expresses this request with a wake-up word, the wake-up word is received by the microphone and transmitted to the electronic card with a processor, the request is converted into text by the speech-to-text module and passed to the natural language processing module, and an interpretation process occurs, after which a translation service for translating 22 languages ​​is activated, the text to be translated is transmitted to the translation service, and the translated text received from the translation service is converted into speech by the speech conversion module and transmitted to the user through the speaker connected to the electronic card with the processor, When the user wants to access current news, the user expresses this request with the wake-up word, and this request is received by the microphone and transmitted to the electronic card with the processor, the request is converted into text by the speech-to-text conversion module and passed to the natural language processing module, and an interpretation process occurs, after which the news reading service runs, the generated text is passed to the news reading service, and the news text from the news reading service is converted into speech by the text-to-speech conversion module and transmitted to the user through the speaker connected to the electronic card with the processor, When the user wants to send an email or read an incoming email, the user expresses this request with the wake-up word, and this request is received by the microphone and transmitted to the electronic card with the processor, the request is converted into text by the speech-to-text module and passed to the natural language processing module, and an interpretation process takes place, after which the email service is activated, an email address previously defined in the system is logged in, the voice to be sent expressed by the user is converted into text by the speech-to-text conversion module and sent to the relevant person through the email service, in the case of an incoming email, the email content is converted into speech by the text-to-speech conversion module and read to the user through the speaker connected to the electronic card with the processor, When the user wants to shop, the user expresses this request with the wake-up word, and this request is received by the microphone and transmitted to the electronic card with the processor, the request is converted into text by the voice-to-text module and passed to the natural language processing module, and an interpretation process occurs, after which a shopping service is activated, and a shopping platform predefined in the system is accessed, the user speaks the product he wants to search for, and this voice is received by the microphone and transmitted to the electronic card with the processor, the spoken words are converted into text by the voice-to-text conversion module and transmitted to the shopping service, and the searched product is found in the search engine and added to the shopping cart, and a purchase occurs, the text of the searched product content is converted into voice by the text-to-speech conversion module and audibly transmitted to the user through the speaker connected to the electronic card, and the user is also informed that the product has been purchased.