Vehicle interaction method, server and computer readable storage medium
By acquiring and analyzing image data in the car cockpit system, determining user characteristics and generating autonomous voice broadcasts, the problem that existing systems cannot provide personalized services is solved, and a higher user experience and emotional connection is achieved.
Patent Information
- Application Number
- CN202510223304.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-26
- Publication Date
- 2025-06-10
AI Technical Summary
The existing car cockpit system cannot think independently based on the user's personalized characteristics, and the services provided are single and lack emotional links, resulting in poor user experience.
By obtaining the current image data in the vehicle cockpit, determining the current feature information of the target occupant, and generating autonomous voice broadcast information based on historical interaction data to achieve personalized services and emotional connections.
It enhances the emotional connection of users, improves the user experience, and provides a more personalized and interactive way of interaction.
Smart Images

Figure CN120116867A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vehicle interaction, and particularly relates to a vehicle interaction method, a server, and a computer-readable storage medium. Background Art
[0002] In the related art, the automotive cockpit system mainly relies on speech recognition and command response, and cannot think independently according to the personalized characteristics of users. Thus, the services provided by the automotive cockpit system are single and lack emotional connection, resulting in poor user experience. Summary of the Invention
[0003] The present application provides a vehicle interaction method, a server, and a computer-readable storage medium.
[0004] An embodiment of the present application provides a vehicle interaction method, the method comprising:
[0005] Obtaining current image data inside a vehicle cockpit;
[0006] Determining current characteristic information of a target occupant inside the vehicle cockpit according to the current image data;
[0007] Generating an autonomous voice broadcast message according to the current characteristic information and historical interaction data, so that the target occupant completes interaction with the vehicle according to the autonomous voice broadcast message.
[0008] In this way, the server obtains the current image data inside the vehicle cockpit. Then, the server determines the current characteristic information of the target occupant inside the vehicle cockpit according to the current image data. Finally, the server generates an autonomous voice broadcast message according to the current characteristic information and historical interaction data, so that the target occupant completes interaction with the vehicle according to the autonomous voice broadcast message. In this way, by analyzing the current image data inside the vehicle cockpit obtained through real-time monitoring, the current characteristic information of the user is determined, and the current characteristic information and historical interaction data are fused to actively understand the user's current emotions and behaviors and provide personalized services, enhancing the emotional connection with the user, thereby improving the user experience.
[0009] In some embodiments, the current image data includes image data of the target occupant and / or image data of the cockpit environment.
[0010] In this way, the current image data obtained by the server includes image data of the target occupant and / or image data of the cockpit environment. In this way, by analyzing the image data of the target occupant and the cockpit environment, an in-depth understanding of the user's emotions and behaviors is achieved, and the interaction method and the in-vehicle environment are actively adjusted, thereby improving the vehicle interaction experience.
[0011] In some embodiments, the historical interaction data includes historical voice interaction data, historical behavior interaction data, historical image data, and user preference data, wherein the user preference data is inferred based on the historical voice interaction data and historical image data.
[0012] Thus, the above historical interaction data includes historical voice interaction data, historical behavior interaction data, historical image data, and user preference data, wherein the user preference data is inferred based on the historical voice interaction data and historical image data. In this way, by analyzing historical interaction data from multiple aspects, the user can be deeply understood, the user preferences can be accurately understood, and the user needs can be predicted autonomously based on the user preferences, so as to provide personalized services.
[0013] In some embodiments, determining the current feature information of the target occupant in the vehicle cockpit according to the current image data includes:
[0014] Preprocessing the current image data to generate standardized image data;
[0015] Based on a first preset model, determining the current feature information according to the standardized image data, where the current feature information includes gender information, age information, clothing information, and / or hairstyle information.
[0016] Thus, the server preprocesses the current image data to generate standardized image data. Then, based on the first preset model, the server determines the current feature information according to the standardized image data, where the current feature information includes gender information, age information, clothing information, and / or hairstyle information. In this way, by preprocessing and analyzing the current image data, the current feature information of the user is determined, and subsequent personalized autonomous voice broadcast information can be generated according to this information, so as to achieve a natural interaction experience.
[0017] In some embodiments, preprocessing the current image data to generate standardized image data includes:
[0018] Based on a preset image processing algorithm, processing the resolution and aspect ratio of the current image data to generate the standardized image data.
[0019] Thus, based on the preset image processing algorithm, the server processes the resolution and aspect ratio of the current image data to generate standardized image data. In this way, the generated standardized image data can ensure the stable operation of the model on different devices and in different scenarios. And since the model does not need to adjust parameters for each size and ratio of the image, the standardized image data can reduce the computational resource consumption of the model when processing images of different sizes and ratios.
[0020] In some implementations, generating autonomous voice broadcast information according to the current feature information and historical interaction data includes:
[0021] Based on the second preset model, determining the target topic, the adapted emotion and the topic content according to the current feature information and the historical interaction data;
[0022] The autonomous voice broadcast information is determined according to the target topic, the adapted emotion and the topic content.
[0023] Thus, based on the second preset model, the server determines the target topic, the adaptive emotion, and the topic content according to the current feature information and the historical interaction data. Then, the server determines the autonomous voice broadcast information according to the target topic, the adaptive emotion, and the topic content. In this way, according to the user's current feature information and the historical interaction data, the autonomous voice broadcast information that meets the user's interests and adapts to the user's current emotions is determined, thereby improving the user's participation, enhancing the interactivity and fun of the interaction, and thus improving the user experience.
[0024] In some implementations, determining a target topic, an adapted emotion, and a topic content according to the current feature information and the historical interaction data includes:
[0025] Based on the retrieval enhancement generation technology, according to the target topic, obtaining target information associated with the target topic;
[0026] The topic content is determined according to the target information.
[0027] Thus, based on the retrieval enhancement generation technology, the server obtains target information associated with the target topic according to the target topic. Then, the server determines the topic content according to the target information. In this way, by retrieving the target topic that matches the user's interests and adapts to the user's current emotions, the target information of interest to the user is obtained, and natural and relevant conversation content is generated, thereby improving the user experience.
[0028] In certain embodiments, the method further comprises:
[0029] Obtain user satisfaction rating information for historical autonomous voice broadcast information;
[0030] Determining language style information for interacting with the target occupant according to the satisfaction score information;
[0031] The autonomous voice broadcast information is generated according to the language style information, the current feature information and the historical interaction data.
[0032] In this way, the server obtains the satisfaction rating information of the user for the historical autonomous voice broadcast information. Then, the server determines the language style information for interacting with the target occupant according to the satisfaction rating information. Finally, the server generates the autonomous voice broadcast information according to the language style information, the current feature information, and the historical interaction data. In this way, by adjusting the language style according to the user's satisfaction rating, a more user-preferred interaction experience can be provided, thereby improving user satisfaction. Moreover, by continuously collecting user satisfaction ratings and continuously optimizing the voice broadcast information, the interaction becomes more natural, personalized, and closer to the user's needs, thereby enhancing the user experience.
[0033] An embodiment of the present application provides a server, which includes a processor and a memory. A computer program is stored on the memory. When the computer program is executed by the processor, the above-mentioned vehicle interaction method is implemented.
[0034] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the vehicle interaction method as described above are implemented.
[0035] Additional aspects and advantages of the embodiments of the present application will be given in part in the following description, will become apparent in part from the following description, or will be understood through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where:
[0037] Figure 1 is one of the flow diagrams of the vehicle interaction method of some embodiments of the present application;
[0038] Figure 2 is another flow diagram of the vehicle interaction method of some embodiments of the present application;
[0039] Figure 3 is yet another flow diagram of the vehicle interaction method of some embodiments of the present application;
[0040] Figure 4 is still another flow diagram of the vehicle interaction method of some embodiments of the present application;
[0041] Figure 5 is yet still another flow diagram of the vehicle interaction method of some embodiments of the present application;
[0042] Figure 6 is another flow diagram of the vehicle interaction method of some embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0043] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and should not be construed as a limitation to the embodiments of the present application.
[0044] In the current vehicle cockpit system, the interaction technology mainly relies on speech recognition and command response. In this way, to a certain extent, driving safety and convenience are improved, but its limitations are also very obvious. Since it is unable to deeply analyze and think autonomously about the personalized characteristics of users such as gender, age, clothing, and hairstyle, the services provided by the vehicle cockpit system often appear single and lack an emotional connection, resulting in a poor user experience.
[0045] First of all, the interaction method of the vehicle cockpit system relies too much on speech recognition and command response. This means that the interaction function of the system mainly focuses on executing the direct instructions of the user, and is unable to actively provide information and services based on the personalized characteristics of the user. This passive interaction method limits the flexibility and initiative of the system, making the user feel a simple conversation with the machine during use, rather than an in-depth communication with an intelligent assistant.
[0046] Secondly, the vehicle cockpit system cannot think autonomously based on the personalized characteristics of the user, and the services it provides often lack pertinence and personalization. For example, the system cannot adjust the interaction content and style according to the characteristics of the user such as gender, age, clothing, and hairstyle, nor can it predict and meet the needs of the user based on the user's historical interaction data. This lack of personalized service not only reduces the user's satisfaction, but also limits the system's ability to provide emotional value.
[0047] Based on the above problems, please refer to Figure 1 , the embodiments of the present application provide a vehicle interaction method, the method including:
[0048] 01: Obtain the current image data inside the vehicle cockpit;
[0049] 02: Determine the current characteristic information of the target occupant inside the vehicle cockpit according to the current image data;
[0050] 03: Generate an autonomous voice broadcast message according to the current characteristic information and historical interaction data, so that the target occupant completes the interaction with the vehicle according to the autonomous voice broadcast message.
[0051] The embodiment of the present application also provides a server, including a memory and a processor. The vehicle interaction method of the embodiment of the present application can be implemented by the server of the embodiment of the present application. Specifically, a computer program is stored in the memory, and the processor is configured to obtain the current image data in the vehicle cockpit, determine the current feature information of the target occupant in the vehicle cockpit according to the current image data, and generate an autonomous voice broadcast message according to the current feature information and the historical interaction data, so that the target occupant can complete the interaction with the vehicle according to the autonomous voice broadcast message.
[0052] The embodiment of the present application also provides a vehicle interaction device. The vehicle interaction method of the embodiment of the present application can be implemented by the vehicle interaction device of the embodiment of the present application. Specifically, the vehicle interaction device includes an acquisition module, a determination module, and a generation module. The acquisition module is configured to obtain the current image data in the vehicle cockpit. The determination module is configured to determine the current feature information of the target occupant in the vehicle cockpit according to the current image data. The generation module is configured to generate an autonomous voice broadcast message according to the current feature information and the historical interaction data, so that the target occupant can complete the interaction with the vehicle according to the autonomous voice broadcast message.
[0053] Specifically, the vehicle interaction method provided by the embodiments of the present application is used to actively generate autonomous voice broadcast information according to the current characteristic information and historical interaction data of the target occupant when there is only one driver in the vehicle cockpit or all passengers in the cockpit are not busy with other things and not engaged in conversation, so as to mobilize the emotions of the vehicle occupants and relieve the dull atmosphere in the vehicle. For example, the system processes the real-time image information of all occupants in the vehicle obtained through the Occupancy Monitoring System (OMS), and identifies that there are three occupants in the vehicle, namely a little girl, a middle-aged man, and a middle-aged woman. At this time, the people in the car are not talking, and except for the middle-aged man driving, each of them has no ongoing business. At this time, the cockpit system will determine the respective characteristic information of the three occupants according to the obtained real-time image information. The characteristic information of the little girl is "Age: about 6 - 8 years old; Gender: female; Clothing: wearing casual clothes with cartoon patterns; Hair style: long hair, braided". The characteristic information of the middle-aged man is "Age: about 40 - 50 years old; Gender: male; Clothing: wearing business casual clothes, possibly wearing a tie; Hair style: short hair, neat". The gender characteristics of the middle-aged woman are "Age: about 40 - 50 years old; Gender: female; Clothing: wearing an elegant dress; Hair style: medium-length hair, possibly permed". Based on the above characteristic information and historical interaction data, the cockpit system determines that the three occupants are a family, and at this time they are triggering a trip to XX Zoo together. In this way, the generated autonomous voice broadcast information may be "Good morning, everyone! It seems that you are all ready to start a pleasant journey today. The sun is shining brightly and the temperature is suitable today. It's really a great day for an outing. I know that there are restaurants with very high ratings near XX Zoo. Do you need me to recommend some for you?" The generated autonomous voice broadcast information may also be "Good morning, everyone! The cartoon pattern on this little friend's clothes is so cute! It must have been carefully selected by the elegant mom. It's very suitable for going out to play. What kind of cartoon character is this? Little friend, can you give me a science popularization?"
[0054] The Occupancy Monitoring System (OMS) can monitor the age, status, and emotions of passengers, etc.
[0055] It should be noted that the vehicle interaction method provided by the embodiments of the present application may only be a functional module in the vehicle cockpit system, and this function needs to be enabled by the user by clicking a button or through other opening methods.
[0056] The target occupant refers to the driver or passenger determined through image recognition or other sensor technologies, and the vehicle cockpit system needs to perform detailed feature extraction and analysis processing on them. In some embodiments, the vehicle cockpit system acquires the image information of all occupants in the vehicle cockpit, and through simple recognition and analysis, determines to perform detailed feature extraction and analysis on the image information of a certain occupant to confirm the current feature information. In some embodiments, the vehicle cockpit system performs detailed feature extraction and analysis on all occupants in the acquired image information, then determines the target occupant, and performs subsequent processing based on the current feature information of the target occupant.
[0057] The server acquires the current image data in the cockpit in real time.
[0058] Next, the acquired image data is processed and analyzed to identify and determine the current feature information of the target occupant in the vehicle cockpit.
[0059] Finally, the historical interaction data of the target occupant is acquired, and based on the current feature information and historical interaction data of the target occupant, an autonomous voice broadcast message is automatically generated so that the target occupant can complete the interaction with the vehicle according to the autonomous voice broadcast message. In some embodiments, generating the autonomous voice broadcast message will also combine the internal state of the vehicle and external environment information.
[0060] For example, when the user enters the vehicle after work, the user's mood is slightly tired. The vehicle cockpit system acquires the user's image through the OMS system or other image sensors in the vehicle cockpit, recognizes that the user is in a low mood, and combines the user's historical interaction data to learn that the user likes to listen to light music. The vehicle cockpit system will autonomously generate an autonomous voice broadcast message: "Dear user, you've had a hard day at work. Now, some relaxing music will be played for you to unwind."
[0061] In summary, in the vehicle interaction method and server provided by the embodiments of the present application, the server acquires the current image data in the vehicle cockpit. Then, the server determines the current feature information of the target occupant in the vehicle cockpit based on the current image data. Finally, the server generates an autonomous voice broadcast message based on the current feature information and historical interaction data so that the target occupant can complete the interaction with the vehicle according to the autonomous voice broadcast message. In this way, by analyzing the current image data in the vehicle cockpit obtained through real-time monitoring, the current feature information of the user is determined, and the current feature information and historical interaction data are integrated to actively understand the user's current mood and behavior and provide personalized services, enhancing the user's emotional connection and thus improving the user experience.
[0062] In some embodiments, the current image data includes the image data of the target occupant and / or the image data of the cockpit environment.
[0063] Specifically, the current image data refers to real-time image information obtained from a camera or other image capture device inside the vehicle cockpit, including the image data of the target occupant and the image data of the cockpit environment. It should be noted that due to the limited space inside the vehicle cockpit and the large number of vehicle devices, the occupants or objects inside the cockpit in the acquired image data may block each other, and certain technical means (such as an occlusion management system, etc.) are required for processing to better identify and analyze the occupant characteristics.
[0064] In this way, the current image data obtained by the server includes the image data of the target occupant and / or the image data of the cockpit environment. In this way, by analyzing the image data of the target occupant and the cockpit environment, an in-depth understanding of the user's emotions and behaviors is achieved, and the interaction method and the in-vehicle environment are actively adjusted, thereby improving the vehicle interaction experience.
[0065] In some embodiments, the historical interaction data includes historical voice interaction data, historical behavior interaction data, historical image data, and user preference data, where the user preference data is inferred based on the historical voice interaction data and the historical image data.
[0066] Specifically, the historical interaction data refers to the data collected and stored by the vehicle cockpit system during past interactions, including historical voice interaction data, historical behavior interaction data, historical image data, and user preference data inferred from these data. These data help the system better understand the user's needs and preferences, thereby providing more personalized and emotional services.
[0067] The historical voice interaction data refers to all voice communications recorded by the system between the user and the cockpit interaction system, including the voice requests made by the user and the voice responses given by the system. It is recorded and transcribed through the voice recognition module in the intelligent cockpit system and stored in the corresponding database.
[0068] The historical behavior interaction data refers to the system's monitoring of the user's behaviors inside the cockpit through built-in sensors, cameras, or other devices, such as button operations, touch screen interactions, etc. It is recorded through the sensors and software in the intelligent cockpit system and stored in the corresponding database.
[0069] The historical image data refers to the user image records obtained through the OMS system or other cameras, including the user's images and environmental images. It is captured through the OMS system or other cameras and stored in the corresponding database.
[0070] User preference data refers to the preferences of users regarding system functions and content, such as the types of music they like, the categories of news they prefer, volume settings, and seat position settings. It is obtained by analyzing the user's historical interaction data, such as the functions they often click on, the content they view, the interaction methods they use, etc., and reasoning in combination with the user profile.
[0071] Thus, the historical interaction data includes historical voice interaction data, historical behavior interaction data, historical image data, and user preference data. Among them, the user preference data is inferred based on the historical voice interaction data and historical image data. In this way, by analyzing historical interaction data from multiple aspects, it is possible to deeply understand the user, accurately understand the user's preferences, and autonomously predict the user's needs based on the user's preferences, thereby providing personalized services.
[0072] Please refer to Figure 2 , in some embodiments, step 02 (determining the current characteristic information of the target occupant in the vehicle cockpit according to the current image data) includes:
[0073] 021: Preprocess the current image data to generate standardized image data;
[0074] 022: Based on the first preset model, determine the current characteristic information according to the standardized image data.
[0075] In some embodiments, the generation module is further configured to preprocess the current image data to generate standardized image data. The determination module is further configured to determine the current characteristic information based on the first preset model according to the standardized image data.
[0076] In some embodiments, the processor is further configured to preprocess the current image data to generate standardized image data. And based on the first preset model, determine the current characteristic information according to the standardized image data.
[0077] Specifically, the first preset model refers to a model based on machine learning or deep learning technology that can determine the current characteristic information according to the input standardized image data, such as the fine-tuned qwen2-VL-7B open-source image-text large model. This open-source image-text large model itself has general image recognition capabilities and dialogue capabilities, so no pre-training is required. However, to ensure the accuracy in specific scenarios and conform to a specific output format, it is necessary to collect specialized data for annotation and perform instruction fine-tuning on the model. The forms of training data include pictures of people with characteristics such as different genders, ages, clothing, and hairstyles, and corresponding conversation scripts are designed. The forms of training data may be as follows:
[0078]
[0079] It should be noted that in some embodiments, other deep learning frameworks or algorithms can also be used to analyze user features. For example, a convolutional neural network (CNN) or a generative adversarial network (GAN) can be used.
[0080] First, preprocess the current image data obtained from the OMS system to generate standardized image data to ensure the consistency of image quality and the accuracy of subsequent analysis. For example, adjust the brightness and contrast of the image, crop out unnecessary backgrounds, and ensure that the resolution and aspect ratio of the image meet the requirements of the analysis model.
[0081] Next, a first preset model is determined in advance, and the preprocessed image data is input into the first preset model. After receiving the standardized image data, the first preset model performs analysis to determine the feature information in the standardized image, including gender information, age information, clothing information, and hairstyle information, etc. For example, the first preset model identifies the gender of the target occupant by analyzing facial features, skin color, hairstyle, etc. The first preset model estimates the age of the target occupant based on features such as facial wrinkles and skin laxity. The first preset model identifies the clothing style, color, and pattern of the target occupant to infer their clothing information. The first preset model analyzes the length, color, and style of the hair to identify the hairstyle of the target occupant.
[0082] The following takes the Qwen-VL multi-modal large model as an example to illustrate the feature extraction and processing of image data in the vehicle interaction method provided by the embodiments of the present application. In terms of extracting image features, the Qwen-VL multi-modal large model adopts a comprehensive method including a large-scale language model, a visual encoder, and a position-aware vision-language adapter. First, the Qwen-VL multi-modal large model uses a Vision Transformer-based visual encoder, especially the ViT-bigG version of OpenCLIP, to segment the input image into fixed-size image patches and convert them into a sequence of image features. Then, the Qwen-VL multi-modal large model uses a position-aware vision-language adapter to compress these long sequences of image features into a fixed-length representation using the cross-attention mechanism, while retaining the two-dimensional absolute position encoding to ensure that the position information in the image is retained. In addition, the Qwen-VL multi-modal large model also introduces multi-modal rotary position embedding (M-RoPE), which effectively fuses the position information in text, images, and videos by decomposing the position encoding into three components: time, height, and width. And, after being trained, the Qwen-VL multi-modal large model can automatically focus attention on key features and then complete the analysis.
[0083] In this way, by preprocessing and analyzing the current image data, the current feature information of the user is determined. Subsequently, personalized autonomous voice broadcast information can be generated based on this information, thereby achieving a natural interaction experience.
[0084] Please refer to Figure 3 , in some embodiments, step 021 (preprocessing the current image data to generate standardized image data) includes:
[0085] 0211: Based on a preset image processing algorithm, process the resolution and aspect ratio of the current image data to generate standardized image data.
[0086] In some embodiments, the generation module is further configured to process the resolution and aspect ratio of the current image data based on a preset image processing algorithm to generate standardized image data.
[0087] In some embodiments, the processor is further configured to process the resolution and aspect ratio of the current image data based on a preset image processing algorithm to generate standardized image data.
[0088] Specifically, the preset image processing algorithm refers to a series of algorithms used to process and analyze image data in a vehicle cockpit system, including an object detection algorithm (Object Detection Algorithm, ODA) that can identify people in the image and crop them out, an image enhancement algorithm (Image Enhancement Algorithm, IEA) to improve image quality, and a normalization algorithm (Normalization Algorithm, NA) to scale the image pixel values to a specific range, etc. These algorithms are pre-designed and integrated into the system to effectively process the images captured by the camera inside the vehicle cockpit.
[0089] Based on a preset image processing algorithm, preprocess the acquired image data, including cropping, enhancement, and normalization processing of the image, for subsequent feature analysis. Cropping processing means cropping the image to a specified size and area, for example, only retaining the face area. Enhancement processing means enhancing the image, for example, adjusting brightness, contrast, sharpness, etc., to improve image quality. Normalization processing means normalizing the image data to a unified format, for example, scaling the pixel values to the [0, 1] interval.
[0090] In this way, the generated standardized image data can ensure the stable operation of the model on different devices and in different scenarios. Moreover, since the model does not need to adjust parameters for images of each size and ratio, the standardized image data can reduce the computational resource consumption of the model when processing images of different sizes and ratios.
[0091] Please refer toFigure 4 , in some embodiments, step 03 (generating the autonomous voice broadcast information according to the current feature information and historical interaction data) includes:
[0092] 031: Based on the second preset model, determine the target topic, the adapted emotion, and the topic content according to the current feature information and historical interaction data;
[0093] 032: Determine the autonomous voice broadcast information according to the target topic, the adapted emotion, and the topic content.
[0094] In some embodiments, the determination module is further configured to determine the target topic, the adapted emotion, and the topic content based on the second preset model according to the current feature information and historical interaction data, and determine the autonomous voice broadcast information according to the target topic, the adapted emotion, and the topic content.
[0095] In some embodiments, the processor is further configured to determine the target topic, the adapted emotion, and the topic content based on the second preset model according to the current feature information and historical interaction data, and determine the autonomous voice broadcast information according to the target topic, the adapted emotion, and the topic content.
[0096] Specifically, the second preset model refers to an algorithm model for generating the autonomous voice broadcast information. This model determines the specific content of the autonomous voice broadcast according to the current feature information of the cockpit occupants, historical interaction data, as well as the target topic, the adapted emotion, and the topic content.
[0097] Using the second preset model, determine one or more target topics according to the current feature information of the target occupant extracted from the current image data and historical interaction data. At the same time, adapt the corresponding emotion according to the user's current feature information and context, and generate content related to the topic.
[0098] Based on the determined target topic, the adapted emotion, and the topic content, the system generates an attractive and highly relevant autonomous voice broadcast information to stimulate user participation and response.
[0099] For example, the user is a young male office worker. Today is the weekend. He is wearing exquisite clothes and has a hairstyle (such as hair wax). The user's historical voice requests include "How is the weather today?" and "A suitable place for a date for two people?" The system replies are "Sunny, moderate temperature" and "Location A" respectively. The system analyzes the user's interests as "weather" and "dating", and determines the target topic as "Is the date location suitable" based on user characteristics and historical interaction data. The system uses emotion recognition technology to analyze the user's current emotion as "happy". Based on the target topic, adapted emotions and topic content, combined with speech synthesis technology, an autonomous voice broadcast message is generated: "Location A is now sunny and the temperature is moderate. You are dressed very handsomely today and it is suitable for travel. Please pay attention to sun protection. I wish you a happy day today!"
[0100] The system recognizes from the image data that the driver is a young male in business attire, and the historical interaction data shows that he has a strong interest in financial news. Based on this information, the system determines the target topic as "latest financial news". Through the second preset model, the system adapts to the serious and solemn emotions and generates the following autonomous voice broadcast information: "Good morning, sir. I noticed that you are wearing business attire today and look ready to start a busy day. The stock market has performed well recently, and you may be interested in the latest financial news. This morning's financial news mentioned some important market trends and investment opportunities. Do you need me to report it to you in detail?"
[0101] In this way, based on the user's current feature information and historical interaction data, autonomous voice broadcast information that meets the user's interests and adapts to the user's current emotions is determined, thereby increasing user participation, enhancing the interactivity and fun of the interaction, and thus improving the user experience.
[0102] See also Figure 5 In some embodiments, step 031 (determining the target topic, the adapted emotion and the topic content according to the current feature information and the historical interaction data) includes:
[0103] 0311: Based on the retrieval enhancement generation technology, obtain the target information associated with the target topic according to the target topic;
[0104] 0312: Determine the topic content based on the target information.
[0105] In some embodiments, the acquisition module is further used to acquire target information associated with the target topic based on the retrieval enhancement generation technology. The determination module is further used to determine the topic content based on the target information.
[0106] In some embodiments, the processor is further configured to obtain target information associated with the target topic based on retrieval-augmented generation technology, and determine the topic content according to the target information.
[0107] Specifically, Retrieval-Augmented Generation (RAG) is a natural language processing technology that combines retrieval and generation strategies, and can assist the generation process by retrieving relevant information to improve the accuracy and relevance of the generated content.
[0108] First, based on retrieval-augmented generation technology, relevant target information is obtained from a knowledge base or the Internet according to the target topic. For example, when the target topic is determined to be "driving safety", the obtained target information may be "traffic rules and regulations, safe driving skills, road traffic signs and markings, common traffic violations and penalties, driving safety precautions, and driver mental qualities, etc.". When the target topic is determined to be "dating location", the obtained target information may be "location type, surrounding environment of the location, traffic convenience of the location, and safety of the location, etc.". When the target topic is "healthy diet", the obtained target information may be "the latest healthy diet suggestions or relevant news".
[0109] Next, based on the obtained target information, the system determines specific conversation content that is attractive and highly relevant to stimulate user participation and response.
[0110] For example, the user is a young woman, and the last conversation content is asking "where is there delicious food nearby". The system analyzes that the user's interest point is "delicious food". According to the user characteristics and historical interaction data, the target topic is determined to be "food recommendation". The system uses retrieval generation technology to obtain target information associated with "food recommendation" from the Internet, such as nearby restaurants, cuisines, and special dishes. According to the target information and the user's mood, the topic content is determined to be "There is a newly opened Sichuan restaurant nearby. The dishes have authentic flavors and affordable prices. You can go and try it.".
[0111] In this way, by retrieving the target topic that meets the user's interests and adapts to the user's current mood, obtaining the target information that the user is interested in, and generating natural and relevant conversation content, the user experience is improved.
[0112] Please refer to Figure 6 , in some embodiments, the method further includes:
[0113] 04: Obtaining satisfaction rating information of the user for the historical autonomous voice broadcast information;
[0114] 05: Determining language style information for interacting with the target occupant according to the satisfaction rating information;
[0115] 06: Generate autonomous voice broadcast information based on the language style information, current feature information, and historical interaction data.
[0116] In some embodiments, the acquisition module is further configured to acquire the satisfaction rating information of the user for the historical autonomous voice broadcast information. The determination module is further configured to determine the language style information for interacting with the target occupant according to the satisfaction rating information. The generation module is further configured to generate autonomous voice broadcast information based on the language style information, current feature information, and historical interaction data.
[0117] In some embodiments, the processor is further configured to acquire the satisfaction rating information of the user for the historical autonomous voice broadcast information. And determine the language style information for interacting with the target occupant according to the satisfaction rating information. And generate autonomous voice broadcast information based on the language style information, current feature information, and historical interaction data.
[0118] Specifically, the satisfaction rating information refers to the evaluation and feedback of the user on the historical autonomous voice broadcast information generated by the vehicle cockpit system, usually presented in the form of a score, representing the user's satisfaction with the autonomous voice broadcast information.
[0119] The language style information refers to the language expression mode of the autonomous voice broadcast information generated by the vehicle cockpit system, including serious style, humorous style, lively style, lyrical style, etc.
[0120] Acquire the satisfaction rating of the user for the historical autonomous voice broadcast information, for example, collect the user rating through voice feedback, click operation, etc.
[0121] Next, analyze the language style preferred by the user according to the user satisfaction rating, such as formal, humorous, kind, etc.
[0122] Finally, generate autonomous voice broadcast information that meets the user's preferences based on the language style preferred by the user, current feature information, and historical interaction data.
[0123] For example, the user is a young male, and the content of the last conversation is asking "What's the news today?". The system analyzes that the user's interest point is "news", and determines the target topic as "news broadcast" according to the user characteristics and historical interaction data. The system uses emotion recognition technology to analyze that the user's current emotion is "curious". The system analyzes that the language style preferred by the user is "humorous style" according to the user satisfaction rating. Based on the target topic, adapted emotion, language style preferred by the user, and current feature information, combined with voice synthesis technology, generate the autonomous voice broadcast information: "Hey, do you know? There's a big thing happening today, XXX announced his resignation! This is really a wonder of the world!"
[0124] It should be noted that after certain processing, the vehicle interaction method provided by the embodiments of this application can be applied to other scenarios, such as smart home, robot interaction, etc., to provide a wider range of personalized services.
[0125] In this way, by adjusting the language style according to the user's satisfaction score, an interaction experience that better conforms to the user's preferences can be provided, thereby improving user satisfaction. Moreover, by continuously collecting user satisfaction scores and continuously optimizing the voice broadcast information, the interaction becomes more natural, personalized and closer to the user's needs, thus enhancing the user experience.
[0126] This application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the vehicle interaction method as described above are implemented.
[0127] It can be understood that the computer program includes computer program code. The computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), and software distribution media, etc.
[0128] In the description of this specification, the descriptions referring to terms such as "specifically", "further", "specially", "understandably", etc. mean that the specific features, structures, materials or characteristics described in combination with the embodiments or examples are included in at least one embodiment or example of this application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.
[0129] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment or part of executable code including one or more steps for implementing a specific logical function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art of the embodiments of this application.
[0130] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.
Claims
1. A vehicle interaction method, characterized in that: The method comprises: Acquire current image data in the vehicle cabin; Determining current feature information of the target occupant in the vehicle cabin according to the current image data; Autonomous voice broadcast information is generated according to the current feature information and historical interaction data, so that the target occupant completes the interaction with the vehicle according to the autonomous voice broadcast information.
2. The method according to claim 1, characterized in that The current image data includes image data of the target occupant and / or image data of a cabin environment.
3. The method according to claim 1, characterized in that The historical interaction data includes historical voice interaction data, historical behavior interaction data, historical image data and user preference data, wherein the user preference data is inferred based on the historical voice interaction data and historical image data.
4. The method according to claim 1, characterized in that: The determining, based on the current image data, current feature information of the target occupant in the vehicle cabin includes: Preprocessing the current image data to generate standardized image data; Based on the first preset model, the current feature information is determined according to the standardized image data, and the current feature information includes gender information, age information, clothing information and / or hairstyle information.
5. The method according to claim 4, characterized in that The preprocessing of the current image data to generate standardized image data includes: Based on a preset image processing algorithm, the resolution and aspect ratio of the current image data are processed to generate the standardized image data.
6. The method according to claim 1, characterized in that The generating of autonomous voice broadcast information according to the current feature information and the historical interaction data includes: Based on the second preset model, determining the target topic, the adapted emotion and the topic content according to the current feature information and the historical interaction data; The autonomous voice broadcast information is determined according to the target topic, the adapted emotion and the topic content.
7. The method according to claim 6, characterized in that The determining of the target topic, the adapted emotion and the topic content according to the current feature information and the historical interaction data includes: Based on the retrieval enhancement generation technology, according to the target topic, obtaining target information associated with the target topic; The topic content is determined according to the target information.
8. The method according to claim 1, characterized in that The method further comprises: Obtain user satisfaction rating information for historical autonomous voice broadcast information; Determining language style information for interacting with the target occupant according to the satisfaction score information; The autonomous voice broadcast information is generated according to the language style information, the current feature information and the historical interaction data.
9. A server, characterized in that: The server includes a processor and a memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
