Intelligent network connection automobile digital virtual human dialogue method and device, vehicle and medium

Abstract

The invention relates to the technical field of automobile digital virtual persons, and discloses an intelligent network connection automobile digital virtual person dialogue method and device, a vehicle and a medium, and the method comprises the steps: generating a corresponding digital virtual person in response to a person setting operation of a user; voice data of a user and state data of a vehicle are collected; recognizing the voice data, and determining a user intention; identifying a current scene of the vehicle and a dialogue service corresponding to the current scene according to the user intention and the state data; and determining a control instruction based on the digital virtual person, the current scene and the dialogue service, and controlling the digital virtual person to execute a corresponding action based on the control instruction so as to complete the dialogue with the user. According to the invention, in consideration of the individual demand of the user, the corresponding digital virtual human is generated through the figure setting operation, the current scene and the corresponding dialogue service are determined based on the voice data of the user and the state data of the vehicle, and the control instruction is acquired to control the digital virtual human to execute the corresponding action. The service experience of the user on voice interaction can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital virtual humans in automobiles, and in particular to a method, device, vehicle and medium for communicating with digital virtual humans in intelligent networked automobiles. Background Art

[0002] At present, in the process of digitalization and intelligent upgrading and transformation of intelligent connected vehicles, voice products, such as the anthropomorphic design of in-car assistants, are gradually showing their importance, and the anthropomorphic voice intelligent interactive experience will become a place for differentiated competition among various car companies. With the rapid development of computer graphics, graphics rendering, motion capture, deep learning, speech synthesis and other technologies, virtual digital humans are often used to integrate multimodal interaction methods to provide anthropomorphic intelligent interactive experience for each terminal.

[0003] Virtual digital humans are digital characters that are close to human images and created using digital technology. They have image ability, expression ability, perception and interaction ability. They can automatically read and analyze external input information through intelligent systems, decide on the subsequent output text of the digital human based on the analysis results, and drive the character model to generate corresponding voice and actions to enable the digital human to interact with users.

[0004] However, the related technologies have relatively simple settings for digital virtual humans, with only a single voice or character posture, ignoring the personalized needs of different users; and the study of driving scenarios and emotional states is often ignored during the interaction process, resulting in poor scene perception and emotional performance of the dialogue system, greatly reducing the user's service experience of voice interaction. Summary of the invention

[0005] In view of this, the present invention provides a digital virtual human dialogue method, device, vehicle and medium for intelligent connected vehicles to solve the problem that the existing dialogue system proposed in the above technical background ignores the personalized needs of users and lacks in-depth research on scene perception and emotional expression, resulting in many defects, which in turn seriously affects the user experience.

[0006] In a first aspect, the present invention provides a method for communicating with a digital virtual human in an intelligent networked vehicle, the method comprising:

[0007] generating a corresponding digital virtual person in response to a character setting operation of a user;

[0008] Collect user's voice data and vehicle status data;

[0009] Recognize voice data and determine user intent;

[0010] Identify the vehicle's current scenario and the corresponding conversation service based on the user's intent and status data;

[0011] The control instructions are determined based on the digital virtual human, the current scene and the dialogue service, and the digital virtual human is controlled to perform corresponding actions based on the control instructions to complete the dialogue with the user.

[0012] The present invention generates a corresponding digital virtual human in response to the user's character setting operation, determines the current scene and its corresponding dialogue service according to the collected user's voice data and vehicle status data, and obtains control instructions to control the digital virtual human to perform corresponding actions. It can take into account the user's personalized needs, solve the problems of weak scene perception and insufficient emotional expression in the dialogue process of the digital virtual human of the smart connected car, and greatly enhances the humanized and intelligent interactive experience between the user and the digital virtual human.

[0013] In an optional implementation, generating a corresponding digital virtual person in response to a character setting operation of a user includes:

[0014] In response to the user's character setting operation on the preset character setting elements and the preset image elements, a target character setting and a target image are generated accordingly;

[0015] Generate a digital virtual person based on the target persona and target image;

[0016] The image of the digital virtual person is optimized according to a preset optimization method to obtain an optimized digital virtual person.

[0017] The present invention obtains a target personality and a target image in response to a character setting operation of a user, obtains a digital virtual person based on the target personality and the target image, and optimizes the image thereof, which not only meets the personalized needs of the user and generates a corresponding digital virtual person, but also ensures the quality of the generated digital virtual person and improves its intelligent and anthropomorphic image.

[0018] In an optional implementation, identifying the current scene of the vehicle and the dialogue service corresponding to the current scene according to the user intention and the state data includes:

[0019] Determine the corresponding scene and emotional voice based on user intent and status data;

[0020] Determine the vehicle's current scene and the dialogue service corresponding to the current scene based on the scene and emotional speech.

[0021] The present invention determines the corresponding scene and emotional speech through user intention and status data, and based on it determines the current scene of the vehicle and the dialogue service corresponding to the current scene. It can take into account the driving scene and emotional state during the dialogue process, help improve scene perception and emotional expression, and thus meet the user's intelligent dialogue needs.

[0022] In an optional implementation, the control instruction includes an action instruction and / or a voice instruction, the control instruction is determined based on the digital virtual person, the current scene and the dialogue service, and the digital virtual person is controlled to perform corresponding actions based on the control instruction to complete the dialogue with the user, including:

[0023] Matching the reply language and action and / or emotional speech of the dialogue service based on the digital virtual person and the current scene;

[0024] Configuring animation instructions and / or voice instructions based on the response language and the action and / or emotional voice;

[0025] The digital virtual human is controlled to perform corresponding actions according to animation instructions and / or voice instructions to complete the conversation with the user.

[0026] The present invention matches the reply language, action and / or emotional speech of the dialogue service between the digital virtual human and the current scene, and determines animation instructions and / or voice instructions based on them, so that the interaction between the user and the digital virtual human is more humanized and intelligent, which can greatly improve the quality and efficiency of the dialogue and meet the user's needs for intelligent dialogue.

[0027] In an optional implementation, configuring a voice instruction based on the reply language and the emotional voice includes:

[0028] The reply words and the emotional speech are input into a preset acoustic model to output the target emotional speech, and the target emotional speech is determined as a voice command.

[0029] The present invention synthesizes the current reply language and emotional speech through a preset acoustic model, and outputs corresponding voice instructions containing emotional features, which can achieve more accurate emotional voice dialogue and enhance the user's service experience of voice interaction.

[0030] In a second aspect, the present invention provides a digital virtual human dialogue device for an intelligent networked vehicle, the device comprising:

[0031] A character setting module, generating a corresponding digital virtual person in response to a user's character setting operation;

[0032] A data collection module, used to collect user's voice data and vehicle status data;

[0033] Intent recognition module, used to recognize voice data and determine user intent;

[0034] The scene engine module is used to identify the current scene of the vehicle and the dialogue service corresponding to the current scene based on the user's intention and status data;

[0035] The dialogue service module is used to determine control instructions based on the digital virtual human, the current scene and the dialogue service, and control the digital virtual human to perform corresponding actions based on the control instructions to complete the dialogue with the user.

[0036] The digital virtual human dialogue device for intelligent networked vehicles of the present invention generates a corresponding digital virtual human through character setting operations, determines the current scene and its corresponding dialogue service based on the user's voice data and the vehicle's status data, and obtains control instructions to control the digital virtual human to perform corresponding actions. It can take into account the user's personalized needs, solve the problems of weak scene perception and insufficient emotional expression of the digital virtual human in the dialogue process of the intelligent networked vehicle, and to a certain extent improves the humanized and intelligent interactive experience between the user and the digital virtual human.

[0037] In an optional implementation, the character setting module includes:

[0038] A character setting unit, used to provide different characters including preset character elements, so that the user can generate a target character after performing a character setting operation;

[0039] An image setting unit, used to provide different images including preset image elements, so that the user can generate a corresponding target image after performing a character setting operation;

[0040] A character generation unit, used to generate a digital virtual person according to a target person setting and a target image;

[0041] The image optimization unit is used to optimize the image of the digital virtual person according to a preset optimization method to obtain an optimized digital virtual person.

[0042] The present invention provides a target personality and a target image obtained after a personality setting operation through a personality setting unit and an image setting unit, generates a digital virtual person through a personality generation unit, and optimizes the image of the digital virtual person in combination with an image optimization unit, which not only meets the personalized needs of users and generates a corresponding digital virtual person, but also ensures the quality of the generated digital virtual person and improves its intelligent and anthropomorphic image.

[0043] In an optional implementation, the scene engine module includes:

[0044] Data candidate unit, used to provide emotional speech corresponding to different scenes and different characters;

[0045] A scene recognition unit, used to determine the corresponding scene and emotional speech from the data candidate unit according to the user intention and state data, and determine the current scene of the vehicle and the dialogue service corresponding to the current scene based on the scene and the emotional speech;

[0046] The scene output unit is used to output the current scene of the vehicle and the dialogue service corresponding to the current scene.

[0047] The scene recognition unit of the present invention determines the corresponding scene and emotional speech from the data candidate unit according to the user intention and status data, and determines the current scene of the vehicle and the dialogue service corresponding to the current scene based on it. It can take into account the driving scene and emotional state during the dialogue process, help improve scene perception and emotional expression, and thus meet the user's intelligent dialogue needs.

[0048] In an optional implementation, the control instruction includes an action instruction and / or a voice instruction, and the dialogue service module includes:

[0049] The dialogue design unit is used to define the responses for different scenarios and the actions and / or emotional voices for different characters;

[0050] A dialogue configuration unit, used to match the reply language and action and / or emotional voice of the dialogue service from the dialogue design unit based on the digital virtual person and the current scene, and configure animation instructions and / or voice instructions based on the reply language and action and / or emotional voice;

[0051] The dialogue execution unit is used to control the digital virtual human to perform corresponding actions according to animation instructions and / or voice instructions to complete the dialogue with the user.

[0052] The dialogue configuration unit of the present invention matches the reply language and action and / or emotional voice of the dialogue service from the dialogue design unit based on the digital virtual person and the current scene, and determines animation instructions and / or voice instructions based thereon, thereby making the interaction between the user and the digital virtual person more humanized and intelligent, which can greatly improve the quality and efficiency of the dialogue and meet the user's needs for intelligent dialogue.

[0053] In a third aspect, the present invention provides a vehicle, comprising a controller, the controller comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute a method for digital virtual human dialogue in an intelligent connected vehicle according to the first aspect or any corresponding embodiment thereof.

[0054] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute a method for dialogue with a digital virtual human of an intelligent connected vehicle according to the first aspect or any corresponding embodiment thereof.

[0055] The digital virtual human dialogue method and device for intelligent network-connected vehicles of the present invention generates a corresponding digital virtual human in response to the user's character setting operation, determines the current scene and its corresponding dialogue service based on the collected user's voice data and vehicle status data, and obtains control instructions to control the digital virtual human to perform corresponding actions. It can fully consider the user's personalized needs, and at the same time solve the problems of weak scene perception and insufficient emotional expression of the digital virtual human in the dialogue process of the intelligent network-connected vehicle, which helps to enhance the humanized and intelligent interactive experience between the user and the digital virtual human. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 is a flow chart of a method for communicating with a digital virtual human in an intelligent networked vehicle according to an embodiment of the present invention;

[0058] Figure 2 is a flow chart of another method for communicating with a digital virtual human in an intelligent networked vehicle according to an embodiment of the present invention;

[0059] Figure 3 It is a schematic diagram of the architecture of the voice design system;

[0060] Figure 4 It is a schematic diagram of emotional dialogue design;

[0061] Figure 5 It is a schematic diagram of the scene dialogue configuration;

[0062] Figure 6 is a predicted schematic diagram of the synthesized audio;

[0063] Figure 7 is a structural block diagram of a digital virtual human dialogue device for an intelligent networked vehicle according to an embodiment of the present invention;

[0064] Figure 8 It is a schematic diagram of the structure of a controller of a vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0066] An embodiment of the present invention provides an embodiment of a method for dialogue between a digital virtual human in an intelligent connected vehicle. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0067] In this embodiment, a method for communicating with a digital virtual person in an intelligent network-connected vehicle is provided. Figure 1 is a flow chart of a method for communicating with a digital virtual human in an intelligent networked vehicle according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0068] Step S101, generating a corresponding digital virtual person in response to a character setting operation of a user.

[0069] It should be noted that the specific method of character setting operation in this embodiment is not limited here, and is determined based on actual needs and adaptability. For example, the user selects the corresponding character by touching the car screen; or adopts voice-related technology to select the corresponding character. Specifically, the user wakes up the digital virtual person setting function by setting a voice wake-up word, performs voice recognition on the relevant voice spoken by the user, and designs the digital virtual person based on the recognition result to meet the user's personalized needs, which is only an exemplary description.

[0070] Step S102, collecting the user's voice data and the vehicle's status data.

[0071] It should be noted that the specific content and acquisition method of the voice data and status data in this embodiment are not limited here, and can be determined based on actual needs. For example, the user's voice is collected in real time through the microphone installed in the car; the vehicle's status data includes but is not limited to the vehicle's status signal, location signal, traffic signal, weather signal, etc., where each type of signal data is obtained according to conventional data acquisition methods in the field, such as using the vehicle navigation system to obtain the vehicle's corresponding location signal in real time, which is only an exemplary description.

[0072] Step S103: recognizing voice data and determining user intention.

[0073] In this embodiment, the user intention is mainly reflected in the current user's demand for the use of the vehicle. For example, after the user says "help me navigate home", voice recognition can be used to determine that the user intends to use the navigation function. This is only an example and is not intended to be limiting. Other user intentions in the automotive field can be adaptively adjusted according to actual driving scenarios.

[0074] Step S104, identifying the current scene of the vehicle and the dialogue service corresponding to the current scene according to the user intention and status data.

[0075] It should be noted that the current scene of the vehicle is intended to indicate the current usage environment and conditions of the vehicle. The specific scene of the vehicle in this embodiment can be adaptively set based on actual project needs, such as navigation scenes (such as using the navigation function), entertainment scenes (such as listening to music), vehicle control scenes (such as controlling the vehicle to travel at a constant speed), query scenes (such as checking stocks) and chat scenes (such as chatting with the car assistant), which are only for illustrative purposes.

[0076] Step S105, determining a control instruction based on the digital virtual human, the current scene and the dialogue service, and controlling the digital virtual human to perform corresponding actions based on the control instruction to complete the dialogue with the user.

[0077] In this embodiment, the control instruction is mainly used to control the digital virtual person to perform corresponding actions to achieve dialogue interaction with the user. The specific content and expression form are not limited here, and are adaptively adjusted based on actual needs. For example, the control instruction is a voice instruction, and the voice instruction enables the digital virtual person to interact with the user by voice, which is only an exemplary explanation.

[0078] The digital virtual human dialogue method for a smart connected car according to an embodiment of the present invention generates a corresponding digital virtual human in response to a user's character setting operation, determines the current scene and its corresponding dialogue service based on the collected user's voice data and vehicle status data, and obtains control instructions to control the digital virtual human to perform corresponding actions. It can take into account the user's personalized needs, solve the problems of weak scene perception and insufficient emotional expression during the dialogue process of the digital virtual human of the smart connected car, and greatly enhances the humanized and intelligent interactive experience between the user and the digital virtual human.

[0079] In this embodiment, a method for communicating with a digital virtual person in an intelligent network-connected vehicle is provided. Figure 2 FIG. 1 is a flow chart of another method for communicating with a digital virtual human in an intelligent networked vehicle according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0080] Step S201, generating a corresponding digital virtual person in response to the user's character setting operation.

[0081] Specifically, the above step S201 includes:

[0082] Step S2011, in response to the user's character setting operation on the preset character setting elements and the preset image elements, a target character setting and a target image are correspondingly generated.

[0083] In this embodiment, the specific contents of the preset person setting elements and the preset image elements are not limited here, and can be adaptively adjusted based on actual project requirements. For example, according to the user's social attributes and personality attributes, their appearance, shape, figure, clothing, personality characteristics, personality traits, preferences and other attributes are set and used as specific person setting elements, which is only for illustrative purposes.

[0084] Step S2012, generating a digital virtual human according to the target personality and target image.

[0085] Step S2013, optimizing the image of the digital virtual human according to a preset optimization method to obtain an optimized digital virtual human.

[0086] In this embodiment, the specific method of the preset optimization method is adaptively set based on actual needs. For example, the preset optimization method includes a dressing optimization method, that is, dressing the digital virtual person according to holidays and user preferences; the preset optimization method includes a broadcast optimization method, that is, using lip synchronization technology, such as Wav2Lip, DeepFake, PaddleGAN, etc., to give the digital virtual person image more vivid and realistic lip movements, so that it can present a more natural and fluent voice expression during the dialogue interaction with the user.

[0087] The embodiment of the present invention obtains a target personality and a target image in response to a user's character setting operation, obtains a digital virtual person based on the target personality and the target image, and optimizes the image thereof, which not only meets the user's personalized needs and generates a corresponding digital virtual person, but also ensures the quality of the generated digital virtual person and improves its intelligent and anthropomorphic image.

[0088] Step S202: Collect the user's voice data and the vehicle's status data. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0089] Step S203: Recognize the voice data and determine the user's intention. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0090] Step S204, identifying the current scene of the vehicle and the dialogue service corresponding to the current scene according to the user intention and status data.

[0091] Specifically, the above step S204 includes:

[0092] Step S2041, determining the corresponding scene and emotional speech according to the user intention and status data.

[0093] It should be noted that emotional speech can help voice interaction be warmer and more humanized. In this embodiment, the corresponding scene is first determined according to the user's intention and status data, and then the emotional speech is determined based on the designed scene emotional arrangement.

[0094] Step S2042, determining the current scene of the vehicle and the dialogue service corresponding to the current scene based on the scene and the emotional voice.

[0095] In this embodiment, the specific content of the dialogue service is not limited here. For example, the dialogue service includes a multi-round dialogue service, a question-and-answer chat service, an inquiry service, etc., which is only used as an exemplary description.

[0096] In the embodiment of the present invention, the corresponding scene and emotional speech are determined by user intention and status data, and the current scene of the vehicle and the dialogue service corresponding to the current scene are determined based on them. This can take into account the driving scene and emotional state during the dialogue process, help improve scene perception and emotional expression, and thus meet the user's intelligent dialogue needs.

[0097] Step S205, determining a control instruction based on the digital virtual human, the current scene and the dialogue service, and controlling the digital virtual human to perform corresponding actions based on the control instruction to complete the dialogue with the user.

[0098] In this embodiment, the control instructions include action instructions and / or voice instructions, which are used to control the digital virtual human to perform corresponding actions according to the user's preferences, aiming to improve the user experience and satisfaction, wherein the action instructions are intended to achieve the digital virtual human's action response to the user (such as eye contact), and the voice instructions are intended to achieve natural language interaction between the digital virtual human and the user. Specifically, the above step S205 includes:

[0099] Step S2051, matching the reply language and action and / or emotional voice of the dialogue service based on the digital virtual human and the current scene.

[0100] In this embodiment, emotional speech (also called emotional TTS) is speech carrying emotional features, and the specific type of emotional features is adaptively set based on actual needs, such as happiness, which is only used as an exemplary description.

[0101] It should be noted that TTS (Text To Speech) is a technology that involves multiple disciplines such as acoustics, linguistics, natural language understanding, signal processing, pattern recognition, etc., and converts text information into natural and fluent voice output. As an important component of human-computer dialogue, it aims to enable machines to "speak".

[0102] Step S2052, configuring animation instructions and / or voice instructions based on the reply language and the action and / or emotional voice.

[0103] It should be noted that in this embodiment, voice instructions are configured based on reply words and emotional speech, including: inputting the reply words and emotional speech into a preset acoustic model to output the target emotional speech, and determining the target emotional speech as a voice instruction. Among them, the preset acoustic model is a trained deep learning model (also called an acoustic model) for emotional speech synthesis, and its specific model type is adaptively selected according to actual data. Specifically, by synthesizing the current reply words and emotional speech through the preset acoustic model, and outputting the corresponding voice instructions containing emotional features, more accurate emotional voice dialogue can be achieved, which improves the user's service experience of voice interaction.

[0104] In this embodiment, the animation instruction is used to instruct the digital virtual human to perform a specific animation set in advance. It should be noted that the animation includes specific set actions (such as waving, clapping, spinning, jumping, etc.) and effects (also known as special effects, which are intended to add visual impact and expressiveness to the actions of the digital virtual human, such as flames, explosions, light and shadow, etc., to make the animation scene more vivid and attractive). The specific content of the animation can be adaptively adjusted according to the actual project requirements.

[0105] Step S2053, controlling the digital virtual human to perform corresponding actions according to the animation instructions and / or voice instructions to complete the conversation with the user.

[0106] In the embodiment of the present invention, the digital virtual human and the current scene match the reply language and action and / or emotional voice of the dialogue service, and determine animation instructions and / or voice instructions based on them, so that the interaction between the user and the digital virtual human is more humanized and intelligent, which can greatly improve the quality and efficiency of the dialogue and meet the user's needs for intelligent dialogue.

[0107] In a specific embodiment, according to the above-mentioned intelligent networked vehicle digital virtual human dialogue method, an intelligent networked vehicle digital virtual human emotional voice system is designed, that is, a voice design system with emotions and anthropomorphism, and the system includes the following contents:

[0108] 1. Build an anthropomorphic digital virtual human.

[0109] In this embodiment, the digital virtual person includes personality guidance and image linkage.

[0110] 1.1 Character setting guidance: relevant requirements for character setting.

[0111] 1.1.1 Character setting: defines the overall framework of character creation, which directly affects the impression of digital virtual people in the minds of users. Character setting includes many elements, such as name, background story, occupation, personality, hobbies, age, design style (such as 2D / 3D, character proportions, hairstyle, clothing), representative expressions / actions / catchphrases, photos and videos uploaded by users, etc.

[0112] 1.1.2 Character design coordinates all emotional voice interactions, image movements, specific scene animations, reply words, TTS, scene emotions, etc., which need to be designed around character design.

[0113] 1.1.3 Different solutions should be adopted for different characters. The character's name can be externalized as a voice wake-up word based on the user's preferences.

[0114] 1.2 Image linkage: combine roles or images from different fields to adapt to users’ personalized needs.

[0115] 1.2.1 In real scenes, emotional expression can be achieved through image effects that are appropriate to the scene, and based on interaction with the entire scene. For example: navigation scenes, music scenes, and certain scenes with emotional touchpoints, such as holiday scenes and proactive service scenes, can use image linkage to enhance the sense of companionship.

[0116] 1.2.2 Image refers to the image of a virtual digital person, which is usually divided into anthropomorphic images such as hair, eyes, mouth, nose, ears, hands, feet, etc.; or anthropomorphic images such as animals and objects. For example: Michelin tires, Tom Cat, Pop Mart, Ai Yun, etc. The image is the digital virtual person steward of the car's central control, which can chat with the car owner, execute commands, provide reminders and help, and other intelligent services, that is, a digital virtual person who resonates with the owner.

[0117] 1.2.3 Use AI technologies such as Stable Diffusion or Midjourney to generate virtual digital human images.

[0118] 1.2.4 Based on the voice interaction cloud and its local dynamic behavior database and intelligent recommendation services, drive the virtual human image to expand its usage scenarios.

[0119] 1.2.5 When the virtual digital person is broadcasting a conversation, lip synchronization technology (such as Wav2Lip, to make the spoken audio and text more humanized) is used to input the synthesized TTS audio into the model, matching the phonemes and lip movements of the speech, so that the lip movements of the virtual digital person can accurately express the speech information.

[0120] 1.2.6 Provide peripheral materials of digital assets, support the dressing up of digital virtual people and the matching of holiday costumes, aiming to create an anthropomorphic digital virtual person with high affinity and intelligence.

[0121] 2. Scene design.

[0122] In this embodiment, the scene design includes various arrangement design requirements and design compliance principles.

[0123] 2.1 Requirements for choreography and design.

[0124] 2.1.1 Emotional speech can help voice interaction be more humanized. In scene design, emotional speech is based on the determined persona and is reflected through image animation and emotional dialogue design. Emotional dialogue is divided into reply speech and TTS design (including timbre and emotional style).

[0125] 2.1.2 Through various features input by users, the trained character model is an intelligent digital person with specific characteristics and self-learning ability. The conversations, car usage characteristics, habits and other effective owner information during the driving time with the owner can improve its own intelligence and character richness. Specifically, the digital virtual person is like a "good friend" of the owner, and various scene designs must be effective and in line with the owner's preferences to enhance the user's human-computer interaction experience.

[0126] 2.1.3 More sophisticated design and arrangement are required for high-frequency user scenarios, such as navigation, music, vehicle control, telephone, weather, stock market, video and other scenarios, and targeted design of corresponding scenarios is required.

[0127] 2.1.4 Visual editing platform module, used to receive and analyze user voice information, and based on preset definition and arrangement strategies, define and arrange the input process, processing process and output process of the scene module, so that users can view and modify scene-related information at any time, with significant advantages of intuitiveness and flexibility.

[0128] 2.2 Design follows principles.

[0129] 2.2.1 Extreme and restrained: A design concept that pursues extreme performance while maintaining restraint and simplicity. Among them, "extreme" means that the design of the reply should be the best, the most outstanding and without regrets; while "restraint" emphasizes that the design of the reply should follow the concept of simplicity, clarity, sincerity and avoidance of redundancy. It should be noted that general task-based scenarios can adopt a general design, and special scenarios need to be focused and designed with emphasis.

[0130] 2.2.2 Select high-frequency and emotionally engaging scenarios. The architecture diagram of the voice design system is as follows: Figure 3 It should be noted that Figure 3ASR (Automatic Speech Recognition) is a technology that converts human speech into text; NLU (Natural Language Understanding) is a technology that enables computers to understand and use natural languages ​​of human society, such as Chinese and English, to achieve natural language communication between humans and machines; DM (Dialogue Management) obtains tasks based on user input, then clarifies the information required for the task, connects to the business platform to complete the task, and finally returns the task execution results to the user; NLG (Natural Language Generation) is designed to enable computers to generate and understand human natural language text. Specifically, in the process of voice interaction, ASR allows the machine to hear, NLU allows the machine to understand, and dialogue management, as the brain of voice interaction, controls the machine to perform corresponding processing after understanding.

[0131] 3. Dialogue design.

[0132] In this embodiment, the dialogue design includes emotional dialogue design, scenario dialogue design, cloud platform dynamic configuration and prediction model.

[0133] 3.1 Emotional dialogue design.

[0134] In this embodiment, the emotional dialogue design includes the response language design and the TTS design, and the design should follow the principle of consistency of the character setting. Figure 4 It is a schematic diagram of emotional dialogue design.

[0135] 3.1.1 Response design: Match the goal of the user dialogue. Among them, task-type scenarios focus on simplicity, directness, and efficiency; entertainment scenarios focus on fun and interaction. It should be noted that dialogue design must also follow the two major principles of scenario design. For example: in a negative interaction scenario, when the user initiates navigation home without setting a home address, the emotionally designed response is: "You haven't told me where your home is. Is it in my heart?", see Figure 4 Schematic diagram of emotional dialogue design.

[0136] 3.1.2 TTS design: Based on the style, timbre, intonation, tone, speed, etc. of the digital virtual person, provide corpus with different emotions / feelings and emotional speech audio of the person (obtained through a model that can generate different emotional speech after deep learning training). Among them, emotional TTS carries emotional features, such as: happy, sad, angry, naughty, cute, comforting, etc.

[0137] 3.2 Dynamic configuration of the cloud platform for scenario dialogue design.

[0138] In this embodiment, emotion tags can be configured online in the cloud. When designing a dialogue in the scenario design module, emotion tags are added and then added to the command semantics to respond to the vehicle side. Figure 5 This is a schematic diagram of the scene dialogue configuration. As can be seen from the figure, after the user selects the scene label, reply language and emotion type provided by the cloud platform, the platform automatically responds to the user's selection and completes the corresponding configuration.

[0139] In a specific embodiment, the user can use the wake-up voice query: "Help me navigate home", the voice assistant will recognize the real-time user audio and perform NLU to understand the "navi" navigation vertical domain intention (in the automotive field, navi stands for Navigation service), navigate home and then access the cloud platform; intelligently match the arranged atomic reply language according to the navigation domain scenario, and match the emotional tags of "cute" or "happy" according to the weather conditions of the day, and add the semantic atomic instructions to the semantic protocol in the format of Table 1 below before the semantic atomic instructions are issued; after the semantics are integrated in the cloud, the voice assistant will arbitrate and issue it to the registered business party, and then the receiving business party will parse the instruction and call back to the voice to link TTS emotional broadcast and virtual image broadcast.

[0140] Table 1

[0141] Fields meaning Is it necessary Remark status state yes - text Reply yes - emotion Image Action yes -

[0142] For example, when the user's voice input command is "Switch to high-speed priority", the following choreographed reply is matched before the DM semantics are sent:

[0143]

[0144]

[0145] It should be noted that the domain scenarios in this embodiment are arranged by combining various scenarios developed with the semantic skill tree, such as navigation, music, video, car control, broadcast control, weather, telephone, ticket inquiry, and chat. In addition, the virtual persona trained based on the fusion of various features input by the user expresses and communicates in a way that is more in line with the needs of the user. In actual applications, large models such as chatGPT can be used to input the feature labels of the virtual persona to obtain a more rich and intelligent reply.

[0146] 3.3 Prediction model.

[0147] In this embodiment, text information and phoneme timestamps of the audio are extracted to form a training set for training. The phoneme stamps and pronunciation frame lengths of the text information pronunciation audio are predicted by the trained timestamp prediction model to generate a phoneme sequence. The phoneme sequence is then input into the acoustic model to obtain synthesized audio for personalized voice broadcasting of digital virtual people. Figure 6 It is a prediction diagram of synthesized audio.

[0148] It should be noted that the prediction model determines the speech features of the original text according to the target emotion and emotional timbre, and synthesizes the target emotional speech based on the speech features and target timbre of the original text. This method only needs to collect one or a few speech data with different target emotions when synthesizing speech with different target timbres and different target emotions, thereby reducing the cost of emotional speech synthesis and reducing the impact of the emotional expression ability of speakers with corresponding timbre on speech quality. It can more accurately determine speech features, improve the sound quality of synthesized speech, and improve the service experience of voice interaction.

[0149] In summary, the digital virtual human dialogue method for an intelligent connected vehicle according to an embodiment of the present invention has the following advantages:

[0150] 1. The digital virtual human of this embodiment can more intuitively display rich and diverse anthropomorphic emotional states. Combined with the well-designed emotional voice dialogue scenes and lip movement models, the digital human can have rich facial expressions, body movements and human warmth when linked.

[0151] 2. The synthesized audio of this embodiment adds a timestamp prediction model, predicts the phoneme stamp and pronunciation frame length of the text audio to generate a phoneme sequence, and then inputs the phoneme sequence into the acoustic model to synthesize the audio, which can achieve more accurate emotional voice dialogue.

[0152] 3. The design of the reply language in the scene arrangement of this embodiment is based on the personality and image attributes of the digital virtual person, such as age, personality, preferences, etc., and a large model such as chatGPT is added to the cloud for training to train a large model that conforms to the attributes of the digital virtual person. The large model is allowed to automatically arrange customized reply language, and combined with emotional annotations for more appropriate emotional replies, which can overcome the many drawbacks of manual design of reply language, further speed up the rhythm of dialogue, and enhance the interactive experience.

[0153] In this cost embodiment, a digital virtual human dialogue device for an intelligent networked vehicle is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" refers to a combination of software and / or hardware that can implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0154] The present invention provides a digital virtual human dialogue device for an intelligent network-connected vehicle, such as Figure 7 As shown, the device comprises:

[0155] The character setting module 701 generates a corresponding digital virtual person in response to the user's character setting operation.

[0156] The data collection module 702 is used to collect the user's voice data and the vehicle's status data.

[0157] The intention recognition module 703 is used to recognize voice data and determine the user's intention.

[0158] The scene engine module 704 is used to identify the current scene of the vehicle and the dialogue service corresponding to the current scene according to the user's intention and status data.

[0159] The dialogue service module 705 is used to determine the control instruction based on the digital virtual human, the current scene and the dialogue service, and control the digital virtual human to perform corresponding actions based on the control instruction to complete the dialogue with the user.

[0160] In some optional embodiments, the character setting module includes: a character setting unit, which is used to provide different characters containing preset character elements, so that the user can generate a corresponding target character after performing a character setting operation; an image setting unit, which is used to provide different images containing preset image elements, so that the user can generate a corresponding target image after performing a character setting operation; a character generation unit, which is used to generate a digital virtual person according to the target character and the target image; and an image optimization unit, which is used to optimize the image of the digital virtual person according to a preset optimization method to obtain an optimized digital virtual person.

[0161] The character setting module of the embodiment of the present invention provides a target character setting and a target image obtained after a character setting operation through a character setting unit and an image setting unit, generates a digital virtual person through a character generation unit, and optimizes the image of the digital virtual person in combination with an image optimization unit. This not only meets the personalized needs of users and generates a corresponding digital virtual person, but also ensures the quality of the generated digital virtual person and improves its intelligent and anthropomorphic image.

[0162] In some optional embodiments, the scene engine module includes: a data candidate unit, used to provide different scenes and emotional speech corresponding to different characters; a scene recognition unit, used to determine the corresponding scene and emotional speech from the data candidate unit according to user intention and status data, and determine the current scene of the vehicle and the dialogue service corresponding to the current scene based on the scene and the emotional speech; a scene output unit, used to output the current scene of the vehicle and the dialogue service corresponding to the current scene.

[0163] The scene engine module of the embodiment of the present invention determines the corresponding scene and emotional speech from the data candidate unit according to the user intention and status data through the scene recognition unit, and determines the current scene of the vehicle and the dialogue service corresponding to the current scene based on the scene recognition unit. It can take into account the driving scene and emotional state during the dialogue process, help improve scene perception and emotional expression, and thus meet the user's intelligent dialogue needs.

[0164] In some optional embodiments, the dialogue service module includes: a dialogue design unit, which is used to define reply words corresponding to different scenarios and actions and / or emotional voices corresponding to different personalities; a dialogue configuration unit, which is used to match the reply words and actions and / or emotional voices of the dialogue service from the dialogue design unit based on the digital virtual person and the current scenario, and configure animation instructions and / or voice instructions based on the reply words and actions and / or emotional voices; a dialogue execution unit, which is used to control the digital virtual person to perform corresponding actions according to the animation instructions and / or voice instructions to complete the dialogue with the user.

[0165] The dialogue service module of the embodiment of the present invention matches the reply words and actions and / or emotional voices of the dialogue service from the dialogue design unit based on the digital virtual person and the current scene through the dialogue configuration unit, and determines animation instructions and / or voice instructions based thereon, so that the interaction between the user and the digital virtual person is more humanized and intelligent, which can greatly improve the quality and efficiency of the dialogue and meet the user's needs for intelligent dialogue.

[0166] In some optional embodiments, the device also includes: a visual editing platform module, which is used to define and arrange the input process, processing process and output process of the scene engine module based on preset definition and arrangement strategies.

[0167] The visual editing platform module of the embodiment of the present invention can view and modify conversation-related information at any time, and has significant advantages of being intuitive and flexible.

[0168] In some optional embodiments, the device further includes: an emotional speech synthesis module, which is used to synthesize emotional speech according to a preset acoustic model, and provide voice instructions corresponding to emotional features to the dialogue service module.

[0169] The emotional speech synthesis module of the embodiment of the present invention synthesizes the current reply and emotional speech through a preset acoustic model, and outputs corresponding voice instructions containing emotional characteristics, which can achieve more accurate emotional voice dialogue and enhance the user's service experience of voice interaction.

[0170] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0171] The digital virtual human dialogue device of the intelligent connected car in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0172] The digital virtual human dialogue device for a smart connected car according to an embodiment of the present invention can take into account the personalized needs of users, solve the problems of weak scene perception and insufficient emotional expression during the dialogue process of the digital virtual human of a smart connected car, and to a certain extent improve the humanized and intelligent interactive experience between the user and the digital virtual human.

[0173] A vehicle is also provided in an embodiment of the present invention, and the vehicle includes a controller. The controller in this embodiment is a vehicle controller, which is used to perform operations such as power supply / power off, sleep and wake up of the sub-controllers and network nodes under it, and each power supply interface thereof can collect the real-time current output. Other controllers with the above functions are applicable.

[0174] Figure 8 is a schematic diagram of the structure of the controller provided by an optional embodiment of the present invention, such as Figure 8 As shown, the controller includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the controller, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple controllers can be connected, and each controller provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0175] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0176] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0177] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the controller, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0179] The controller also includes a communication interface 30 for the main control chip to communicate with other devices or a communication network.

[0180] A computer-readable storage medium is also provided in an embodiment of the present invention. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor main control chip or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0181] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for communicating with a digital virtual person of an intelligent networked vehicle, characterized in that: The method comprises: generating a corresponding digital virtual person in response to a character setting operation of a user; Collecting the user's voice data and the vehicle's status data; Recognize the voice data and determine the user's intention; Identifying a current scene of the vehicle and a dialogue service corresponding to the current scene according to the user intention and the state data; A control instruction is determined based on the digital virtual person, the current scene and the dialogue service, and the digital virtual person is controlled to perform corresponding actions based on the control instruction to complete the dialogue with the user.

2. The method for digital virtual human dialogue for intelligent networked vehicles according to claim 1, characterized in that: The generating a corresponding digital virtual person in response to the user's character setting operation includes: In response to the user's character setting operation on the preset character setting elements and the preset image elements, a target character setting and a target image are correspondingly generated; Generate a digital virtual person according to the target persona and the target image; The image of the digital virtual person is optimized according to a preset optimization method to obtain an optimized digital virtual person.

3. The method for digital virtual human dialogue for intelligent networked vehicles according to claim 1, characterized in that: The identifying, according to the user intention and the state data, a current scene of the vehicle and a dialogue service corresponding to the current scene, comprises: Determine the corresponding scene and emotional speech according to the user intention and the status data; Based on the scene and the emotional speech, a current scene of the vehicle and a dialogue service corresponding to the current scene are determined.

4. The method for digital virtual human dialogue for intelligent networked vehicles according to claim 1, characterized in that: The control instruction includes an action instruction and / or a voice instruction, and the determining of the control instruction based on the digital virtual person, the current scene and the dialogue service, and controlling the digital virtual person to perform a corresponding action based on the control instruction to complete the dialogue with the user, includes: Matching the reply language and action and / or emotional voice of the dialogue service based on the digital virtual person and the current scene; Configure animation instructions and / or voice instructions based on the reply language and the action and / or emotional voice; The digital virtual person is controlled to perform corresponding actions according to the animation instructions and / or the voice instructions to complete the conversation with the user.

5. The method for digital virtual human dialogue for intelligent networked vehicles according to claim 4, characterized in that: Configuring voice instructions based on the reply language and the emotional voice includes: The reply language and the emotional speech are input into a preset acoustic model to output a target emotional speech, and the target emotional speech is determined as a voice instruction.

6. A digital virtual human dialogue device for an intelligent networked vehicle, characterized in that: The device comprises: A character setting module, generating a corresponding digital virtual person in response to a user's character setting operation; A data collection module, used to collect the user's voice data and the vehicle's status data; An intention recognition module, used to recognize the voice data and determine the user's intention; A scene engine module, used for identifying a current scene of the vehicle and a dialogue service corresponding to the current scene according to the user intention and the state data; The dialogue service module is used to determine the control instruction based on the digital virtual human, the current scene and the dialogue service, and control the digital virtual human to perform corresponding actions based on the control instruction to complete the dialogue with the user.

7. The digital virtual human dialogue device for intelligent networked vehicles according to claim 6, characterized in that: The character setting module includes: A character setting unit, used to provide different characters including preset character elements, so that the user can generate a target character after performing a character setting operation; An image setting unit, used to provide different images including preset image elements, so that the user can generate a corresponding target image after performing a character setting operation; A character generation unit, used to generate a digital virtual person according to the target person setting and the target image; The image optimization unit is used to optimize the image of the digital virtual person according to a preset optimization method to obtain an optimized digital virtual person.

8. The digital virtual human dialogue device for intelligent networked vehicles according to claim 6, characterized in that: The scene engine module includes: Data candidate unit, used to provide emotional speech corresponding to different scenes and different characters; A scene recognition unit, configured to determine a corresponding scene and emotional speech from the data candidate unit according to the user intention and the state data, and determine a current scene of the vehicle and a dialogue service corresponding to the current scene based on the scene and the emotional speech; The scene output unit is used to output the current scene of the vehicle and the dialogue service corresponding to the current scene.

9. The digital virtual human dialogue device for intelligent networked vehicles according to claim 6, characterized in that: The control instruction includes an action instruction and / or a voice instruction, and the dialogue service module includes: The dialogue design unit is used to define the responses for different scenarios and the actions and / or emotional voices for different characters; A dialogue configuration unit, configured to match the reply language and the action and / or the emotional voice of the dialogue service from the dialogue design unit based on the digital virtual person and the current scene, and configure animation instructions and / or voice instructions based on the reply language and the action and / or the emotional voice; A dialogue execution unit is used to control the digital virtual human to perform corresponding actions according to the animation instructions and / or the voice instructions to complete the dialogue with the user.

10. A vehicle, characterized in that: The vehicle includes a controller, which includes: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the digital virtual human dialogue method for an intelligent connected vehicle according to any one of claims 1 to 5 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the digital virtual human dialogue method for an intelligent connected vehicle according to any one of claims 1 to 5.

Images