Interaction method and device, computer equipment and storage medium

Abstract

The invention relates to an interaction method and device, computer equipment and a storage medium. The method comprises the following steps: determining a target intention related to a user according to multi-modal information related to a target vehicle; and starting a dialogue service flow, the dialogue service flow comprising: sending an instruction corresponding to the target intention to an application to which the atomization service function for realizing the target intention belongs, so that the application calls the atomization service function for realizing the target intention according to the instruction.

Description

Technical Field

[0001] The present invention relates to the field of vehicles, and in particular to an interaction method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, voice interaction has become an important part of modern smart devices, especially in the field of smart cars. It greatly improves the convenience and safety of users during driving. Through simple voice commands, a series of operations such as navigation settings, music playback, phone calls, window control, etc. can be completed, effectively reducing the driver's distraction during driving.

[0003] Most smart cars on the market are equipped with voice interaction systems based on speech recognition and natural language processing technologies. These voice interaction systems can recognize user voice commands and convert them into corresponding operation commands, achieving rapid response to functions such as playing specified songs, adjusting air conditioning temperature, and checking weather.

[0004] However, the vehicle's built-in voice functions are rich and varied, but it is often difficult for users to fully understand all available functions. After the first round of basic voice speaking, if further adjustments or exploration of other functions are required, users usually need to exit the voice interaction interface and use the touch screen to find and operate specific pages. This process is cumbersome and easily interrupts the driving process. Summary of the invention

[0005] One of the purposes of the present invention is to provide an interaction method, apparatus, computer device and storage medium to solve the problem of how to improve the convenience of interaction between a vehicle and a user in the vehicle.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] An interaction method comprises: determining a target intent related to a user based on multimodal information related to a target vehicle; starting a dialogue service flow, the dialogue service flow comprising: sending an instruction corresponding to the target intent to an application to which an atomic service function for realizing the target intent belongs, so that the application calls the atomic service function for realizing the target intent according to the instruction.

[0008] In a possible implementation, the application for distribution of the target vehicle sends an identification of the recommended atomic service function to the middleware.

[0009] In one possible implementation, middleware is used to receive an identifier of a recommended atomic function corresponding to an atomic service function for realizing a target intent; based on the identifier of the recommended atomic function, an interface component of the recommended atomic function is found; and in a dialogue service flow interface, a user interface component of the recommended atomic function is displayed.

[0010] In a possible implementation, the number of recommended atomic functions is multiple; in the dialogue service flow interface, the control for displaying the recommended atomic function includes: combining multiple interface components to obtain an interface component combination; and displaying the interface component combination.

[0011] In a possible implementation, the method further includes: calling a recommendation atomization function to which the user interface component selected by the user belongs.

[0012] In one possible implementation, calling the recommendation atomization function to which the user interface component selected by the user belongs includes: calling the recommendation atomization function to which the user interface component selected by the user belongs through a software development kit of the recommendation atomization function to which the user interface component selected by the user belongs.

[0013] In a possible implementation, the method further includes: in response to detecting an operation on an option recommended for a visible or audible operation, calling a service end to execute a corresponding atomic service function.

[0014] In a possible implementation, determining a target intent associated with a user based on multimodal information associated with a target vehicle includes: utilizing a target model on a cloud server to determine the target intent based on multimodal information associated with the target vehicle.

[0015] An interactive device, the device comprising: an intention determination unit, used to determine a target intention related to a user based on multimodal information related to a target vehicle; a startup unit, used to start a dialogue service flow, the dialogue service flow comprising: sending an instruction corresponding to the target intention to an application to which an atomic service function for realizing the target intention belongs, so that the application calls the atomic service function for realizing the target intention according to the instruction.

[0016] A computer device comprises: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the interaction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0017] A vehicle comprising:

[0018] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the interaction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0019] A computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable a computer to execute the interactive method of the first aspect or any corresponding embodiment thereof.

[0020] A computer program product includes computer instructions, wherein the computer instructions are used to enable a computer to execute the interactive method of the first aspect or any corresponding embodiment thereof.

[0021] Beneficial effects of the present invention:

[0022] According to the multimodal information related to the target vehicle, the target intent related to the user is determined; the dialogue service flow is started, and the dialogue service flow includes: sending the instruction corresponding to the target intent to the application to which the atomic service function for realizing the target intent belongs, so that the application calls the atomic service function for realizing the target intent according to the instruction, and the target intent can be realized. The user does not need to perform a series of operations on the car screen to trigger the operation that the user intends to execute, which simplifies the user's operation and improves the convenience of interaction between the vehicle and the user. At the same time, the multimodal information related to the target vehicle is obtained, and the target intent is determined according to the target model and the multimodal information related to the target vehicle. The multimodal information related to the target vehicle has a strong correlation with the user's intention. The accuracy of the determined target intent is high, and the atomic service function is called more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of an example architecture of an interactive system that can be used to execute the interactive method provided in an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of a flow chart of an interactive method provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of an example for determining target intent;

[0027] Figure 4 A flowchart for an example of determining target intent;

[0028] Figure 5 A flow chart of an example for determining a target intent associated with a navigation app on a target vehicle;

[0029] Figure 6 A flow chart of an example for determining a target intent associated with a music app on a target vehicle;

[0030] Figure 7 A schematic diagram of an example of recommending songs and sound effects;

[0031] Figure 8 A system architecture diagram showing an example of interaction between a target vehicle and a user;

[0032] Fig. 9 A schematic diagram of the identification and components of the atomic service functions;

[0033] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0035] refer to Figure 1 , which shows a schematic diagram of an example architecture of an interactive system that can be used to execute the interactive method provided by an embodiment of the present invention.

[0036] An example architecture of an interactive system that can be used to execute the interactive method provided by the embodiment of the present invention includes: a data collection unit, a data processing unit, and an atomic service function unit.

[0037] The data acquisition unit is used to collect voice data of the user's interactive voice, the internal environment image of the target vehicle, the external environment image of the target vehicle, the sensor data of the target vehicle, etc. The data processing unit includes a voice data processing module, an image processing module, a sensor data processing module, an encoding module, and a target model. The atomic service function includes: the atomic service function of each APP with atomic service function on the target vehicle. The atomic service function can be understood as the smallest functional unit, and the atomic service function can perform a single operation.

[0038] The voice data processing module can perform voice recognition on the user's interactive voice to obtain a voice recognition result of the user's interactive voice. The voice data processing unit can use a sound source localization algorithm to determine the seat position of the user who speaks the interactive voice in the target vehicle. The voice data processing unit can determine the identity of the user by matching the voiceprint feature of the user's interactive voice with the preset voiceprint feature. The preset voiceprint feature is the voiceprint feature of the user who has recorded the voiceprint feature.

[0039] The image processing module can process the internal environment image of the target vehicle to determine the age group to which the user belongs, the user's gender, the user's sitting position, the user's identity information, etc. The image processing unit can include a driver monitoring system (DMS) and an occupant monitoring system (OMS). The driver monitoring system monitors the direction of the driver's line of sight in the target vehicle in real time. The occupant monitoring system can process the vehicle's internal environment image captured by the camera of the target vehicle to determine the gender, age, gesture, user's sitting position, etc. of the passenger in the target vehicle.

[0040] The sensor data processing module can process the sensor data collected by the sensor of the target vehicle to determine the information related to the driving of the target vehicle and the information related to the weather in the area where the target vehicle is located. The information related to the driving of the target vehicle may include: the speed of the target vehicle, the remaining fuel of the target vehicle or the remaining power of the battery of the target vehicle, etc. The information related to the weather in the area where the target vehicle is located may include: the temperature of the weather and the type of the weather. The type of weather may be raining, snowing, etc.

[0041] The encoding module is used to encode the multimodal information related to the target vehicle to obtain a representation of the multimodal information related to the target vehicle.

[0042] It should be noted that the target model can be on the target vehicle or on a cloud server.

[0043] The target model is used to receive the representation of multimodal information related to the target vehicle and predict the representation of the target intention. Thus, the target intention is determined. The target model is a trained neural network. Before the interactive method provided by the embodiment of the present invention is executed for the first time, the target model is trained using the training samples of the target model and the labeling operation of the training samples of the target model.

[0044] As an example, the target model is a pre-trained language model. As an example, the pre-trained language model used as the target model is a large language model (LLM). When the input of the target model is a natural language text corresponding to a natural language speech, the pre-trained language model performs a natural language processing (NLP) task.

[0045] Training the target model can enable the target model to learn: which information in the input of the target model to use to predict the operation that satisfies the user's interactive voice instruction intention, and how to determine the target intention. The training samples of the target model may include: information used for training on the type of information involved in the multimodal information related to the target vehicle. For a piece of information, if the multimodal information related to the target vehicle includes the information, then the type of the information is the type of information involved in the multimodal information related to the target vehicle. The labeled operation of the training sample of the target model can be understood as: when the training sample of the target model is input into the target model, the operation that the target model is expected to output. When training the target model, the loss between the operation predicted by the target model and the labeled operation of the training sample can be calculated, and the parameters of the target model can be updated based on the loss.

[0046] refer to Figure 2 , which shows a flow chart of the interaction method provided by an embodiment of the present invention.

[0047] In step S201, a target intention associated with the user is determined based on multimodal information associated with the target vehicle.

[0048] The multimodal information related to the target vehicle includes: speech recognition results of the user's interactive speech, parameter information of the target vehicle, external environment information of the target vehicle, user's action information, user's riding position information, user's attribute information, user's identity information, information related to the APP on the target vehicle determined within a historical time period of using the APP on the target vehicle, navigation information related to the target vehicle, music information related to the target vehicle, and point of interest information related to the target vehicle, wherein the point of interest information related to the target vehicle indicates the points of interest reached by the target vehicle within a target historical time period and / or the points of interest reached by the target vehicle multiple times.

[0049] The speech recognition result of the user's interactive speech is obtained by performing speech recognition on the user's interactive speech. Performing speech recognition on the user's interactive speech specifically refers to performing speech recognition on the speech data of the user's interactive speech.

[0050] As an example, the parameter information of the target vehicle may include: information related to the driving of the target vehicle, and information related to the cabin of the target vehicle.

[0051] As an example, the information related to the driving of the target vehicle includes: the speed of the target vehicle, the remaining fuel of the target vehicle, or the remaining power of the battery of the target vehicle.

[0052] As an example, information related to the target vehicle's cabin includes: the brightness of the target vehicle's screen, the volume of the target vehicle's audio system, the temperature of the target vehicle's air conditioner, the air volume of the target vehicle's air conditioner, the angle of the target vehicle's seat, etc.

[0053] The external environment information of the target vehicle is information related to the external environment of the target vehicle.

[0054] As an example, the external environment information of the target vehicle includes: information related to the weather in the area where the target vehicle is located, information related to the driving safety of the road where the target vehicle is currently located, points of interest (POI) near the target vehicle, etc.

[0055] As an example, the information related to the weather in the area where the target vehicle is located includes the temperature of the weather, the type of the weather, and the like.

[0056] As an example, the information related to the driving safety of the vehicle on the road that the target vehicle is currently on includes one or more of the following: information indicating whether the road that the target vehicle is currently on is slippery, information indicating whether the road that the target vehicle is currently on is icy, information indicating whether the road that the target vehicle is currently on is bumpy, information indicating whether the road that the target vehicle is currently on is a plateau road, etc.

[0057] As an example, the user's action information includes at least some of the following items: the user's gesture, the direction in which the user's hand points, the direction in which the user's line of sight is directed, the user's action, and the like.

[0058] As an example, the attribute information of the user includes: the age group to which the user belongs, the gender of the user, etc.

[0059] The user's identity information is used to identify the user.

[0060] As an example, the user's identity information includes: the user's identifier.

[0061] The user's sitting position information indicates the user's sitting position. As an example, the user's sitting position is one of the following items: main driving position, co-pilot position, rear left position, rear middle position, rear right position.

[0062] The navigation information related to the target vehicle may include: a departure point of the current navigation of the target vehicle, a destination of the current navigation of the target vehicle, and a navigation route of the current navigation of the target vehicle.

[0063] The music information related to the target vehicle may include: a song playlist to which the identifier of the song currently played by the target vehicle belongs, and a play progress of the song to which the identifier belongs in the song playlist. The play progress indicates which song is being played.

[0064] The end time of the historical time period of using the APP on the target vehicle is before the current time. The APP on the target vehicle is used during the historical time period of using the APP on the target vehicle.

[0065] As an example, for a navigation APP on a target vehicle, the multimodal information related to the target vehicle includes: information related to the navigation APP on the target vehicle determined within a historical period of time when the navigation APP on the target vehicle was used. The information related to the navigation APP on the target vehicle determined within a historical period of time when the APP on the target vehicle was used includes: a navigation route determined within a historical period of time when the APP on the target vehicle was used, the name of the departure point of the navigation route determined within a historical period of time when the navigation APP on the target vehicle was used, the name of the destination of the navigation route determined within a historical period of time when the navigation APP on the target vehicle was used, etc.

[0066] As an example, for a music app on a target vehicle, the multimodal information related to the target vehicle includes: information related to the music app on the target vehicle determined during a historical period of use of the music app of the target vehicle. The information related to the music app of the target vehicle includes: an identification of a determined recommended song.

[0067] refer to Figure 3 , which shows a schematic diagram of an example of determining target intent.

[0068] The user speaks an interactive voice, triggering the target vehicle to obtain multimodal information related to the target vehicle. The user's interactive voice is the user's voice for interaction. The user's interactive voice can be a natural language voice, that is, a voice in the form of a natural language. The user's interactive voice can reflect the operation that the user expects the target vehicle to perform. In order to obtain multimodal information related to the target vehicle, the voice data of the user's interactive voice, sensor data, the internal environment image of the target vehicle, the external environment image of the target vehicle and other data collected by the target vehicle are obtained. The data collected by the target vehicle is processed to obtain multimodal information related to the target vehicle. The multimodal information related to the target vehicle includes at least some of the following items: the voice recognition result of the user's interactive voice, the parameter information of the target vehicle, the external environment information of the target vehicle, the user's action information, the user's attribute information, the user's identity information, and the information related to the APP on the target vehicle determined during the historical time period of using the APP on the target vehicle.

[0069] Each information in the multimodal information related to the target vehicle is encoded to obtain a representation of the multimodal information related to the target vehicle.

[0070] A representation of multimodal information related to the target vehicle is input into the target model, and the target model outputs the target intention.

[0071] In step S201, multimodal information related to the target vehicle may be input into a target model, and the target model outputs a target intention.

[0072] The target model can be on the target vehicle or on a cloud server.

[0073] It should be noted that the target model can use part of the multimodal information related to the target vehicle to predict the target intention.

[0074] As an example, the interior temperature of the target vehicle is high, the user's hand is toward the window and makes a downward gesture, and the target intention is to open the window.

[0075] As an example, a user says an interactive voice "play songs suitable for people of my age group". The target intention is to play songs suitable for users of the age group to which the user belongs.

[0076] As an example, the user says the interactive voice "Move the seat back a little bit". The target intention is to move the seat of the user's sitting position back.

[0077] refer to Figure 4 , which shows a flowchart of an example of determining target intent.

[0078] The internal environment image of the target vehicle indicates the internal environment of the target vehicle. By processing the internal environment image of the target vehicle, the age group to which the user's age belongs, the user's gestures, etc. can be obtained. The external environment image of the target vehicle indicates the external environment of the target vehicle. The sensor data and the external environment image and the sensor data are processed to obtain the external environment information of the target vehicle. The external environment information of the target vehicle may include: information related to the weather, i.e., information related to the weather in the area where the target vehicle is located, information related to the driving safety of the vehicle, i.e., information related to the driving safety of the vehicle on the road where the target vehicle is currently located, points of interest (POI) near the target vehicle, etc. The voice data of the user's interactive voice is processed to obtain the user's seating position and the voice recognition results of the user's interactive voice.

[0079] It should be noted that Figure 4The various information shown in is information in the multimodal information related to the target vehicle that may be used to determine the target intention. The target model can use part of the multimodal information related to the target vehicle to predict the target intention. In other words, only the multimodal information related to the target vehicle is provided to the target model, and the target model decides which information in the multimodal information related to the target vehicle is used to determine the target intention.

[0080] Each information in the multimodal information related to the target vehicle is encoded to obtain a representation of the multimodal information related to the target vehicle.

[0081] A representation of multimodal information related to the target vehicle is input to a target model, and the target model outputs a representation of the target intent. Thus, the target intent is determined.

[0082] refer to Figure 5 , which shows a flowchart of an example of determining a target intent associated with a navigation app on a target vehicle.

[0083] The user speaks the interactive voice "What kind of community is that?" The speech recognition result of the user's interactive voice in the multimodal information related to the target vehicle is "What kind of community is that?" The external environment information of the target vehicle in the multimodal information related to the target vehicle includes: multiple points of interest near the target vehicle, supporting facilities of the points of interest near the target vehicle, etc. The user's action information in the multimodal information related to the target vehicle includes: the direction in which the user's hand is pointing. The target intention is to find the name of the point of interest in the direction in which the user's hand is pointing and the point of interest type is a community from the multiple points of interest near the target vehicle.

[0084] refer to Figure 6 , which shows a flowchart of an example of determining a target intent associated with a music APP on a target vehicle.

[0085] When the target vehicle is traveling according to the navigation route in the navigation information related to the target vehicle, the user speaks the interactive voice "play a song", and the multiple users in the target vehicle include children. The internal image of the target vehicle is processed to determine the age group to which the age of the user belongs. The destination in the navigation information related to the target vehicle is an amusement park. The target model determines the target intent based on the age group of the children and the destination amusement park, and the target intent is to play cheerful children's songs.

[0086] In step S202, the dialog service flow is started, and the dialog service flow includes: sending instructions corresponding to the target intent to the application to which the atomic service function for realizing the target intent belongs, so that the application calls the atomic service function for realizing the target intent according to the instructions.

[0087] In the embodiment of the invention, each APP with an atomic service function on the target vehicle can have multiple atomic service functions. When the dialog flow service is started, an instruction for realizing the target intention can be sent to the corresponding application, so that the application calls the atomic service function for realizing the target intention according to the instruction.

[0088] As an example, the vehicle center APP on the target vehicle has atomic service functions such as the open window atomic service function for opening the window, the close window atomic service function for closing the window, etc. The air conditioning APP on the target vehicle has atomic service functions such as the turn on air conditioning atomic service function for turning on the air conditioning, the turn off air conditioning atomic service function for turning off the air conditioning, and the adjust air conditioning temperature atomic service function for adjusting the air conditioning temperature.

[0089] Once the target intent is determined, the dialogue service flow can be started to call the atomic service function for realizing the target intent. After calling the atomic service function for realizing the target intent, the call result corresponding to the atomic service function for realizing the target intent can be displayed in the dialogue service flow interface, wherein the dialogue service flow interface is displayed by the vehicle computer screen of the target vehicle.

[0090] The call result corresponding to the atomic service function for achieving the target intent is the result obtained by the atomic service function for achieving the target intent or the result indicating the completion of the target intent indication. For example, if the atomic service function for achieving the target intent is that the air-conditioning APP on the target vehicle has an atomic service function for turning on the air-conditioning, then the call result corresponding to the atomic service function for achieving the target intent is text indicating that the air-conditioning has been turned on. If the atomic service function for achieving the target intent is the atomic service function for searching for POIs of the search APP on the target vehicle, then the call result corresponding to the atomic service function for achieving the target intent is the search result searched out by calling the atomic service function for searching for POIs of the search APP on the target vehicle.

[0091] As an example, when the target vehicle is driving, the user becomes interested in a community that the target vehicle passes through, and the user says the interactive voice "What community is ahead?" The target intention is to search for the name of the community ahead of the target vehicle. The atomic service function for searching POIs of the search APP on the target vehicle is called to search for the name of the community ahead of the target vehicle. The user says the interactive voice "How much does it cost to rent a three-bedroom apartment?" The target intention is to search for the rent of a three-bedroom apartment in the community ahead of the target vehicle. The atomic service function for searching the rent of a house agency APP on the target vehicle is called to search for the rent of a three-bedroom apartment in the community ahead of the target vehicle. The user says the interactive voice "Call and ask about the situation," and the target intention is to call the relevant personnel of the housing transaction platform. The atomic service function of making a call of the Bluetooth phone APP is called to call the relevant personnel of the housing transaction platform.

[0092] As an example, the user speaks an interactive voice related to a music APP on the target vehicle. The target model determines the target intent related to the music APP on the target vehicle, and calls the corresponding atomic service function of the music APP on the target vehicle to realize the target intent.

[0093] In this embodiment, the target intent related to the user is determined based on the multimodal information related to the target vehicle; the dialogue service flow is started, and the dialogue service flow includes: sending instructions corresponding to the target intent to the application to which the atomic service function for realizing the target intent belongs, so that the application calls the atomic service function for realizing the target intent according to the instructions, thereby realizing the target intent. The user does not need to perform a series of operations on the vehicle screen to trigger the operation that the user intends to perform, which simplifies the user's operation and improves the convenience of interaction between the vehicle and the user. At the same time, the multimodal information related to the target vehicle is obtained, and the target intent is determined based on the target model and the multimodal information related to the target vehicle. The multimodal information related to the target vehicle has a strong correlation with the user's intention. The accuracy of the determined target intent is high, and the atomic service function is called more accurately.

[0094] In a possible implementation, the method further includes: the target vehicle's application for distribution sends an identifier of the recommended atomic service function to the middleware.

[0095] The recommended atomic service function may be a recommended atomic service function that is recommended and associated with the atomic service function of the target intent when the atomic service function used to achieve the target intent is called. The identifier of the recommended atomic service function is used to uniquely identify each recommended atomic service function. When the application of the target vehicle decides to distribute the recommended atomic service function, the identifier of the recommended atomic service function needs to be sent to the middleware. The middleware acts as a bridge, and the middleware is responsible for receiving service requests from the application, and scheduling and coordinating the corresponding recommended atomic service functions according to the service requests.

[0096] refer to Figure 7 , which shows a schematic diagram of an example of recommended songs and sound effects.

[0097] The user says the interactive voice "Recommend a song". The dialogue service flow is started, and the atomic service function of searching songs of the music APP on the target vehicle is called to search for the recommended song. The screen of the target vehicle displays the logo of the recommended song.

[0098] The recommended atomic service function corresponding to the atomic service function for achieving the target intention is the atomic service function of the music APP on the target vehicle for setting the sound effects of the recommended songs.

[0099] In this embodiment, by analyzing the atomic service functions that the user has called, the middleware can more accurately understand the user's preferences and needs. The middleware can intelligently recommend other related recommended atomic service functions, and the user does not need to search and filter the atomic service functions by himself, which improves the efficiency of the user's acquisition and use of services and enhances the convenience of interaction between the vehicle and the user.

[0100] In a possible implementation, the method further includes: using middleware to receive an identifier of a recommended atomic function corresponding to an atomic service function for realizing a target intent. According to the identifier of the recommended atomic function, an interface component of the recommended atomic function is found. In the dialogue service flow interface, the user interface component of the recommended atomic function is displayed.

[0101] The user interface component of the recommended atomic service function corresponding to the atomic service function for achieving the target intent may include: the name of the recommended atomic service function corresponding to the atomic service function for achieving the target intent, and parameters related to the recommended atomic service function corresponding to the atomic service function for achieving the target intent.

[0102] As an example, the target intent is to recommend songs, and the atomic service function used to achieve the target intent is the atomic service function of searching songs in the music app on the target vehicle. Start the conversation service flow, and call the atomic service function of searching songs in the music app on the target vehicle to search for recommended songs. The atomic service function corresponding to the atomic service function used to achieve the target intent is the atomic service function of the music app on the target vehicle for setting the sound effects of the recommended songs.

[0103] After the target vehicle's application for distribution sends the identifier of the recommended atomic service function to the middleware, the middleware receives the identifier of the recommended atomic function corresponding to the atomic service function used to achieve the target intention, and based on the identifier of the recommended atomic function, it is able to find the user interface component of the recommended atomic function corresponding to the identifier of the recommended atomic function, and then display the control of the recommended atomic function in the dialogue service flow interface through the user interface component.

[0104] The user can call the recommended atomic service function corresponding to the atomic service function for achieving the target intent by clicking on the user interface component of the recommended atomic service function corresponding to the atomic service function for achieving the target intent, speaking a voice instruction to call the recommended atomic service function corresponding to the atomic service function for achieving the target intent, etc.

[0105] In this embodiment, after using the middleware to receive the identification of the recommended atomic function corresponding to the atomic service function for achieving the target intent, the user interface component of the recommended atomic function is found out through the identification of the recommended atomic function, and then the user clicks on the user interface component or calls the recommended atomic service function corresponding to the atomic service function for achieving the target intent through voice, so as to facilitate the user to continue to interact with the vehicle and improve the convenience of interaction between the vehicle and the user.

[0106] In a possible implementation, the number of recommended atomic functions is multiple, and in the above-mentioned dialogue service flow interface, the control for displaying the recommended atomic function includes: combining multiple user interface components to obtain a user interface component combination; and displaying the user interface component combination.

[0107] There are multiple recommended atomic functions. According to data changes, the middleware first combines the atomic function components corresponding to the voice request input by the current user to the voice application, and the user interface components corresponding to each recommended atomic function to obtain the user interface component combination, and then displays the user interface combination, and then uses the user interface components of the recommended atomic service functions corresponding to the atomic service functions for achieving the target intent, so that the user can call the recommended atomic service functions corresponding to the atomic service functions for achieving the target intent by clicking on the user interface components of the recommended atomic service functions corresponding to the atomic service functions for achieving the target intent, speaking a voice indicating the call of the recommended atomic service functions corresponding to the atomic service functions for achieving the target intent, and so on.

[0108] refer to Figure 8 , which shows a system architecture diagram of an example of interaction between a target vehicle and a user.

[0109] The multimodal information related to the target vehicle is input into the target model, and the target model outputs the target intent. The skill distribution function of the voice APP of the target vehicle will send the instruction corresponding to the target intent to the APP of the target vehicle center APP, navigation APP, music APP and other APPs for realizing the target intent. The APP to which the atomic service function for realizing the target intent belongs calls the atomic service function for realizing the target intent according to the instruction corresponding to the target intent, obtains the call result corresponding to the atomic service function for realizing the target intent, and sends the call result to the middleware. The middleware displays the UI component or a combination of multiple UI components corresponding to the call result through the call result. In response to detecting the user's operation on a certain UI component, the instruction corresponding to the operation is generated, and the instruction corresponding to the target intent is sent to the APP of the corresponding UI component of the target vehicle and the APP of the corresponding UI component to which the atomic service function for realizing the target intent belongs.

[0110] Please refer to Fig. 9 , which shows the identification of atomic service functions and a schematic diagram of components.

[0111] Receive the voice command input by the user to the voice application or receive the event trigger information, and determine the target intent corresponding to the multimodal information. For each target intent, combine the atomic functions to determine the target atomic function. The voice application distributes the vertical domain skills to the corresponding applications, such as navigation, music, vehicle center and other APP ends through the voice SDK. The application end of each application processes the voice command, executes the command function, and synchronizes the corresponding data to a component (Vertical Pod Autoscaler, VPA) in the Kubernetes cluster for automatically adjusting Pod resource requests (CPU and memory) through the CUI middleware SDK. According to the data changes, the middleware first combines the atomic function combination corresponding to the voice request input by the current user to the voice application through the template UI and displays it on the screen on the interface, and then requests the cloud large model or the offline large model to make corresponding recommendations. The user clicks or the voice is visible to say the recommended option, and calls the business end to execute the corresponding function. Among them, the atomic function combination can be determined in the following way: a single atomic function interface is marked with a unique identifier id, and then the atomic UI display is associated with the corresponding identifier. For example: air conditioner switch id—air conditioner switch control interface—air conditioner switch UI component; air conditioner temperature id—air conditioner switch reading interface—air conditioner temperature UI component; air conditioner ** id—air conditioner ** read / write interface—air conditioner ** UI component. Another example: music information id—music content reading interface—music content UI component; sound effect information id—sound effect content reading interface—sound effect content UI component; another example: other atomic capability id—read / write interface—UI component, etc.

[0112] In this embodiment, by combining UI components, multiple related atomic functions can be integrated into one interface, so that the user can see all recommended functions at a glance, which facilitates the user to continue to interact with the vehicle and improves the convenience of interaction between the vehicle and the user.

[0113] In one possible implementation, the recommendation atomization function to which the user interface component selected by the user belongs is called.

[0114] The user can send a selection operation for selecting the recommended atomic function by clicking the mouse, tapping the keyboard, touching the screen or using voice, and accordingly, the computer device can respond to the selection operation for selecting the recommended atomic function by determining the recommended atomic function to which the user interface component selected by the user belongs from the user interface component combination, and then call the recommended atomic function to which the user interface component selected by the user belongs.

[0115] As an example, a user is driving a smart car and wants to play music using the in-car entertainment system. At this point, the user can select a user interface component, such as a "play music" button or speak a corresponding voice command, through the user interface of the smart cockpit (such as a touch screen or a voice assistant). When the user selects or triggers this user interface component, the corresponding recommendation atomization function is called to execute the user's request. For example, if the user selects the "play music" button, a recommendation atomization function responsible for music playback is called, which selects a song from the user's music library and starts playing it. Similarly, if the user requests to play a specific song or artist through a voice assistant, the corresponding recommendation atomization function is also called to recognize the user's voice command and perform the playback operation accordingly.

[0116] In this embodiment, users can select appropriate interface components from the recommended atomic functions for interaction according to their needs and preferences, so that users can obtain services that best meet their needs, thereby improving user satisfaction.

[0117] In a possible implementation, the above-mentioned calling of the recommendation atomization function belonging to the user interface component selected by the user includes: calling the recommendation atomization function belonging to the user interface component selected by the user through the software development kit of the recommendation atomization function belonging to the user interface component selected by the user.

[0118] The software development kit (SDK) can provide a set of software tools, interfaces, libraries, etc. for simplifying the software development, testing and deployment process. In the present embodiment, the SDK is used to receive and execute the recommended atomic function to which the user interface component selected by the user belongs. Specifically, in the SDK, the middleware integrates the software development kits of all atomic functions. Specifically, the SDK corresponding to the target atomic function is matched from the SDK library of the middleware, and the recommended atomic function to which the user interface component selected by the user belongs is called according to the SDK.

[0119] In this embodiment, the recommended atomization function to which the user interface component selected by the user belongs is called through the SDK of the recommended atomization function to which the user interface component selected by the user belongs, which can significantly improve development efficiency and reduce development costs.

[0120] In one possible implementation, in response to detecting an operation on an option recommended for a visible or audible operation, a service end is called to execute a corresponding atomic service function.

[0121] When it is detected that the user operates on an option recommended for a "Visible and Sayable" operation, the business end may be automatically called to execute the corresponding atomic service function.

[0122] As an example, a user is driving a smart car and browsing various functional options through the touch screen interface of the smart cockpit. The smart cockpit system supports the "see and say" function, that is, the user can perform the corresponding operation by speaking the name of the option visible on the screen. Specifically, a series of functional options are displayed on the touch screen interface of the smart cockpit, such as "navigate to destination", "play music", "adjust air conditioning temperature", etc. The voice recognition technology detects that the user has spoken the name of a visible option on the screen, such as "play music". When the target intention is confirmed (that is, the "play music" option is spoken and selected), it will call the atomic function associated with the "play music" function to achieve the target intention.

[0123] In this embodiment, by operating on the options recommended by the visible options, the business end is called to execute the corresponding atomic service function. The user does not need to manually click the buttons on the touch screen, but only needs to say the name of the visible option to perform the operation, which improves the convenience and efficiency of the operation.

[0124] An interactive device is also provided in an embodiment of the present invention, and the device is used to implement the above-mentioned method embodiment and preferred implementation mode, which have been described and will not be repeated here. As used below, the term "unit" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived. The device in the embodiment of the present invention is presented in the form of a functional unit, where the functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0125] An interactive device, the interactive device comprising: an intention determination unit, used to determine a target intention related to a user based on multimodal information related to a target vehicle; a startup unit, used to start a dialogue service flow, the dialogue service flow comprising: sending an instruction corresponding to the target intention to an application to which an atomic service function for realizing the target intention belongs, so that the application calls the atomic service function for realizing the target intention according to the instruction.

[0126] Furthermore, the above-mentioned device also includes: an identification sending unit, which is used for the target vehicle's application for distribution to send the identification of the recommended atomic service function to the middleware.

[0127] Furthermore, the above-mentioned device also includes: an identification receiving unit, which is used to use the middleware to receive the identification of the recommended atomic function corresponding to the atomic service function used to achieve the target intention; a user interface component search unit, which is used to find the user interface component of the recommended atomic function according to the identification of the recommended atomic function; and a user interface component display unit, which is used to display the user interface component of the recommended atomic function in the dialogue service flow interface.

[0128] Furthermore, it is recommended that the number of atomic functions is multiple, and the user interface component display unit is used to combine multiple user interface components to obtain a user interface component combination; and display the user interface component combination.

[0129] Furthermore, the above-mentioned device also includes: a first atomic function calling unit, which is used to call the recommended atomic function belonging to the user interface component selected by the user.

[0130] Furthermore, the first atomic function calling unit is used to call the recommended atomic function belonging to the user interface component selected by the user through the software development kit of the recommended atomic function belonging to the user interface component selected by the user.

[0131] Furthermore, the above-mentioned device also includes: a second atomic function calling unit, which is used to call the business end to execute the corresponding atomic service function in response to detecting an operation of an option recommended for a visible or audible operation.

[0132] Furthermore, the intention determination unit is used to utilize the target model on the cloud server to determine the target intention associated with the user based on the multimodal information associated with the target vehicle.

[0133] refer to Fig.10 , Fig.101 is a schematic diagram of the hardware structure of a computer device provided by an embodiment of the present invention. The computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple vehicles can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above-mentioned hardware chip can be a dedicated integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field programmable gate array, a general array logic, or any combination thereof. Among them, the memory 20 stores instructions that can be executed by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment. The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the vehicle, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state hard disk; the memory 20 may also include a combination of the above types of memories. The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 may be connected via a bus or in other ways. The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc.The output device 40 may include a display device, an auxiliary lighting device (e.g., LED), a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device may be a touch screen.

[0134] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0135] A portion of the embodiments of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in a computer-readable medium includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which the computer program instructions are executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0136] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. An interactive method, characterized in that: The method comprises: Determine the target intention associated with the user based on the multimodal information associated with the target vehicle; Start a dialog service flow, which includes: sending an instruction corresponding to the target intent to an application to which the atomic service function for realizing the target intent belongs, so that the application calls the atomic service function for realizing the target intent according to the instruction.

2. The interactive method according to claim 1, characterized in that: The method further comprises: The target vehicle's application for distribution sends an identification of the recommended atomic service function to the middleware.

3. The interactive method according to claim 2, characterized in that: The method further comprises: Receiving, by means of the middleware, an identification of a recommended atomic function corresponding to the atomic service function for achieving the target intent; According to the identification of the recommendation atomization function, the user interface component of the recommendation atomization function is found; In the conversation service flow interface, a user interface component that displays the recommendation atomization function.

4. The interactive method according to claim 3, characterized in that: There are multiple recommended atomic functions. In the dialogue service flow interface, the controls that display the recommended atomic functions include: Combining multiple user interface components to obtain a user interface component combination; The user interface component combination is displayed.

5. The interactive method according to claim 4, characterized in that: The method further comprises: Invoke the recommendation atomization function to which the user-selected UI component belongs.

6. The interactive method according to claim 5, characterized in that: The calling of the recommendation atomization function of the user interface component selected by the user includes: The software development kit of the recommendation atomization function to which the user interface component selected by the user belongs calls the recommendation atomization function to which the user interface component selected by the user belongs.

7. The interactive method according to claim 1, characterized in that: The method further comprises: In response to detecting an operation for an option recommended for a visible-or-speakable operation, a business end is called to execute a corresponding atomic service function.

8. The interactive method according to claim 1, characterized in that: The determining of the target intention related to the user according to the multimodal information related to the target vehicle includes: The target model on the cloud server is used to determine the target intention associated with the user based on the multimodal information related to the target vehicle.

9. An interactive device, characterized in that: The device comprises: an intention determination unit, for determining a target intention associated with the user based on multimodal information associated with the target vehicle; A starting unit is used to start a dialogue service flow, which includes: sending instructions corresponding to the target intent to the application to which the atomic service function for realizing the target intent belongs, so that the application calls the atomic service function for realizing the target intent according to the instructions.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the interactive method according to any one of claims 1 to 8 by executing the computer instructions.

11. A vehicle, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the interactive method according to any one of claims 1 to 8 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the interactive method according to any one of claims 1 to 8.

13. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the interactive method according to any one of claims 1 to 8.

Images