Interaction method and device based on digital twin model, electronic device and vehicle

Abstract

This application provides an interaction method, device, electronic device, and vehicle based on a digital twin model. The method includes: constructing a digital twin model and visually displaying the digital twin model on a smart device in a vehicle; wherein the digital twin model includes a vehicle sub-model and an environment sub-model; acquiring a first type of data and / or a second type of data; wherein the first type of data includes vehicle data and environment data, and the second type of data is multimodal user interaction data; analyzing and processing the first type of data and / or the second type of data based on a large in-vehicle model of the vehicle to determine a target animation effect for at least one sub-model in the vehicle sub-model and the environment sub-model; and controlling the digital twin model on the smart device to visually display the target animation effect. This method enables the visualization of the human-vehicle interaction process through a digital twin model, thereby improving the intuitiveness of the human-vehicle interaction process.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to an interaction method, device, electronic device, and vehicle based on a digital twin model. Background Technology

[0002] With the rapid development of vehicle intelligence, users and vehicles typically need to interact in various ways. Specifically, users control the vehicle through interactive operations; or the vehicle uses visual interactions to inform the user of the vehicle's status or the environmental conditions in which the vehicle is located.

[0003] In some technologies, interaction between the vehicle and the user is achieved through pop-up windows or text displayed on the vehicle's smart screen. However, the interaction process in these technologies is not intuitive, resulting in a poor user experience.

[0004] Therefore, there is an urgent need for a solution that can improve the intuitiveness of the human-vehicle interaction process in vehicles. Summary of the Invention

[0005] The interactive method, device, electronic device, and vehicle based on the digital twin model provided in this application realize the visualization of the human-vehicle interaction process through the digital twin model, thereby improving the intuitiveness of the human-vehicle interaction process.

[0006] In a first aspect, embodiments of this application provide an interaction method based on a digital twin model, comprising:

[0007] Construct a digital twin model and visualize it on the vehicle's smart devices; the digital twin model includes a vehicle sub-model and an environment sub-model, with the vehicle sub-model corresponding to the vehicle and the environment sub-model corresponding to the environment in which the vehicle is located;

[0008] Acquire first-type data and / or second-type data; wherein, the first-type data includes vehicle data and environmental data, and the second-type data is multimodal user interaction data;

[0009] Based on the vehicle's large in-vehicle model, the first type of data and / or the second type of data are analyzed and processed to determine the target animation effect of at least one sub-model in the vehicle sub-model and the environment sub-model.

[0010] The digital twin model controlled on a smart device is used to visualize and display the target animation effect.

[0011] In one possible implementation, the vehicle sub-model includes multiple movable parts and corresponding animation effects for the movable parts; the target animation effect of the vehicle sub-model includes at least one target movable part and corresponding animation effects for the target movable part.

[0012] In one possible implementation, based on the vehicle's large onboard model, the second type of data is analyzed and processed to determine the target animation effect of the vehicle sub-model, including:

[0013] Based on the in-vehicle large model, intent analysis and processing are performed on the second type of data to obtain user intent;

[0014] Based on a preset first mapping relationship, at least one target movable part in the vehicle sub-model corresponding to the user's intention and the animation effect corresponding to the target movable part are determined; wherein, the first mapping relationship represents at least one target movable part corresponding to the user's intention and the animation effect corresponding to each target movable part under the user's intention.

[0015] In one possible implementation, the first mapping relationship also represents the control instructions corresponding to each target movable part under the user's intention, and the control instructions are used to control the physical parts in the vehicle corresponding to the target movable parts;

[0016] The method also includes:

[0017] Based on the preset first mapping relationship, determine the control commands corresponding to each target movable part under the user's intention;

[0018] Based on a preset second mapping relationship, the functional interface corresponding to the target movable part is determined; wherein, the second mapping relationship represents the functional interface of the physical part corresponding to the movable part;

[0019] Through the functional interface, control commands are sent to the physical component corresponding to the target movable component to control the physical component to perform actions associated with the animation effect.

[0020] In one possible implementation, based on the vehicle's large onboard model, the first type of data is analyzed and processed to determine the target animation effect for at least one sub-model among the vehicle sub-model and the environment sub-model, including:

[0021] Based on the large vehicle model, the vehicle data in the first type of data is analyzed and processed to obtain at least one target movable part in the vehicle sub-model and the corresponding animation effect of the target movable part; and / or,

[0022] Based on the large vehicle model, the environmental data in the first type of data is analyzed and processed to obtain the target animation effect of the environmental sub-model.

[0023] In one possible implementation, the method further includes, prior to visualizing the target animation effect using a digital twin model controlled on a smart device:

[0024] Based on the first type of data, determine the current driving scenario of the vehicle;

[0025] The target animation effect is validated based on the current driving scenario, and the validation result is obtained.

[0026] If the verification result is determined to be successful, then the step of visualizing the target animation effect on the digital twin model controlled on the smart device will be executed.

[0027] In one possible implementation, the method further includes:

[0028] Based on the target animation effect, generate a prompt message and perform one or more of the following steps:

[0029] Display prompts to users through smart devices;

[0030] The system plays notification messages to the user through the vehicle's speakers.

[0031] In one possible implementation, the method further includes:

[0032] The digital twin model and target animation effects are sent synchronously to the user terminal via a cloud server, so that the user terminal can visualize the digital twin model and the target animation effects on a preset interface.

[0033] Secondly, embodiments of this application provide an interactive device based on a digital twin model, comprising:

[0034] The model building module is used to build digital twin models and visualize them on the vehicle's smart devices. The digital twin model includes a vehicle sub-model and an environment sub-model. The vehicle sub-model corresponds to the vehicle, and the environment sub-model corresponds to the environment in which the vehicle is located.

[0035] The data acquisition module is used to acquire a first type of data and / or a second type of data; wherein the first type of data includes vehicle data and environmental data, and the second type of data is multimodal user interaction data;

[0036] The processing module is used to analyze and process the first type of data and / or the second type of data based on the vehicle's large in-vehicle model, and determine the target animation effect of at least one sub-model in the vehicle sub-model and the environment sub-model.

[0037] The visualization module is used to control the visualization and animation effects of the digital twin model on smart devices.

[0038] In one possible implementation, the vehicle sub-model includes multiple movable parts and corresponding animation effects for the movable parts; the target animation effect of the vehicle sub-model includes at least one target movable part and corresponding animation effects for the target movable part.

[0039] In one possible implementation, based on the vehicle's large onboard model, the second type of data is analyzed and processed to determine the target animation effect of the vehicle sub-model. The processing module is used for:

[0040] Based on the in-vehicle large model, intent analysis and processing are performed on the second type of data to obtain user intent;

[0041] Based on a preset first mapping relationship, at least one target movable part in the vehicle sub-model corresponding to the user's intention and the animation effect corresponding to the target movable part are determined; wherein, the first mapping relationship represents at least one target movable part corresponding to the user's intention and the animation effect corresponding to each target movable part under the user's intention.

[0042] In one possible implementation, the first mapping relationship also represents the control instructions corresponding to each target movable part under the user's intention, and the control instructions are used to control the physical parts in the vehicle corresponding to the target movable parts;

[0043] The processing module is also used for:

[0044] Based on the preset first mapping relationship, determine the control commands corresponding to each target movable part under the user's intention;

[0045] Based on a preset second mapping relationship, the functional interface corresponding to the target movable part is determined; wherein, the second mapping relationship represents the functional interface of the physical part corresponding to the movable part;

[0046] Through the functional interface, control commands are sent to the physical component corresponding to the target movable component to control the physical component to perform actions associated with the animation effect.

[0047] In one possible implementation, based on the vehicle's large onboard model, the first type of data is analyzed and processed to determine the target animation effect for at least one sub-model among the vehicle sub-model and the environment sub-model. The processing module is used for:

[0048] Based on the large vehicle model, the vehicle data in the first type of data is analyzed and processed to obtain at least one target movable part in the vehicle sub-model and the corresponding animation effect of the target movable part; and / or,

[0049] Based on the large vehicle model, the environmental data in the first type of data is analyzed and processed to obtain the target animation effect of the environmental sub-model.

[0050] In one possible implementation, before the digital twin model on the smart device is used to visualize the target animation effect, the processing module is further configured to:

[0051] Based on the first type of data, determine the current driving scenario of the vehicle;

[0052] The target animation effect is validated based on the current driving scenario, and the validation result is obtained.

[0053] If the verification result is determined to be successful, then the step of visualizing the target animation effect on the digital twin model controlled on the smart device will be executed.

[0054] In one possible implementation, the processing module is further configured to:

[0055] Based on the target animation effect, generate a prompt message and perform one or more of the following steps:

[0056] Display prompts to users through smart devices;

[0057] The system plays notification messages to the user through the vehicle's speakers.

[0058] In one possible implementation, the device further includes a transmitting module, which is used to:

[0059] The digital twin model and target animation effects are sent synchronously to the user terminal via a cloud server, so that the user terminal can visualize the digital twin model and the target animation effects on a preset interface.

[0060] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0061] The memory stores instructions that the computer executes;

[0062] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0063] Fourthly, embodiments of this application provide a vehicle, including a vehicle body and electronic devices as described in the third aspect above.

[0064] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0065] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0066] The interactive method, device, electronic device, and vehicle based on a digital twin model provided in this application, by constructing and visually displaying the digital twin model, intuitively present the vehicle and its environment to the user. Through various data sources, the interaction process between the vehicle and the user, and between the user and the vehicle, is realized. Furthermore, based on these various data sources, a target animation effect is determined for at least one sub-model (vehicle sub-model and environment sub-model) within the digital twin model, and the digital twin model is controlled to visually display this target effect, enabling an intuitive display of the two-way interaction process between the vehicle and the user, and between the user and the vehicle. In summary, by replacing traditional voice or text feedback with the ontological expression of the animation effect of the digital twin model, the intuitiveness of the human-vehicle interaction process is improved. Attached Figure Description

[0067] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0068] Figure 1 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 1 ;

[0069] Figure 2 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 2 ;

[0070] Figure 3 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 3 ;

[0071] Figure 4 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 4 ;

[0072] Figure 5 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 5 ;

[0073] Figure 6 A schematic diagram of the structure of the interactive device based on the digital twin model provided in this application;

[0074] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0075] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0077] First, let me explain the terms used in this application:

[0078] Digital twin model: refers to a virtual model constructed through digital technology that is highly consistent with the actual vehicle entity and the external environment. It is used to map the vehicle status and environmental status in real time.

[0079] In-vehicle large model: refers to an artificial intelligence-based model configured in an offline application on an in-vehicle terminal. It is used to process multimodal data and generate intent analysis results.

[0080] Target animation effects refer to the dynamic representation within a digital twin model generated based on data analysis results. Examples include the animated movements of movable parts in a vehicle sub-model, or scene rendering animation in an environment sub-model.

[0081] With the rapid development of vehicle intelligence, the interaction needs between users and vehicles in modern intelligent vehicle systems are becoming increasingly complex. Specifically, vehicles can provide users with feedback on their current vehicle status and environmental information; for example, during driving, vehicles can provide users with fault information, fuel level information, weather information, and road condition information. Alternatively, users can interact with and control the vehicle through interactive operations; for example, users can control the vehicle's windows via voice commands.

[0082] In some embodiments, interaction is implemented through smart devices in the vehicle, either in the form of text or pop-up windows. For example, on the vehicle's central control screen, users can interact via static text pop-ups, and the results of interactive operations can be displayed to users via text information.

[0083] However, the above embodiments lack dynamic visualization of vehicle status and environmental information, making it difficult for users to quickly understand complex information or operational intentions. For example, while driving, users need to pay attention to read the fault information displayed in the text pop-up window on the central control screen. In such a solution, the interaction process between the vehicle and the user is not intuitive; users cannot intuitively understand the vehicle's status, resulting in a poor user experience and posing driving safety risks. As another example, in voice control scenarios, users cannot directly confirm whether the voice control command has been executed, or may have doubts about the accuracy of the vehicle's response, again resulting in unintuitive feedback on the user's interaction with the vehicle.

[0084] In other embodiments, a virtual avatar is displayed on the central control screen, and voice announcements are made through the vehicle's speakers, enabling interactive communication between the user and the virtual avatar and achieving a human-like human-vehicle interaction. However, in the above embodiments, the virtual avatar cannot fully allow the user to intuitively perceive the vehicle's status and environmental information.

[0085] Based on the above scenarios, it can be seen that there is a technical problem in the relevant technologies where the interaction process between users and vehicles is not intuitive.

[0086] The interactive method based on a digital twin model provided in this application constructs and visualizes the digital twin model, enabling an intuitive presentation of the vehicle and its environment to the user. It utilizes various data sources to realize the interaction process between the vehicle and the user, as well as between the user and the vehicle. Furthermore, based on this data, it determines the target animation effect for at least one sub-model (vehicle or environment) within the digital twin model and controls the visualization of this target effect within the digital twin model. This achieves an intuitive display of the two-way interaction process between the vehicle and the user, and between the user and the vehicle. It solves the technical problem of the lack of intuitiveness in the two-way human-vehicle interaction process, thereby improving the intuitiveness of the human-vehicle interaction process.

[0087] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0088] Figure 1 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 1 ,like Figure 1 As shown, the method includes:

[0089] Step 101. Construct a digital twin model and visualize the digital twin model on the vehicle's smart devices.

[0090] The digital twin model includes a vehicle sub-model and an environment sub-model. The vehicle sub-model corresponds to the vehicle, and the environment sub-model corresponds to the environment in which the vehicle is located.

[0091] For example, a digital twin model is constructed that includes both the environment and the vehicle. Specifically, the digital twin model includes a vehicle sub-model and an environment sub-model. It can be understood that the vehicle sub-model and the environment sub-model can be constructed separately and then integrated to form the final digital twin model.

[0092] In constructing the vehicle sub-model, multiple movable parts are first built, and corresponding animation nodes are configured for each movable part. The movable parts are virtual components in the vehicle sub-model that correspond to the physical parts in the vehicle entity.

[0093] It is understandable that movable parts need to completely cover all controllable and displayable vehicle components, including the body, wheels, doors, lights, air vents, seats, and steering wheel. Furthermore, the animation nodes corresponding to the movable parts enable those nodes to execute corresponding animation effects. Animation nodes have node parameters that allow the animation effects of the movable node to synchronize with the physical motion characteristics of the corresponding physical components within the vehicle entity.

[0094] This enables the construction of lightweight vehicle sub-models that correspond one-to-one with the physical components of the vehicle entity.

[0095] In constructing the environmental sub-model, multiple environmental elements are first built. Environmental elements are the elements that may appear in the environment in which the vehicle entity is located, including but not limited to: road elements, weather elements, obstacle elements, etc.

[0096] Each environmental element has a corresponding animation effect. For example, the weather element can present different weather animation effects based on meteorological data; on rainy days, the environmental element will show a rain animation effect, and on foggy or hazy days, the environmental element will show a gray fog effect. Another example is the obstacle element, which is a preset icon with a blinking animation effect.

[0097] Furthermore, the environmental sub-model and vehicle sub-model constructed above are integrated. Specifically, the vehicle sub-model is embedded at the center of the environmental sub-model, ultimately forming a digital twin model.

[0098] The digital twin model obtained above is then visualized on the vehicle's smart devices. For example, the vehicle's smart device could be the vehicle's central control screen, where the digital twin model is visualized.

[0099] Step 102. Obtain the first type of data and / or the second type of data.

[0100] The first category of data includes vehicle data and environmental data, while the second category is multimodal user interaction data.

[0101] For example, the first type of data includes vehicle data and environmental data. Vehicle data includes, but is not limited to, data characterizing the vehicle's state, such as vehicle speed and tire pressure. Environmental data includes, but is not limited to, data characterizing the environmental information of the vehicle's environment, such as road conditions and weather data.

[0102] Based on the first type of data, the vehicle can inform the user of its status and / or the environmental information of the vehicle's surroundings, thereby realizing the interaction process between the vehicle and the user.

[0103] Specifically, vehicle data, which falls under the first category of data, can be acquired through bus transmission using sensors configured on the vehicle itself, or through wireless communication protocols. Environmental data, which also falls under the first category of data, can be acquired through bus transmission using external sensors configured on the vehicle itself.

[0104] For example, the second type of data is user interaction data. User interaction data can be multimodal data, such as user voice, user gestures, and user operations on smart devices.

[0105] The second type of data will be used for explanation. Through this second type of data, users can control the vehicle, thereby realizing the user-vehicle interaction process.

[0106] The second type of data can be collected through microphones installed in the vehicle or through the vehicle's smart devices (such as the central control screen).

[0107] Based on the first and second types of data mentioned above, a two-way interaction process between the vehicle and the user, and between the user and the vehicle, can be realized. Specifically, through one or more of the first and second types of data, and based on a digital twin model, the interaction process can be intuitively realized with the user.

[0108] Step 103. Based on the vehicle's large onboard model, analyze and process the first type of data and / or the second type of data to determine the target animation effect of at least one sub-model among the vehicle sub-model and the environment sub-model.

[0109] For example, the in-vehicle large model can be pre-trained on a cloud server and configured on the vehicle.

[0110] Optionally, based on the initial large language model, it can be trained on a cloud server to enable it to perform intent analysis on user interaction data to obtain user intent; to perform parsing and analysis on vehicle data to identify abnormal physical components within the vehicle entity; and to perform parsing and analysis on environmental data to obtain environmental information about the vehicle's surroundings.

[0111] Furthermore, the trained large model is configured on the vehicle to form an in-vehicle large model.

[0112] In step 103, the data processed by the large vehicle model can be: first type of data, second type of data, and second type of data. Correspondingly, the objects for the target animation effect output by the large vehicle model can be: vehicle sub-model, environment sub-model, vehicle sub-model, and environment sub-model.

[0113] For example, one scenario in step 103 above is: based on the vehicle's large onboard model, the first type of data is analyzed and processed to determine the target animation effects of the vehicle sub-model and the environment sub-model.

[0114] In this case, it is understandable that the target animation effect of the vehicle sub-model can be determined based on vehicle data, and the target animation effect of the environment sub-model can be determined based on environment data.

[0115] For example, one scenario in step 103 above is: based on the vehicle's large onboard model, the first type of data is analyzed and processed to determine the target animation effect of the environmental sub-model.

[0116] In this case, it is understandable that the target animation effect of the environment sub-model can be determined based on the environmental data in the first type of data.

[0117] For example, one scenario in step 103 above is: based on the vehicle's large onboard model, the first type of data is analyzed and processed to determine the target animation effect of the vehicle sub-model.

[0118] In this case, it is understandable that the target animation effect of the vehicle sub-model can be determined based on the vehicle data in the first type of data.

[0119] For example, one scenario in step 103 above is: based on the vehicle's large onboard model, the second type of data is analyzed and processed to determine the target animation effect of the vehicle sub-model.

[0120] In this context, it is understandable that user interaction data for the second type of data can determine the target animation effect of the vehicle sub-model.

[0121] For example, one scenario in step 103 above is: based on the vehicle's large onboard model, the first type of data and the second type of data are analyzed and processed to determine the target animation effects of the vehicle sub-model and the environment sub-model.

[0122] All of the above-mentioned different situations can be implemented in step 103. For example, the user interacts with the vehicle via voice, saying, "If it's raining outside, close the sunroof and turn on the windshield wipers." This user voice is considered the second type of data. At the same time, the vehicle acquires environmental data from the first type of data, detects the rainfall in the vehicle's environment, and if the rainfall exceeds a preset threshold, it is determined to be a rainy day, i.e., it is raining outside.

[0123] Furthermore, the target animation effects for the vehicle sub-model can be determined as: performing a sunroof closing animation and a windshield wiper opening animation. The target animation effect for the environment sub-model can be determined as: performing a rainy weather animation for the weather elements.

[0124] Step 104. Visualize the target animation effect of the digital twin model on the smart device.

[0125] For example, based on the determined target animation effect, the digital twin model displayed on the smart device is controlled to execute the aforementioned target animation effect, thereby enabling intuitive interaction with the user.

[0126] It is understood that the aforementioned target animation effect can be the animation effect corresponding to at least one of the vehicle sub-model and environment sub-model in the digital twin model.

[0127] In this embodiment, when constructing the digital twin model, the physical components of the vehicle entity (such as doors, windows, and lights) and environmental elements (such as roads, weather, and obstacles) are first mapped to movable components and environmental elements in the virtual model. Each movable component is configured with a corresponding animation node, and the node parameters are synchronized with the physical motion characteristics of the physical component to ensure that the digital twin model is consistent with the real vehicle state. When user interaction data (such as voice commands) or vehicle data (such as abnormal tire pressure) triggers the target animation effect, the system uses the visualization of the digital twin model to allow users to intuitively perceive the vehicle state or interaction results without text or voice feedback. This achieves intuitive interaction where what you see is what you control.

[0128] The interaction method based on a digital twin model provided in this application improves the intuitiveness and real-time performance of human-vehicle interaction by constructing a digital twin model and dynamically mapping vehicle state and environmental information. Specifically, through the vehicle sub-model in the digital twin model, the state and actions of the components corresponding to the vehicle entity can be visualized in real time and intuitively; through the environment sub-model in the digital twin model, the environmental information of the environment in which the vehicle entity is located can be visualized in real time and intuitively.

[0129] Furthermore, by combining user interaction data and analyzing the intent of the large in-vehicle model, the results are transformed into target animation effects for the movable parts of the vehicle sub-model within the digital twin model. Through the dynamic feedback of the virtual model, users can intuitively experience the results and responses of interactive operations without relying on text information or voice prompts during human-vehicle interaction. In summary, this addresses to some extent the problems of simplistic presentation of interactive information and the disconnect between user intent and vehicle interaction response, thereby improving the intuitiveness and real-time nature of the human-vehicle interaction process.

[0130] As can be seen from practical applications, a vehicle entity contains multiple physical components. As explained in the foregoing embodiments, the vehicle sub-model in the digital twin model corresponds to the vehicle itself. Therefore, the vehicle sub-model contains components that correspond one-to-one with the vehicle entity.

[0131] In one example, the vehicle sub-model includes multiple movable parts and corresponding animation effects for the movable parts; the target animation effect of the vehicle sub-model includes at least one target movable part and corresponding animation effects for the target movable part.

[0132] For example, as can be seen from the foregoing embodiments, the vehicle sub-model in the digital twin model corresponds to the vehicle entity. It can be understood that the vehicle sub-model includes multiple movable parts. Each movable part is a virtual component corresponding to a physical part of the vehicle entity.

[0133] Each movable part in the vehicle sub-model has a corresponding animation effect, which is implemented based on the animation nodes configured for the movable part. In other words, the animation effect refers to the dynamic behavior of the movable part in a specific state, such as a door sliding open or a window raising or lowering.

[0134] Furthermore, based on the structure of the vehicle sub-model described above, the target animation effect for the vehicle sub-model may include at least one target movable part and the corresponding animation effect for each target movable part.

[0135] In a digital twin model, the vehicle sub-model is divided into multiple movable parts. Each movable part has a preset animation effect. For example, a movable part could be a door or a sunroof, and the animation effect could be sliding or rotating. When generating a target animation effect, the system first identifies the movable parts that need to be dynamically displayed and then calls the corresponding animation effect.

[0136] For example, the system first recognizes the user's request to open the sunroof, the target movable part is the vehicle's sunroof, and calls the sunroof's sliding opening animation effect to finally form the target animation effect.

[0137] In this embodiment, each movable component is bound to a corresponding animation node. The node parameters enable the animation of the movable component to match the physical behavior of the physical component. For example, the opening and closing angle of the car door is parametrically controlled through the node parameters of the animation node.

[0138] Optionally, onboard sensors can be configured on the physical components corresponding to the movable parts to collect real-time physical motion data of the physical components. The animation nodes configure node parameters based on the sensor data to adjust the animation effects of the movable parts in real time, thereby achieving consistency between the vehicle sub-model in the digital twin model and the actual vehicle state, and avoiding lag or distortion in the animation effects.

[0139] In the above example, a vehicle sub-model is constructed based on movable parts that correspond one-to-one with the physical components of the vehicle entity. This enables a one-to-one mapping between the vehicle sub-model and the vehicle entity in the digital twin model. Furthermore, for this vehicle sub-model, the target animation effect not only includes the target movable parts and their corresponding animation effects, but also further refines the mapping relationship between movable parts and animation effects. This avoids the problem of unintuitive feedback in human-vehicle interaction in related technologies. For the actions of the physical components of the vehicle entity, corresponding animation effects are executed on the target movable parts in the vehicle sub-model, allowing users to more intuitively perceive the operation results, thereby improving the intuitiveness and accuracy of the interaction.

[0140] Figure 2 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 2 In this embodiment Figure 1 Based on the illustrated embodiment, one scenario in step 103 above will be described in detail.

[0141] In one possible scenario, based on user interaction data, it is possible to enable the movable parts of the vehicle sub-model in the digital twin model to perform corresponding target animation effects. For example... Figure 2 As shown, the method includes:

[0142] Step 201. Based on the in-vehicle large model, perform intent analysis processing on the second type of data to obtain user intent.

[0143] Step 202. Based on the preset first mapping relationship, determine at least one target movable part in the vehicle sub-model corresponding to the user's intention, and the animation effect corresponding to the target movable part.

[0144] The first mapping relationship represents at least one target movable part corresponding to the user's intention, and the animation effect corresponding to each target movable part under the user's intention.

[0145] For example, the second type of data is user interaction data. As can be seen from the foregoing embodiments, user interaction data can be multimodal data. For instance, user voice. User voice is collected through microphones inside the vehicle and serves as the second type of data.

[0146] Based on the in-vehicle large model, intent analysis processing is performed on the aforementioned user voice. Intent analysis processing refers to parsing the semantics of user voice through Natural Language Processing (NLP) or machine learning algorithms, and determining the user's intent based on the semantic information.

[0147] Subsequently, based on the preset first mapping relationship, at least one target movable part corresponding to the user's intention is matched, and the animation effect corresponding to each of the at least one target movable parts under the user's intention is obtained, thereby obtaining the target animation effect of the vehicle sub-model.

[0148] In a specific embodiment, taking the cabin temperature adjustment scenario as an example, the user inputs "I'm cold" via voice. Based on the in-vehicle big data model, the user's voice in the second type of data above is used for intent recognition, and the user's intent is "to increase the cabin temperature and turn on the driver's seat heating".

[0149] Based on the first mapping relationship, the target movable parts corresponding to the aforementioned user intent are determined to be the vehicle air conditioning vents, the vehicle body, and the driver's seat. The animation effect corresponding to the vehicle air conditioning vents under this user intent is: dynamic characteristics of warm airflow; the animation effect corresponding to the vehicle body under this user intent is: a closing, embracing posture towards the driver's seat; the animation effect corresponding to the driver's seat under this user intent is: a warm light gradient effect.

[0150] The second type of data mentioned above, in addition to user interaction data in the form of voice, can also be user operation data on the vehicle's smart devices.

[0151] In a specific embodiment, taking a zero-level component touch interaction scenario as an example, a user clicks on the left front door component of the vehicle sub-model in the digital twin model on the central control screen. In response to the user's click, a floating control panel pops up at the clicked location on the central control screen. This floating control panel contains various function buttons, such as door locks, child locks, window lifts, and rearview mirrors. In response to the user's click or swipe operation on the rearview mirror function button, the large vehicle model recognizes the user's intent as "adjust the angle of the left rearview mirror."

[0152] Based on the first mapping relationship, the target movable part corresponding to the above user intent is determined to be the left rearview mirror. The animation effect of the left rearview mirror under this user intent is: rotation at the corresponding angle.

[0153] It is understandable that when the second type of data contains user voice, intent analysis can be achieved through natural language processing technology. When the second type of data includes user gestures or touch operations, the user's operational intent can be identified through computer vision algorithms. For example, when a user clicks on a window component on the central control screen, the system uses computer vision to identify the click location and matches it with the corresponding functional interface (such as the window control interface), thus achieving unified processing of multimodal data.

[0154] In the above embodiments, the intent analysis function of the in-vehicle large model is used to determine the user intent in the user interaction data. Based on the mapping relationship, the target movable parts in the vehicle sub-model corresponding to the user intent and the corresponding animation effects of the target movable parts are determined. This avoids the problem of disconnect between user intent and human-vehicle interaction response. The vehicle sub-model can directly and intuitively visualize the execution response of the user intent, improving the real-time performance and intuitiveness of the interaction. In addition, users do not need to confirm the execution response of their intent through static text or voice broadcast. The intuitive visualization of the target animation effects based on the vehicle sub-model avoids distraction during driving, thereby improving driving safety.

[0155] It should be noted that the above embodiments illustrate one scenario and do not represent a limitation on the actual application of this application. On the contrary, this application may have many other scenarios in practical applications.

[0156] In the foregoing Figure 2 Based on the illustrated embodiment, for user interaction data, the vehicle entity can be controlled to perform the action corresponding to the user intent indicated by the user interaction data.

[0157] In one possible implementation, based on the above embodiments, the first mapping relationship also represents the control instructions corresponding to each target movable part under the user's intention, and the control instructions are used to control the physical parts in the vehicle corresponding to the target movable parts.

[0158] For example, in the first mapping relationship, in addition to the movable part corresponding to the user's intention and the animation effect corresponding to the movable part, it may also include the control instruction corresponding to the movable part under the user's intention. The control instruction can be used to control the entity component in the vehicle entity corresponding to the movable part.

[0159] It can be understood that control commands are instructions that drive physical components of a vehicle to perform specific actions, such as "open the sunroof".

[0160] Figure 3 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 3 Based on this, such as Figure 3 As shown, the method also includes:

[0161] Step 301. Based on a preset first mapping relationship, determine the control instructions corresponding to each target movable part under the user intention.

[0162] Step 302. Based on a preset second mapping relationship, determine the function interfaces corresponding to the target movable parts.

[0163] Among them, the second mapping relationship represents the function interfaces of the entity parts corresponding to the movable parts.

[0164] Exemplarily, combining the foregoing exemplary description, it can be known that for the first mapping relationship, there are also control instructions corresponding to the movable parts under this user intention. Therefore, combining the user intention determined by the foregoing vehicle-mounted large model, based on the preset first mapping relationship, it is also possible to determine the control instructions of each corresponding target movable part under this user intention.

[0165] Furthermore, based on the second mapping relationship, determine the function interfaces corresponding to the target movable parts. Among them, the second mapping relationship includes the function interfaces of the entity parts corresponding to each movable part in the vehicle entity in the vehicle sub-model. The function interface refers to the communication interface of the vehicle entity part, such as a CAN bus interface, or the target address of a communication protocol.

[0166] Optionally, in the second mapping relationship, there is also an identifier (ID) for each movable part. Based on the determined user intention, in the first mapping relationship, determine the target movable parts corresponding to the user intention, as well as the corresponding animation effects and control instructions. Based on the ID of the target movable part, determine the function interface corresponding to the target movable part in the second mapping relationship.

[0167] Step 303. Through the function interface, send the control instruction to the entity part corresponding to the target movable part to control the entity part to perform an action associated with the animation effect.

[0168] Exemplarily, continuing with the foregoing example, based on the control instruction of the target movable part, through the corresponding function interface, send the control instruction to the entity part corresponding to the target movable part. Thus, it is possible to control the entity part in the vehicle entity to perform an action associated with the animation effect.

[0169] Illustrated with the foregoing embodiment, in the above specific embodiment taking the cockpit temperature adjustment scenario as an example, it may further include: after determining that the user intention is "increase the cockpit temperature and turn on the driver's seat heating", synchronously generate control instructions of "adjust the air conditioner temperature to 22°C and turn on the driver's seat heating at level 2", and send them through the function interfaces corresponding to the air conditioner and the driver's seat in the vehicle entity respectively, so as to implement the actions of adjusting the air conditioner temperature and turning on the heating of the driver's seat in the vehicle entity. Achieve linkage control with the animation effect of the vehicle sub-model.

[0170] Optionally, the above control commands can be executed synchronously with the target animation effect via hardware timestamp binding.

[0171] Optionally, after the physical component responds to the control command and completes the action indicated by the control command, the vehicle sub-model returns to its basic posture.

[0172] In another specific embodiment, in a user interaction scenario, intent analysis processing is performed on the second type of data (such as user voice, gestures, or touch operations) based on the in-vehicle large model.

[0173] For example, when a user gives the voice command "close the sunroof", the in-vehicle large model analyzes the voice content and recognizes the user's intention to "close the sunroof".

[0174] Subsequently, based on the preset first mapping relationship, the target movable part (such as sunroof) and the corresponding animation effect (such as sunroof closing action) in the vehicle sub-model corresponding to the user's intention are determined.

[0175] Meanwhile, the first mapping relationship also includes the control command corresponding to the user's intention (such as "sunroof closed"), and the functional interface (such as CAN bus interface) corresponding to the target movable part is determined through the preset second mapping relationship.

[0176] Finally, control commands are sent to the physical components (such as the sunroof motor) in the vehicle corresponding to the target movable part through this functional interface, so as to realize the synchronous linkage between the sunroof closing action and the animation effect.

[0177] In the above embodiments, based on the mapping between user intent and target animation effect, control commands are further used to control physical components to synchronously execute actions associated with the animation effect, thereby achieving linkage between the vehicle sub-model and the vehicle entity in the digital twin model. This improves the consistency of the interactive experience and the efficiency of user confirmation of operation results.

[0178] This embodiment is in Figure 1 Based on the illustrated embodiment, another case in step 103 above will be described in detail.

[0179] In one possible scenario, based on the vehicle data and environmental data in the first type of data, at least one of the vehicle sub-models and the environment sub-model in the digital twin model can perform the corresponding target animation effect.

[0180] In one example, based on a large vehicle model, the vehicle data in the first type of data is analyzed and processed to obtain at least one target movable part in the vehicle sub-model and the animation effect corresponding to the target movable part.

[0181] For example, vehicle data is vehicle-related data used to characterize a vehicle entity, such as vehicle speed or tire pressure.

[0182] Taking vehicle data, including tire pressure, as an example, by acquiring the tire pressure data of the four tires of the vehicle entity, and performing analysis and processing based on the vehicle large model, if it is determined that the tire pressure data of one or more tires is abnormal, then the movable part corresponding to the abnormal tire is identified as the target movable part.

[0183] Furthermore, when the tire pressure data is abnormal, the animation effect corresponding to the target movable part is a continuous red highlight flashing.

[0184] In one specific embodiment, the vehicle tire pressure monitoring system detects an abnormal pressure in the left rear tire, which the system determines as a highest priority safety alarm. The left rear wheel is identified as a target movable part in the vehicle sub-model, and the corresponding animation effect is determined to be a continuous red highlight flashing.

[0185] In practical applications, abnormal tire pressure can pose a safety risk to vehicle operation; therefore, the above scenario represents the highest priority safety warning. Optionally, the target movable part, in addition to the left rear wheel, can also include the vehicle body. The corresponding animation effect for the vehicle body is a slight warning vibration.

[0186] Optionally, after the tire pressure abnormality is resolved, the warning effects of the vehicle sub-model are turned off, and the vehicle returns to its basic posture.

[0187] In one example, based on the large vehicle model, the environmental data in the first type of data is analyzed and processed to obtain the target animation effect of the environmental sub-model.

[0188] For example, environmental data is relevant data used to characterize the environment in which a vehicle is located, such as weather conditions.

[0189] Taking environmental data, including weather, as an example, data from external weather sensors is acquired and analyzed based on an onboard large model to obtain the weather information of the current external environment of the vehicle. If the weather information is determined to be rainy, the target animation effect of the environmental sub-model is determined to be a rainy day animation; if the weather information is determined to be foggy or hazy, the target animation effect of the environmental sub-model is determined to be a gray smoke floating animation.

[0190] Optionally, the data from the meteorological sensor may include rainfall, and the target animation effect in the environmental sub-model can switch the level of the rain animation based on the magnitude of the rainfall. For example, if the rainfall is in the first range, the rain animation is light rain; if the rainfall is in the second range, the rain animation is moderate rain; and if the rainfall is in the third range, the rain animation is heavy rain.

[0191] It can be understood that the lower limit of the third numerical range is greater than or equal to the upper limit of the second numerical range; and the lower limit of the second numerical range is greater than or equal to the upper limit of the first numerical range.

[0192] Taking environmental data, including road data, as an example, road data drives the dynamic changes of lane lines in the environmental sub-model.

[0193] It should be noted that the two examples above can be implemented individually or in combination.

[0194] In the above embodiments, by analyzing and processing the vehicle data and environmental data in the first type of data of the large model, the target animation effect of at least one sub-model in the digital twin model can be determined. Based on this, it can be ensured that during the interaction between the vehicle and the user, no text information or voice broadcast is required, and the interaction is achieved through intuitive animation effects, further improving the intuitiveness of the human-vehicle interaction process.

[0195] In addition, it helps users understand vehicle status and environmental information in a timely manner through intuitive animation effects, thereby improving vehicle driving safety.

[0196] Based on any of the foregoing embodiments, the driving scenario of the vehicle can be determined based on vehicle data, and the target animation effect can be verified based on the driving scenario.

[0197] Figure 4 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 4 .like Figure 4 As shown, before step 104, which controls the visualization of the target animation effect on the digital twin model on the smart device, the method further includes:

[0198] Step 401. Determine the current driving scenario of the vehicle based on the first type of data.

[0199] Step 402. Verify the target animation effect based on the current driving scenario and obtain the verification result.

[0200] Step 403. If the verification result is determined to be successful, then execute the step of visualizing the target animation effect of the digital twin model on the smart device.

[0201] For example, the current driving scenario of the vehicle is determined based on the first type of data. Here, the current driving scenario refers to the driving environment state dynamically determined based on both vehicle data and environmental data.

[0202] For example, based on vehicle speed data and weather data from environmental data, the current driving scenario can be determined to be highway driving in rainy weather.

[0203] Furthermore, the target animation effect is validated based on the current driving scenario to obtain the validation result. Specifically, the reasonableness of the target animation effect can be validated based on preset rules in the current driving scenario. For example: windows are prohibited from being opened while driving at high speed. If the target animation effect meets the preset rules in the current driving scenario, the validation result is determined to be passed; otherwise, the validation result is determined to be failed.

[0204] Only if the verification result is confirmed to be successful will the step of visualizing the target animation effect of the digital twin model on the control smart device be executed.

[0205] It's understandable that if a user requests to open a car window while driving at high speed in rainy weather, the above verification steps would fail, indicating the user's intended action is unreasonable. The target movable part (the window) in the vehicle sub-model will not execute the target animation effect (the window opening animation).

[0206] Synchronously, since the verification result is unsuccessful, the system will not send control commands to the physical component corresponding to the target movable part, and the vehicle's windows will not open, thus improving the security of the interaction.

[0207] Optionally, if the verification result is unsuccessful, only the user's intent is recorded and a warning message is generated, such as "Operation is prohibited in the current driving scenario," without executing the animated visualization or sending control commands.

[0208] In the above embodiments, the driving scenario is determined using vehicle data from the first type of data. Based on a scenario-based verification mechanism, the target animation effect is verified. The target animation effect is only executed when the verification result passes, reducing the risk of conflict between the target animation effect and the driving scenario.

[0209] Figure 5 A flowchart illustrating the interaction method based on a digital twin model provided in this application. Figure 5 Based on any of the foregoing embodiments, such as Figure 5 As shown, the method also includes:

[0210] Step 501. Generate prompt information based on the target animation effect.

[0211] For example, a prompt message is generated to provide feedback to the user based on the target animation effect. For instance, if the target animation effect is a sunroof opening, the prompt message is "Sunroof is open". If the target animation effect is a dynamic animation of warm airflow from an air conditioner vent, the prompt message is "Air conditioner temperature has been increased".

[0212] Step 502. Perform one or more of the following steps: display a prompt message to the user via a smart device; play a prompt message to the user via a speaker in the vehicle.

[0213] In one example, based on the prompt information, the prompt information corresponding to the above-mentioned target animation effect is displayed visually on the smart device in the form of text or pop-up window.

[0214] In another example, a voice prompt is generated based on the prompt information, using a preset tone and intonation; this voice prompt is then played through the vehicle's speakers. Specifically, text-to-speech (TTS) technology converts the text-based prompt information into a voice prompt, which is then played through the vehicle's speakers. The TTS engine supports multiple languages ​​and preset tones, ensuring the naturalness and comprehensibility of the voice prompts.

[0215] The two examples described above can be implemented individually or in combination to achieve multimodal feedback. This ensures that users can perceive the results of their actions in different interaction scenarios (such as when driving and unable to view the screen), enhancing the intuitiveness and comprehensiveness of the interaction.

[0216] In conjunction with the foregoing embodiments, in the specific embodiment of the cabin temperature adjustment scenario, after visually displaying the target animation effects of movable components such as the air conditioning vents, the vehicle body, and the driver's seat, and after adjusting the air conditioning temperature and activating the driver's seat heating, the following prompt message can be generated: "The temperature has been adjusted for you. We hope you have a warm journey."

[0217] Optionally, the above prompts can be displayed in a pop-up window on a smart device, and a corresponding voice prompt can be generated and played through the vehicle's speakers at a preset volume.

[0218] In the above embodiments, based on the intuitive interactive method of the target animation effect of the digital twin model, the display method of text-based prompts and the broadcast method of voice-based prompts are retained. Through multimodal user interaction methods using prompts, the comprehensiveness of the human-vehicle interaction process is improved, ensuring the comprehensiveness of user confirmation of interactive responses.

[0219] As can be seen from the foregoing embodiments, the interaction method based on a digital twin model provided in this application is applied to the vehicle end. In some possible implementations, the vehicle end communicates with a cloud server, and the cloud server communicates with a user terminal. Based on this system architecture, synchronization of the digital twin models of the vehicle, cloud server, and user terminal can be achieved.

[0220] Based on any of the foregoing embodiments, in one example, the method further includes:

[0221] The digital twin model and target animation effects are sent synchronously to the user terminal via a cloud server, so that the user terminal can visualize the digital twin model and the target animation effects on a preset interface.

[0222] For example, a cloud server refers to a remote server used for storing and synchronizing data, and a user terminal refers to a device used by a user, such as a mobile phone.

[0223] The digital twin model constructed above, along with the target animation effects of at least one sub-model within the digital twin model, are packaged into a synchronization data packet and sent to the cloud server.

[0224] The cloud server forwards the data to the user's terminal, such as a mobile phone. This allows the user to view the digital twin model and the corresponding target animation effects on the digital twin model in real time through a preset interface on the application, enabling the user to remotely perceive the status of the vehicle entity and the environment in which the vehicle is located.

[0225] It is understandable that users access a preset interface through a mobile application to view the animation effects of the vehicle sub-model in real time (such as door opening and seat heating). This synchronization mechanism enables real-time data transmission through a cloud server, allowing users to perceive the vehicle status and interaction results in remote scenarios, thus expanding the coverage and convenience of human-vehicle interaction.

[0226] Optionally, the user terminal parses and displays the digital twin model and target animation effects through a local rendering engine, ensuring smooth operation even on low-power devices. The local rendering engine can be Unity or WebGL.

[0227] In the example above, a cloud server can synchronize the digital twin model and target animation effects to the user's terminal, allowing the user to remotely view the vehicle model corresponding to the actual vehicle and the animation effects of each movable part within the vehicle model via their mobile phone. This improves the scalability of the human-vehicle interaction process and enhances the user's remote perception of the vehicle entity, thereby increasing the vehicle's intelligence.

[0228] The interaction method based on a digital twin model provided in this application improves the intuitiveness and real-time performance of human-vehicle interaction by constructing a digital twin model and dynamically mapping vehicle state and environmental information. Through the vehicle sub-model in the digital twin model, the state and actions of the components corresponding to the vehicle entity can be visualized in real time and intuitively; through the environment sub-model in the digital twin model, the environmental information of the environment in which the vehicle entity is located can be visualized in real time and intuitively.

[0229] Furthermore, by combining user interaction data and analyzing the intent of the large vehicle model, the results are transformed into target animation effects of movable parts of the vehicle sub-model in the digital twin model. This allows users to intuitively experience the results and responses of interactive operations without relying on text information or voice prompts during human-vehicle interaction.

[0230] By leveraging the intent analysis capabilities of the in-vehicle large model, user intents within user interaction data are identified. Based on mapping relationships, the target movable parts within the vehicle sub-model corresponding to the user intent, along with their corresponding animation effects, are determined. This allows for a direct and intuitive visualization of the execution response to user intents, enhancing the real-time nature and intuitiveness of the interaction.

[0231] By controlling physical components to synchronously execute actions associated with animation effects through control commands, the linkage between the vehicle sub-model and the actual vehicle in the digital twin model is achieved. This improves the consistency of the interactive experience and the efficiency of user confirmation of operation results.

[0232] By analyzing and processing vehicle and environmental data from the first category of data in the large model, it is possible to ensure that vehicle-to-user interactions are achieved through intuitive animation effects, eliminating the need for text messages or voice announcements, thus further enhancing the intuitiveness of the human-vehicle interaction process. Additionally, timely understanding of vehicle status and environmental information improves driving safety.

[0233] Cloud servers can synchronize digital twin models and target animation effects to user terminals, allowing users to remotely view the vehicle model corresponding to the actual vehicle and the animation effects of its movable parts via their mobile phones. This enhances the scalability of human-vehicle interaction and improves the user's remote perception of the vehicle, thereby increasing the vehicle's intelligence.

[0234] Figure 6 A schematic diagram of the structure of the interactive device based on the digital twin model provided in this application is shown below. Figure 6 As shown, the interactive device 60 based on a digital twin model provided in this embodiment includes:

[0235] The model building module 601 is used to build a digital twin model and visualize the digital twin model on the vehicle's intelligent devices; wherein, the digital twin model includes a vehicle sub-model and an environment sub-model, the vehicle sub-model corresponds to the vehicle, and the environment sub-model corresponds to the environment in which the vehicle is located;

[0236] The data acquisition module 602 is used to acquire a first type of data and / or a second type of data; wherein the first type of data includes vehicle data and environmental data, and the second type of data is multimodal user interaction data;

[0237] The processing module 603 is used to analyze and process the first type of data and / or the second type of data based on the vehicle's large vehicle model, and determine the target animation effect of at least one sub-model in the vehicle sub-model and the environment sub-model.

[0238] The visualization module 604 is used to control the visualization display of the target animation effect of the digital twin model on the smart device.

[0239] In one possible implementation, the vehicle sub-model includes multiple movable parts and corresponding animation effects for the movable parts; the target animation effect of the vehicle sub-model includes at least one target movable part and corresponding animation effects for the target movable part.

[0240] In one possible implementation, based on the vehicle's large onboard model, the second type of data is analyzed and processed to determine the target animation effect of the vehicle sub-model. The processing module 603 is used for:

[0241] Based on the in-vehicle large model, intent analysis and processing are performed on the second type of data to obtain user intent;

[0242] Based on a preset first mapping relationship, at least one target movable part in the vehicle sub-model corresponding to the user's intention and the animation effect corresponding to the target movable part are determined; wherein, the first mapping relationship represents at least one target movable part corresponding to the user's intention and the animation effect corresponding to each target movable part under the user's intention.

[0243] In one possible implementation, the first mapping relationship also represents the control instructions corresponding to each target movable part under the user's intention, and the control instructions are used to control the physical parts in the vehicle corresponding to the target movable parts;

[0244] Processing module 603 is also used for:

[0245] Based on the preset first mapping relationship, determine the control commands corresponding to each target movable part under the user's intention;

[0246] Based on a preset second mapping relationship, the functional interface corresponding to the target movable part is determined; wherein, the second mapping relationship represents the functional interface of the physical part corresponding to the movable part;

[0247] Through the functional interface, control commands are sent to the physical component corresponding to the target movable component to control the physical component to perform actions associated with the animation effect.

[0248] In one possible implementation, based on the vehicle's large onboard model, the first type of data is analyzed and processed to determine the target animation effect for at least one sub-model among the vehicle sub-model and the environment sub-model. The processing module 603 is used for:

[0249] Based on the large vehicle model, the vehicle data in the first type of data is analyzed and processed to obtain at least one target movable part in the vehicle sub-model and the corresponding animation effect of the target movable part; and / or,

[0250] Based on the large vehicle model, the environmental data in the first type of data is analyzed and processed to obtain the target animation effect of the environmental sub-model.

[0251] In one possible implementation, before the digital twin model on the smart device is used to visualize the target animation effect, the processing module 603 is further configured to:

[0252] Based on the first type of data, determine the current driving scenario of the vehicle;

[0253] The target animation effect is validated based on the current driving scenario, and the validation result is obtained.

[0254] If the verification result is determined to be successful, then the step of visualizing the target animation effect on the digital twin model controlled on the smart device will be executed.

[0255] In one possible implementation, the processing module 603 is further configured to:

[0256] Based on the target animation effect, generate a prompt message and perform one or more of the following steps:

[0257] Display prompts to users through smart devices;

[0258] The system plays notification messages to the user through the vehicle's speakers.

[0259] In one possible implementation, the device further includes a transmitting module, which is used to:

[0260] The digital twin model and target animation effects are sent synchronously to the user terminal via a cloud server, so that the user terminal can visualize the digital twin model and the target animation effects on a preset interface.

[0261] It should be noted that some modules in the aforementioned interactive devices based on digital twin models are not included. Figure 6 As shown in the diagram. In practical applications, this module (not shown) can be related to... Figure 6 The diagram shows independent modules with modular connections, which can also be integrated into... Figure 6 The modules already shown in the diagram.

[0262] The interactive device based on the digital twin model provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0263] Figure 7A schematic diagram of the structure of the electronic device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the electronic device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0264] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0265] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0266] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0267] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0268] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0269] This application also provides a vehicle, including a vehicle body and an electronic device. The electronic device is used to execute the method provided in the above-described method embodiments, and its implementation and technical effects can be referred to the foregoing method embodiments.

[0270] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0271] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0272] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0273] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0274] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0275] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0276] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0277] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0278] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0279] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An interaction method based on a digital twin model, characterized in that, include: A digital twin model is constructed and the digital twin model is visualized on the vehicle's intelligent devices; wherein, the digital twin model includes a vehicle sub-model and an environment sub-model, the vehicle sub-model corresponding to the vehicle and the environment sub-model corresponding to the environment in which the vehicle is located; Acquire a first type of data and / or a second type of data; wherein the first type of data includes vehicle data and environmental data, and the second type of data is multimodal user interaction data; Based on the vehicle's large in-vehicle model, the first type of data and / or the second type of data are analyzed and processed to determine the target animation effect of at least one sub-model among the vehicle sub-model and the environment sub-model. The digital twin model controlled on the smart device visualizes and displays the target animation effect.

2. The method of claim 1, wherein, The vehicle sub-model includes multiple movable parts and corresponding animation effects for the movable parts; the target animation effect of the vehicle sub-model includes at least one target movable part and corresponding animation effect for the target movable part.

3. The method of claim 2, wherein, Based on the vehicle's large in-vehicle model, the second type of data is analyzed and processed to determine the target animation effect of the vehicle sub-model, including: Based on the aforementioned vehicle-mounted large model, intent analysis processing is performed on the second type of data to obtain user intent; Based on a preset first mapping relationship, at least one target movable part in the vehicle sub-model corresponding to the user intention and the animation effect corresponding to the target movable part are determined; wherein, the first mapping relationship represents at least one target movable part corresponding to the user intention and the animation effect corresponding to each target movable part under the user intention.

4. The method of claim 3, wherein, The first mapping relationship also represents the control instructions corresponding to each target movable part under the user's intention, and the control instructions are used to control the physical parts in the vehicle corresponding to the target movable parts; The method further includes: Based on a preset first mapping relationship, the control commands corresponding to each target movable part under the user's intention are determined; Based on a preset second mapping relationship, the functional interface corresponding to the target movable part is determined; wherein, the second mapping relationship represents the functional interface of the physical part corresponding to the movable part; Through the aforementioned functional interface, control commands are sent to the physical component corresponding to the target movable component to control the physical component to perform actions associated with the animation effect.

5. The method of claim 2, wherein, Based on the vehicle's large in-vehicle model, the first type of data is analyzed and processed to determine the target animation effect of at least one sub-model among the vehicle sub-model and the environment sub-model, including: Based on the aforementioned large vehicle model, the vehicle data in the first type of data is analyzed and processed to obtain at least one target movable part in the vehicle sub-model and the animation effect corresponding to the target movable part; and / or, Based on the large vehicle model, the environmental data in the first type of data is analyzed and processed to obtain the target animation effect of the environmental sub-model.

6. The method according to any one of claims 1-5, characterized in that, Before visualizing the target animation effect using a digital twin model controlled on the smart device, the method further includes: Based on the first type of data, determine the current driving scenario of the vehicle; The target animation effect is verified based on the current driving scenario to obtain the verification result; If the verification result is determined to be successful, then the step of visually displaying the target animation effect using the digital twin model on the smart device is executed.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: Based on the target animation effect, generate a prompt message and perform one or more of the following steps: The prompt information is displayed to the user through the smart device; The notification message is played to the user through the vehicle's speakers.

8. The method according to any one of claims 1-5, characterized in that, The method further includes: The digital twin model and the target animation effect are simultaneously sent to the user terminal via a cloud server, so that the user terminal can visualize the digital twin model and the target animation effect on a preset interface.

9. An interactive device based on a digital twin model, characterized in that, include: The model building module is used to build a digital twin model and visualize the digital twin model on the vehicle's smart devices; wherein, the digital twin model includes a vehicle sub-model and an environment sub-model, the vehicle sub-model corresponding to the vehicle, and the environment sub-model corresponding to the environment in which the vehicle is located; The data acquisition module is used to acquire a first type of data and / or a second type of data; wherein the first type of data includes vehicle data and environmental data, and the second type of data is multimodal user interaction data; The processing module is used to analyze and process the first type of data and / or the second type of data based on the vehicle's large on-board model, and determine the target animation effect of at least one sub-model among the vehicle sub-model and the environment sub-model. The visualization module is used to control the visualization of the target animation effect by the digital twin model on the smart device.

10. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

11. A vehicle, characterized in that, Includes the vehicle body and the electronic equipment as described in claim 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-8.

Images