Light application generation method and device, vehicle and electronic equipment

Abstract

The invention relates to a light application generation method and device, a vehicle and electronic equipment, and relates to the technical field of vehicles, in particular to the technical field of the Internet of Vehicles, and the method comprises the steps: collecting multi-modal data of passengers in the vehicle; the multi-modal data and the vehicle configuration information are input into a light application generation type large model, the use intention of a passenger on the vehicle-mounted service in the light application is recognized through the light application generation type large model, and a DSL script of the light application is generated; based on the DSL script, an interface of the light application is displayed on a vehicle-mounted terminal interface, and the interface comprises the configuration condition of at least one vehicle-mounted service in a vehicle-mounted service set covered by the light application; the configuration condition is matched with the use intention. According to the method, the DSL script of the light application can be temporarily generated and the interface of the light application can be regenerated through the multi-modal data collected in real time, so that the light application interface which is matched with the use intention of a user and comprises a plurality of vehicle-mounted services is quickly generated and related configuration options are completed, and immersive vehicle experience meeting the user intention in real time is provided.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more particularly to the field of vehicle networking technology, specifically to a method, apparatus, vehicle, and electronic device for generating lightweight applications. Background Technology

[0002] With the rapid iterative development of artificial intelligence, sensor technology, and smart cockpits, in-vehicle multimodal interaction and generative artificial intelligence are key technologies on the road to intelligent vehicle systems. Traditional in-vehicle interaction systems are mainly based on touch and voice, which can cover most usage scenarios, but this interaction mode still suffers from low operating efficiency and poor scenario adaptability. Traditional vehicle infotainment interfaces are either pre-developed applications or user-defined orchestration scenarios. When faced with new functions and scenario requirements, they need to be redeveloped and deployed, consuming a lot of human and material resources, and cannot meet flexible operating needs and immersive human-vehicle interaction experiences in real time. In this context, the ability to provide real-time dynamic interactive pages between users and the cockpit in multimodal scenarios and support the execution of ecosystem services is particularly important.

[0003] One related technology discloses a method for operating lightweight applications, proposing to optimize the operating efficiency of in-vehicle lightweight applications through an intermediate application layer. However, this technology relies on pre-stored application development information and does not support the function of users generating lightweight applications on demand in real time. Summary of the Invention

[0004] This application provides a method, apparatus, vehicle, and electronic device for generating lightweight applications. It can temporarily generate a DSL script for a lightweight application using real-time collected multimodal data, and then generate the interface of the lightweight application based on the DSL script. The interface includes the configuration information of at least one in-vehicle service covered by the lightweight application. Without requiring prior deployment of the lightweight application, it quickly generates a lightweight application interface adapted to the user's intent, containing multiple in-vehicle services, and completes related configuration options, significantly saving system resources and providing an immersive in-vehicle experience that meets the user's intent in real time.

[0005] According to a first aspect provided in this application, a method for generating lightweight applications is provided, the method comprising: Collect multimodal data of occupants inside the vehicle; the multimodal data includes at least two of the following: dialogue data, facial expression data, motion data, physiological data, and body posture data; The multimodal data and vehicle configuration information are input into the lightweight application generative big model. The lightweight application generative big model identifies the occupant's intention to use the in-vehicle services in the lightweight application and generates the DSL script of the lightweight application. The vehicle configuration information includes at least one of the following: vehicle model information, vehicle condition information, component configuration information, and ecosystem service configuration information. Based on the DSL script, the interface of the lightweight application is displayed on the vehicle infotainment system; the interface includes the configuration of at least one in-vehicle service from the set of in-vehicle services covered by the lightweight application; the configuration is adapted to the intended use.

[0006] As can be seen, by applying the scheme of this application, multimodal data of vehicle occupants, including at least two of the following: dialogue data, facial expression data, action data, physiological data, and body posture data, can be actively collected. This multimodal data is input into a lightweight application generative model, which identifies the occupant's intention to use the in-vehicle services of the lightweight application, i.e., the in-vehicle services the occupant might want to use, and generates a DSL script for the lightweight application. Based on the DSL script, the interface of the in-vehicle services that the occupant might want to use is displayed on the vehicle's infotainment system, recommending in-vehicle service interfaces to the user for selection. The interface includes the configuration of at least one in-vehicle service from the set of in-vehicle services covered by the lightweight application; the configuration is adapted to the usage intention. The DSL script for the lightweight application can be temporarily generated using real-time collected multimodal data, and then the lightweight application recommended to the user based on the DSL script can be actively generated to meet the user's usage needs. Pre-deployment of the lightweight application is not required, significantly saving system resources; only the relevant pre-stored components need to be reused when generating the lightweight application in real time. It can quickly generate lightweight application interfaces that adapt to the user's intent and include multiple in-vehicle services, and complete related configuration options, which greatly saves system resources and provides an immersive in-vehicle experience that meets the user's intent in real time.

[0007] In one possible approach, the DSL script includes a first DSL script for describing a group of in-vehicle services and / or a second DSL script for describing a single in-vehicle service; wherein the group of in-vehicle services includes multiple pre-configured in-vehicle services.

[0008] In one possible approach, the lightweight application generative large model generates the DSL script for the lightweight application through the following operations: The lightweight application generative large model performs semantic parsing on the multimodal data to obtain semantic parsing results; The generative big model for lightweight applications identifies the occupant’s intention to use the lightweight applications based on the vehicle configuration information and semantic parsing results. The lightweight application generative big model generates the DSL script of the lightweight application based on the usage intent.

[0009] As can be seen, the generative lightweight application provided in this application has scalability and sustainability. Traditional methods require customized development and a large investment of human and material resources. This system only needs to access the corresponding ecosystem services (voice, navigation, vehicle control, etc.) to have the capabilities of most in-vehicle applications, which greatly reduces development costs.

[0010] In one possible approach, the lightweight application generative big model generates the DSL script for the lightweight application based on the usage intent, including: The lightweight application generative big model generates the DSL script of the lightweight application based on the usage intent and the occupant's personalized configuration information; wherein, the personalized configuration information includes at least one of the following: interface layout preference information and in-vehicle service preference information.

[0011] As can be seen, by applying the solution of this application embodiment, when generating DSL scripts, the user's usage habits and preferences are taken into account, and the interface layout and interaction methods are dynamically adjusted to provide a more personalized experience.

[0012] In one possible approach, before inputting the multimodal data into a lightweight generative large model, the method further includes: Historical data within a second time window preceding the first time window is acquired; the historical data includes historical action information and / or historical vehicle condition information; wherein, the historical action information represents actions performed by the user on the vehicle; the first time window is the time window for acquiring the multimodal data; the length of the second time window is a first preset duration; The multimodal data is input into a lightweight application generative large model. The lightweight application generative large model identifies the occupant's intent to use the lightweight application and generates a DSL script for the lightweight application, including: The multimodal data and the historical data are input into the lightweight application generative big model. The lightweight application generative big model identifies the occupant's intention to use the lightweight application and generates the DSL script of the lightweight application.

[0013] As can be seen, in this embodiment of the application, the user's self-executed actions or vehicle condition information within the previous time window are input into the lightweight application generative model as additional information. Since the user's self-executed actions a short time ago can also reflect the current intent to some extent, the combination of the above additional information can help the large model to more accurately identify the user's intent and further ensure that the generated lightweight application meets the user's needs.

[0014] In one possible manner, the method includes: Obtain a training sample set; the training samples in the training sample set include sample DSL scripts, as well as sample multimodal data and scores corresponding to the sample DSL scripts; Based on the training sample set, the initial lightweight application generative large model is trained to obtain the lightweight application generative large model.

[0015] As can be seen, in this embodiment, a training sample set containing sample DSL scripts, corresponding sample multimodal data, and scores is obtained, and an initial lightweight application generative large model is trained based on this set to obtain a lightweight application generative large model. Since the correspondence between sample multimodal data and sample DSL scripts allows the model to learn the mapping rules from data to scripts, and the scores can guide the model to optimize the generation quality, the trained model can more accurately generate high-quality lightweight application DSL scripts that meet the requirements based on the input multimodal data, effectively improving the accuracy and practicality of lightweight application generation.

[0016] In one possible approach, the training samples further include: historical sample data within a fourth time window preceding the third time window; the historical sample data includes historical sample action information and / or historical sample vehicle condition information; wherein, the third time window is the time window for acquiring the multimodal sample data; and the length of the fourth time window is a second preset duration.

[0017] As can be seen, in this embodiment, historical sample data from a fourth time window preceding the third time window is additionally included in the training samples. This historical data includes historical action information and / or historical vehicle condition information. Since the third time window is the time node for acquiring multimodal sample data, the historical user actions and vehicle condition information contained in the historical sample data within the fourth time window can provide contextual basis for the user's intent when acquiring multimodal sample data in the time dimension. User behavior in recent historical periods is often related to current intent. Integrating this historical sample data into training allows the initial lightweight application generative model to more comprehensively understand the complex relationship between multimodal data and user intent in different contexts during the learning process. This leads to the training of a higher-performing lightweight application generative model, ultimately enabling it to more accurately meet the user's actual needs when generating lightweight application DSL scripts.

[0018] In one possible approach, training the initial lightweight application generative large model based on the training sample set to obtain the lightweight application generative large model includes: The initial lightweight application generative large model is trained by using the sample multimodal data and the sample historical data as training inputs, the sample DSL script as training outputs, and the rating as the satisfaction level of the training outputs, to obtain the lightweight application generative large model. As can be seen, in this embodiment, a lightweight application generative large model is obtained by training the initial model using sample multimodal data and sample historical data (including historical actions / vehicle condition information) as training input, sample DSL scripts as training output, and ratings as output satisfaction indicators. This achieves deep fusion learning of multidimensional information, enabling the model to capture the correlation between "current multimodal data + recent historical behavior" and user intent. The rating mechanism guides the model to iterate towards higher satisfaction through feedback optimization, improving the accuracy and practicality of the generated scripts. Ultimately, the trained model not only more comprehensively understands user needs and scenarios but also generates lightweight application DSL scripts that highly match the user's actual intent, effectively enhancing the quality assurance and user demand matching capability of lightweight application generation. In one possible approach, displaying the lightweight application interface on the vehicle infotainment system based on the DSL script includes: Parse the script code in the DSL script, wherein the script code includes at least one of service element description code, data source definition code, and UI layout description code; Based on the analysis results, the interface of the light application is visualized and rendered on the vehicle's infotainment system.

[0019] In one possible approach, the visualization rendering based on the parsing results on the vehicle's infotainment interface includes: Determine the in-vehicle service components associated with the parsing results; Determine the rendering order of the in-vehicle service components; Based on the rendering order, the vehicle service interfaces that connect to the vehicle service components are called sequentially. Based on the data returned by the vehicle service interfaces, the interface of the lightweight application is rendered on the vehicle infotainment system interface. The interface of the lightweight application includes a display interface for showing the service content of the vehicle service components.

[0020] As can be seen, in this embodiment, only access to in-vehicle ecosystem services is required to acquire the capabilities of most in-vehicle applications. Compared to traditional custom application development, this significantly reduces the required human and material resources, and improves the scalability and sustainability of generative lightweight applications. Furthermore, description codes for service cards can be predefined, with each service card representing a set of in-vehicle services. When generating the DSL script to describe the lightweight application, the description code of the service card can be directly called. This allows a single description code to cover multiple in-vehicle services, enabling the DSL script to describe lightweight applications more efficiently and improving the efficiency of DSL script generation.

[0021] In one possible approach, the method further includes: In the case where the lightweight application described by the DSL script includes third-party ecosystem services, the intermediate server associated with the third-party ecosystem services is invoked to access the API service interface used to connect to the third-party ecosystem services. The visualization rendering based on the parsing results on the vehicle's infotainment interface to display the interface of the lightweight application includes: Based on the parsing results of the script code in the DSL script and the data returned by the API service interface, the interface of the lightweight application is rendered on the vehicle infotainment system interface; the interface of the lightweight application includes a display interface for showcasing the service content of the third-party ecosystem services.

[0022] As can be seen, by using the above-mentioned ecosystem integration method, it is not necessary to pre-deploy third-party ecosystem software on the vehicle. When the generated lightweight application involves third-party ecosystem service capabilities, the corresponding services can be obtained by accessing the API interface provided by the third-party system through the intermediate server. This improves the flexibility of generating lightweight applications and reduces the pressure of deploying software on the vehicle.

[0023] In one possible approach, the method further includes: In response to a user's denial command to the lightweight application, new multimodal data of the occupant is acquired; The new multimodal data is input into the lightweight application generative big model. The lightweight application generative big model identifies the deviation between the generated lightweight application and the occupant's usage needs, and generates the adjusted DSL script for the lightweight application. Based on the modified DSL script of the lightweight application, the modified interface of the lightweight application is displayed on the vehicle infotainment system.

[0024] As can be seen, the solution provided by this application offers an intuitive and direct interactive interface, enabling users to generate lightweight applications in real time as needed. These real-time generated lightweight applications are not fixed or unchangeable; before they take effect, they can receive user adjustments to the application's configuration information. That is, when there are misunderstandings in the system's understanding, users can optimize multiple times through clarification, or customize the interface to obtain a lightweight application that meets their needs. This further enhances the flexibility of lightweight application generation and improves the user experience.

[0025] In one possible approach, the method further includes: In response to receiving a user's adjustment operation for the lightweight application, the adjustment information corresponding to the adjustment operation is input into the lightweight application generative big model, and the adjusted DSL script of the lightweight application is generated through the lightweight application generative big model. Based on the modified DSL script of the lightweight application, the modified interface of the lightweight application is displayed on the vehicle infotainment system.

[0026] In one possible approach, the method further includes: Based on the DSL script corresponding to the lightweight application before adjustment and the DSL script corresponding to the lightweight application after adjustment, the generative large model of the lightweight application is optimized.

[0027] In one possible approach, inputting the multimodal data into a lightweight generative large model includes: The multimodal data is time-aligned; The time-aligned multimodal data is preprocessed and features are extracted to obtain multimodal feature information; The multimodal feature information is input into the lightweight application generative large model.

[0028] In one possible approach, inputting the multimodal data into the lightweight application generative model includes: inputting the multimodal data into the lightweight application generative model when a triggering condition is met; the triggering condition includes: determining that the lightweight application generation function is enabled based on a user instruction.

[0029] According to a second aspect provided in this application, a lightweight application generation apparatus is provided, the apparatus comprising: The acquisition module is used to collect multimodal data of occupants inside the vehicle; The generation module is used to input the multimodal data and vehicle configuration information into the lightweight application generative big model, identify the usage intent through the lightweight application generative big model, and generate the DSL script of the lightweight application; the vehicle configuration information includes at least one of the following: vehicle model information, vehicle condition information, component configuration information, and ecosystem service configuration information; The display module is used to display the interface of the lightweight application on the vehicle infotainment system based on the DSL script; the interface includes the configuration information of at least one in-vehicle service in the set of in-vehicle services covered by the lightweight application; the configuration information is adapted to the usage intent.

[0030] In one possible embodiment, the DSL script includes a first DSL script for describing a group of in-vehicle services and / or a second DSL script for describing a single in-vehicle service; wherein the group of in-vehicle services includes multiple pre-configured in-vehicle services.

[0031] In one possible approach, the lightweight application generative large model generates the DSL script for the lightweight application through the following operations: The lightweight application generative large model performs semantic parsing on the multimodal data to obtain semantic parsing results; The lightweight application generative big model identifies the usage intent based on the vehicle configuration information and semantic parsing results. The lightweight application generative big model generates the DSL script of the lightweight application based on the usage intent.

[0032] In one possible approach, the lightweight application generative big model generates the DSL script for the lightweight application based on the usage intent, including: The lightweight application generative big model generates the DSL script of the lightweight application based on the usage intent and the occupant's personalized configuration information; wherein, the personalized configuration information includes at least one of the following: interface layout preference information and in-vehicle service preference information.

[0033] In one possible approach, the acquisition module is also used for: Historical data within a second time window preceding the first time window is acquired; the historical data includes historical action information and / or historical vehicle condition information; wherein, the historical action information represents actions performed by the user on the vehicle; the first time window is the time window for acquiring the multimodal data; the length of the second time window is a first preset duration; The multimodal data is input into a lightweight application generative large model. The lightweight application generative large model identifies the occupant's intent to use the lightweight application and generates a DSL script for the lightweight application, including: The multimodal data and the historical data are input into the lightweight application generative big model. The lightweight application generative big model identifies the occupant's intention to use the lightweight application and generates the DSL script of the lightweight application.

[0034] In one possible embodiment, the device further includes: a training module; the training module is used for: Obtain a training sample set; the training samples in the training sample set include sample DSL scripts, as well as sample multimodal data and scores corresponding to the sample DSL scripts; Based on the training sample set, the initial lightweight application generative large model is trained to obtain the lightweight application generative large model.

[0035] In one possible approach, the training module is specifically used for: The initial lightweight application generative large model is trained by using the sample multimodal data and the sample historical data as training inputs, the sample DSL script as training outputs, and the rating as the satisfaction level of the training outputs, to obtain the lightweight application generative large model. In one possible approach, the display module is specifically used for: Parse the script code in the DSL script, wherein the script code includes at least one of service element description code, data source definition code, and UI layout description code; Based on the analysis results, the interface of the light application is visualized and rendered on the vehicle's infotainment system.

[0036] In one possible approach, the display module is specifically used for: Determine the in-vehicle service components associated with the parsing results; Determine the rendering order of the in-vehicle service components; Based on the rendering order, the vehicle service interfaces that connect to the vehicle service components are called sequentially. Based on the data returned by the vehicle service interfaces, the interface of the lightweight application is rendered on the vehicle infotainment system interface. The interface of the lightweight application includes a display interface for showing the service content of the vehicle service components.

[0037] In one possible approach, the apparatus further includes: a calling module, configured to, when the lightweight application described by the DSL script contains third-party ecosystem services, call an intermediate server associated with the third-party ecosystem services to access the API service interface for interfacing with the third-party ecosystem services. The display module is specifically used for: Based on the parsing results of the script code in the DSL script and the data returned by the API service interface, the interface of the lightweight application is rendered on the vehicle infotainment system interface; the interface of the lightweight application includes a display interface for showcasing the service content of the third-party ecosystem services.

[0038] In one possible approach, the acquisition module is also used for: In response to a user's denial command to the lightweight application, new multimodal data of the occupant is acquired; The generation module is further configured to: input the new multimodal data into the lightweight application generative large model, identify the deviation between the generated lightweight application and the occupant's usage needs through the lightweight application generative large model, generate the adjusted DSL script for the lightweight application, and display the adjusted interface of the lightweight application on the vehicle infotainment system based on the adjusted DSL script for the lightweight application.

[0039] In one possible embodiment, the device further includes: an adjustment module, configured to: In response to receiving user adjustments to the lightweight application, The adjustment information corresponding to the adjustment operation is input into the lightweight application generative large model, and the adjusted DSL script of the lightweight application is generated through the lightweight application generative large model. Based on the modified DSL script of the lightweight application, the modified interface of the lightweight application is displayed on the vehicle infotainment system.

[0040] In one possible approach, the multimodal data includes at least two of the following: dialogue data, facial expression data, gesture data, physiological data, and body posture data.

[0041] In one possible approach, the generation module is specifically used for: The multimodal data is time-aligned; The time-aligned multimodal data is preprocessed and features are extracted to obtain multimodal feature information; The multimodal feature information is input into the lightweight application generative large model.

[0042] In one possible approach, a generation module is specifically used for: When the triggering conditions are met, the multimodal data is input into the lightweight application generative large model; the triggering conditions include: determining that the lightweight application generation function is enabled based on user instructions.

[0043] According to a third aspect provided in this application, a vehicle is provided, comprising: the apparatus described in the second aspect and any possible implementation thereof.

[0044] According to a fourth aspect provided in this application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method of the first aspect described above and any possible implementation thereof.

[0045] According to a fifth aspect provided in this application, a computer-readable storage medium is provided that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0046] According to the sixth aspect provided in this application, a computer program product is provided, the computer program product including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0047] It should be noted that the technical effects of any of the implementation methods in aspects two through six can be found in the technical effects of the corresponding implementation methods in aspect one, and will not be repeated here.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0050] Figure 1 This is a flowchart illustrating a lightweight application generation method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating a multimodal preprocessing process according to an exemplary embodiment; Figure 3 This is a flowchart illustrating a lightweight application DSL script generation method according to an exemplary embodiment; Figure 4 This is a flowchart illustrating another lightweight application generation method according to an exemplary embodiment; Figure 5 This is a block diagram illustrating a lightweight application generation apparatus according to an exemplary embodiment; Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0052] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] In the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0054] To address the issues of low operational efficiency and poor scenario adaptability in traditional vehicle-machine interaction modes, this application provides a lightweight application generation method, device, and vehicle.

[0055] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0056] This application relates to vehicles, which may also be referred to as vehicles, mobile carriers, electric vehicles (EVs), hybrid electric vehicles (HEVs), plug-in hybrid electric vehicles (PHEVs), fuel cell vehicles (FCVs), autonomous vehicles, intelligent and connected vehicles (ICVs), driverless vehicles, etc.

[0057] In this application, the vehicle can be a sedan, a sport utility vehicle (SUV), a truck, an electric vehicle, a motorcycle, a tricycle, a special vehicle (such as an ambulance, fire truck, police car, etc.), a driverless taxi, an intelligent connected bus, an autonomous logistics vehicle, an electric truck, etc. Furthermore, this method is also applicable to various special-purpose vehicles, such as agricultural vehicles, mining vehicles, forestry vehicles, airport vehicles, and port vehicles. This application does not impose specific limitations in this regard.

[0058] In this application embodiment, the system mainly includes: a multimodal interactive acquisition system, a large model training system, a large model hosting system, a lightweight application generator, an adaptive system, and a configuration system. The following is a brief introduction to the systems involved: (1) Multimodal interactive acquisition system: This system acquires various perceptual heterogeneous data such as voice, gestures and facial expressions through multiple channels. The data is time-synchronized and attention-weighted fused, and a unified semantic expression is generated after decision-making using a modal confidence scoring mechanism.

[0059] (2) Large model training system: This system is responsible for building a model database that includes component library metadata, domain-specific language (DSL) adaptation templates, and interaction history. It uses knowledge augmentation and supervised fine-tuning (SFT) techniques to provide a model that can accurately identify user intent and generate DSL scripts for lightweight applications.

[0060] (3) Large model carrying system: The system carries a large model that has been specifically trained to preprocess the semantic expressions generated in the above steps, perform context association analysis and intent recognition, with the aim of understanding the user's intent and generating DSL scripts based on it.

[0061] (4) Lightweight application generator: The lightweight application generator supports user-defined lightweight applications, and can also import DSL scripts and convert them into interface elements that users can intuitively understand and use, dynamically display them to users, and allow users to directly modify and interact with them and execute related ecosystem services.

[0062] (5) Intelligent Adaptive System: This system is used to capture users' usage habits and preferences, build personal profiles, and provide further personalized services. It is continuously adjusted and optimized based on historical usage data, enabling the system to better capture users' personalized needs and provide more accurate services.

[0063] (6) Configuration system: The system enables dynamic assembly of components, configuration of ecological services and deployment of user-personalized configuration files through cloud collaboration, and supports on-demand adaptation and real-time updates of vehicle functions.

[0064] The lightweight application generation method provided in this application will be described in detail below with reference to the accompanying drawings. See also... Figure 1 The method may include the following steps: S101: Collect multimodal data of occupants inside the vehicle.

[0065] In this embodiment of the application, the occupants include the driver and passengers.

[0066] In some embodiments of this application, multimodal data includes at least two of the following: dialogue data, facial expression data, motion data, physiological data, and body posture data.

[0067] For example, multimodal data may include: the driver is present; the driver's body temperature is 37.2 degrees Celsius; the driver's heart rate is 88 beats per minute; the respiratory rate is 19 breaths per minute; the gesture is fanning with the hand; the field of vision is directly in front; the pointing area is the car window; the emotion is irritability; and the voice message is "What do you want to eat?"

[0068] For example, voice data can be captured using a microphone inside the vehicle, while gesture, facial expression, and body posture data can be captured using an in-vehicle camera. For instance, the microphone can capture conversations between the driver and passengers such as "It's hot" or "It's too hot," and the in-vehicle camera can capture the actions of passengers fanning themselves with their hands.

[0069] S102: Input multimodal data and vehicle configuration information into the lightweight application generative big model, identify the occupant's intention to use the in-vehicle service in the lightweight application through the lightweight application generative big model, and generate the DSL script of the lightweight application.

[0070] In this embodiment, a pre-trained generative model for lightweight applications can be integrated into the vehicle. For example, supported component library data, pre-edited scene templates, attribute / event documentation, and interaction data are used as a dataset. Components and templates are augmented with attributes, styles, and bound events. Through data augmentation and negative sample generation, over 500,000 data points are fed into the large model for training. Knowledge augmentation and SFT fine-tuning techniques are employed to enable the pre-trained large model to accurately identify user intent and generate DSL scripts for lightweight applications. After training, the model is deployed to the vehicle after manual inspection and automated testing.

[0071] In one embodiment of this application, see [link to embodiment]. Figure 2 Multimodal data is input into lightweight generative large models, including: S201: Time-align multimodal data.

[0072] For example, the collected multimodal data can be timestamped. For instance, data from different devices (cameras, microphones, etc.) can be unified onto a consistent timeline using their respective timestamps. Subsequently, the organized time-series data can be synchronized using a synchronization algorithm.

[0073] S202: Preprocess and extract features from the time-aligned multimodal data to obtain multimodal feature information.

[0074] As an example, preprocessing can include data cleaning. For instance, for audio data, processes such as silence removal and noise reduction can be performed.

[0075] For the preprocessed data, feature extraction can be performed to obtain multimodal feature information.

[0076] For example, feature extraction and text conversion are performed on dialogue data captured by the microphone. Dynamic feature extraction and gesture recognition are performed on gesture data captured by the camera. Key point tracking, feature point extraction, and micro-expression detection are performed on facial expression data captured by the camera.

[0077] S203: Input multimodal feature information into a lightweight generative large model.

[0078] The above embodiments are examples of this application, and feature extraction-related model structures can also be integrated into lightweight generative large models. No limitation is made in this regard.

[0079] In one embodiment of this application, the lightweight application generative big model receives multimodal data or multimodal feature information, identifies the occupant's intention to use the lightweight application based on the multimodal data or multimodal feature information, and generates a DSL script for the lightweight application.

[0080] DSL scripting refers to a domain-specific language focused on a particular application area. Unlike general-purpose cross-domain computer languages, domain-specific languages ​​are used only within certain specific domains. They are characterized by lightweight programming, simple structure, and ease of system execution.

[0081] For example, in-vehicle infotainment systems typically support voice control of modules such as air conditioning and navigation. However, the interaction flow can vary significantly between different vehicle models. For instance, the mapping relationship between voice commands and device controls may differ. In such cases, DSL script code can be used to define the in-vehicle infotainment interaction flow. Furthermore, DSL script code can also be used to define the calling relationships between connected vehicle services and the rules for processing vehicle data.

[0082] S103: Based on DSL scripts, display the interface of the lightweight application on the vehicle infotainment system; the interface includes the configuration of at least one in-vehicle service from the set of in-vehicle services covered by the lightweight application; the configuration is adapted to the user intent.

[0083] In this application embodiment, in-vehicle lightweight applications can be understood as lightweight, scenario-based applications designed specifically for car driving scenarios. They aim to provide users with a safe, convenient, and personalized in-vehicle service experience by simplifying functions, optimizing interactions, and adapting to the in-vehicle environment.

[0084] In this embodiment, the in-vehicle lightweight application may involve multiple in-vehicle services. Accordingly, the interface of the generated lightweight application may include the configuration information of at least one in-vehicle service from the set of in-vehicle services covered by the lightweight application. The configuration information is adapted to the intended use.

[0085] For example, a mini-app may involve multiple in-vehicle services such as "seat temperature adjustment," "air conditioning temperature adjustment," and "window adjustment." The mini-app's interface can then display information about each of these in-vehicle services. For example, for "seat temperature adjustment," icons or images related to seats can be displayed on the mini-app's interface.

[0086] Furthermore, the interface of the lightweight application displays the specific configuration of at least one in-vehicle service from the in-vehicle service set, and this configuration is related to the user's intent. For example, by recognizing the occupant's multimodal data, the lightweight application's generative model determines that the occupant is fatigued and their current intent is to rest. The generated lightweight application could then involve multiple in-vehicle services such as "seat angle adjustment," "air conditioning temperature adjustment," and "window adjustment." For "seat angle adjustment," it would be configured to a comfortable reclining angle; for "air conditioning temperature adjustment," it would be configured to a comfortable temperature for rest; and for "window adjustment," the windows would be closed. It is evident that the configuration of these multiple in-vehicle services is related to the user's intent.

[0087] In one embodiment of this application, after the interface of a lightweight application is displayed on the vehicle's infotainment system, the lightweight application can be recommended to the user according to the specific configuration of the in-vehicle services covered by the lightweight application, and then the various in-vehicle services can be run according to the user's selection. For example, starting to adjust the windows, seats, etc.

[0088] For example, based on the dialogue data between the driver and passenger, such as "the temperature is high" or "it's too hot", and the action of the occupants fanning themselves with their hands captured by the in-vehicle camera, an interface can be generated and recommended to the user to turn on the air conditioning, seat ventilation and other light applications.

[0089] In another embodiment of this application, the interface of the lightweight application displayed on the vehicle's infotainment system can be regarded as a preview interface of the in-vehicle services. After receiving confirmation from the user, each in-vehicle service is run according to the specific configuration of the in-vehicle services covered by the lightweight application. For example, if the user has no objection to the in-vehicle services displayed in the lightweight application and the configuration of each in-vehicle service, he / she can issue a confirmation command through voice, control buttons or other means, and then start running each in-vehicle service according to the specific configuration of the in-vehicle services covered by the lightweight application.

[0090] In this embodiment of the application, the vehicle system can integrate a lightweight application generator to parse various instructions in the DSL code.

[0091] In some embodiments of this application, the lightweight application generator parses the script code in the DSL script, the script code including at least one of service element description code, data source definition code, and UI layout description code; based on the parsing result, it performs visualization rendering on the vehicle interface to display the interface of the lightweight application.

[0092] For example, the service element description code is used to describe the service content included in the lightweight application. For instance, if the lightweight application to be generated includes a seat adjustment service, the service element description code can contain structured information describing the seat adjustment range, temperature, and whether the massage function is enabled. Typically, the lightweight application to be generated sets up multiple service contents, and each service content can be described by the service element description code.

[0093] For example, the data source definition code describes the data source, modification method, etc., of various types of data in the lightweight application. For instance, if the lightweight application to be generated involves obtaining weather status information, then the data source definition code can define the method for obtaining this data volume. During the generation of the lightweight application, the data is obtained through the corresponding method. The acquisition method may include the interface between the vehicle system and the backend API.

[0094] For example, the UI layout description code is used to describe the interface layout of the lightweight application to be generated, such as the arrangement and combination of elements in the lightweight application interface, the navigation logic between the main page and sub-pages, and spatial relationships.

[0095] After parsing the DSL script, it is visualized and rendered on the user interface. The rendering result can include various graphical elements.

[0096] In this embodiment, the generated lightweight application's interface is interactive and can be adjusted in response to user actions. For example, users can invoke services such as orchestration and vehicle control through simple clicks. For orchestration and vehicle control services bound to components, after the user performs clicks, swipes, or other operations, the generative rendering engine identifies the service information bound to the component and executes the corresponding operations. For instance, the orchestration service sends an execution script to the scene orchestration, which then parses, makes decisions, and executes the script.

[0097] In one embodiment of this application, in-vehicle service components and third-party ecosystem service capabilities can be integrated to achieve overall invocation and orchestration.

[0098] For example, an in-vehicle service component can be a single in-vehicle service or a collection of in-vehicle services. For instance, if multiple in-vehicle services are frequently used together, they can be defined as an in-vehicle service component, also known as a service card. For example, a service card can be pre-defined to include three in-vehicle services: "seat heating," "seat massage," and "window adjustment."

[0099] In one embodiment of this application, during the training phase of the lightweight application generative large model, the training data includes DSL scripts describing in-vehicle service components and DSL scripts describing individual in-vehicle services. Each in-vehicle service component includes multiple in-vehicle services bound together.

[0100] Correspondingly, the DSL script generated by the lightweight application generative large model based on multimodal data may include a first DSL script for describing in-vehicle service components and / or a second DSL script for describing a single in-vehicle service; wherein, the in-vehicle service components include multiple pre-configured in-vehicle services.

[0101] For example, when binding three in-vehicle services—"seat heating," "seat massage," and "window adjustment"—as a single in-vehicle service component, the DSL script for the lightweight application generated by the lightweight application generative large model can include a DSL script describing the in-vehicle service component, and can also include a DSL script describing an individual in-vehicle service, such as "ambient lighting adjustment." Thus, the interface of the lightweight application is displayed on the vehicle's infotainment system, including the display content of each in-vehicle service within the in-vehicle service component and the display content of an individual in-vehicle service.

[0102] As can be seen, in this solution, the description scripts of the vehicle service components can be predefined. When generating the DSL script to describe the lightweight application, the DSL script corresponding to the vehicle service component can be generated directly. In this way, a single script code can cover multiple vehicle services, enabling the DSL script to describe lightweight applications more efficiently and improving the efficiency of generating DSL scripts.

[0103] In one embodiment of this application, visualization rendering of the lightweight application interface is performed on the vehicle infotainment interface based on the parsing results. Specifically, this may include: determining the in-vehicle service components associated with the parsing results; determining the rendering order of the in-vehicle service components; sequentially calling the in-vehicle service interfaces that connect to the in-vehicle service components based on the rendering order; and rendering the lightweight application interface on the vehicle infotainment interface based on the data returned by the in-vehicle service interfaces. The lightweight application interface includes a display interface for showcasing the service content of the in-vehicle service components.

[0104] For example, the in-vehicle service interfaces corresponding to each in-vehicle service component can be pre-set. By parsing the DSL script, it is possible to determine which in-vehicle service components are involved in the lightweight application to be generated, and to determine the rendering order of the in-vehicle service components. Based on the rendering order, the corresponding in-vehicle service interfaces are called sequentially. During the rendering of the lightweight application interface, the interface is rendered based on the data returned by the in-vehicle service interfaces.

[0105] For example, in-vehicle service components include navigation services and various types of vehicle control services. If the lightweight application to be generated involves a certain in-vehicle service component, the corresponding in-vehicle service interface is called to obtain the required data for interface rendering. The required data can be determined based on the parsing results of the DSL script.

[0106] As can be seen, in this embodiment of the application, only the access to the vehicle ecosystem service is required to have the capabilities of most vehicle applications. Compared with the traditional custom application development method, this greatly reduces the human and material resources required and improves the scalability and sustainability of generative lightweight applications.

[0107] In one embodiment of this application, a rendering queue can be constructed based on the rendering order of the in-vehicle service components. If the rendering of the lightweight application interface on the vehicle infotainment system fails to execute based on the data returned by the in-vehicle service interface after calling the in-vehicle service interface that connects to the in-vehicle service component, an exception message is reported, and the current in-vehicle service interface is skipped, and the in-vehicle service interface that connects to the next in-vehicle service component in the rendering queue is called. For example, when the intent is to recommend nearby restaurants, the data returned by the in-vehicle service interface can be "Nearby Restaurant 1, its features are XXX", "Nearby Restaurant 1 image", "Nearby Restaurant 1 rating", "Nearby Restaurant 1 hot reviews", and "Nearby Restaurant 1 address", etc.

[0108] In one embodiment of this application, when the lightweight application described by the DSL script includes third-party ecosystem services, an intermediate server associated with the third-party ecosystem services can be invoked to access the API service interface used to connect to the third-party ecosystem services.

[0109] For example, if the generated lightweight application involves third-party ecosystem servers, such as music or video websites, it can access the API interfaces provided by the third-party system through an intermediate server to obtain the corresponding services. The intermediate server can act as a proxy server, preventing the third-party system from directly connecting to the vehicle's core bus or control system, thus improving data access security.

[0110] Specifically, when calling third-party ecosystem services, a request conforming to the standard can be sent to the intermediate server. The intermediate server will then parse, verify, and call the third-party service or in-vehicle service. If a corresponding service is available, the request result will be returned to meet the user's needs.

[0111] Accordingly, in one embodiment of this application, the interface of the lightweight application is displayed on the vehicle screen based on the parsing results, including: rendering the interface of the lightweight application on the vehicle screen based on the parsing results of the script code in the DSL script and the data returned by the API service interface; the interface of the lightweight application includes a display interface for displaying the service content of third-party ecosystem services.

[0112] Because third-party ecosystem services are invoked, the rendering of the lightweight application interface is performed by combining the script parsing results with data returned by the API service interface. For example, if the lightweight application to be generated includes playing a song from xx music, the API service interface connected to xx music is called to obtain data related to the song, which is used for rendering the lightweight application interface, ultimately generating a display interface containing service content for displaying xx music.

[0113] As can be seen, by using the above-mentioned ecosystem integration method, it is not necessary to pre-deploy third-party ecosystem software on the vehicle. When the generated lightweight application involves third-party ecosystem service capabilities, the corresponding services can be obtained by accessing the API interface provided by the third-party system through the intermediate server. This improves the flexibility of generating lightweight applications and reduces the pressure of deploying software on the vehicle.

[0114] By applying the scheme of this application, multimodal data of vehicle occupants can be actively collected, including at least two of the following: dialogue data, facial expression data, action data, physiological data, and body posture data. This multimodal data is input into a lightweight application generative model. The model identifies the occupant's intention to use the in-vehicle services of the lightweight application, i.e., the in-vehicle services the occupant might want to use, and generates a DSL script for the lightweight application. Based on the DSL script, the interface of the in-vehicle services the occupant might want to use is displayed on the vehicle's infotainment system, recommending in-vehicle service interfaces to the user for selection. The interface includes the configuration information of at least one in-vehicle service from the set of in-vehicle services covered by the lightweight application; the configuration information is adapted to the usage intention. The DSL script for the lightweight application can be temporarily generated using real-time collected multimodal data, and then the lightweight application recommended to the user based on the DSL script can be actively generated to meet the user's needs. Pre-deployment of the lightweight application is not required, significantly saving system resources; only pre-stored components need to be reused during real-time generation of the lightweight application.

[0115] In one implementation, see Figure 3 Lightweight application generative large models can generate lightweight application DSL scripts through the following operations: S301: Lightweight application of generative large models to perform semantic parsing on multimodal data and obtain semantic parsing results.

[0116] In this embodiment of the application, data of different modalities are transformed to obtain a structured and abstract semantic representation that can be understood and manipulated by the machine. This semantic representation can be a vector or structured text (e.g., JSON format).

[0117] For example, the core architecture of a lightweight generative large model can adopt a transformer architecture, which achieves multimodal semantic parsing through self-attention and cross-attention mechanisms.

[0118] S302: Lightweight application generative large model identifies the occupant's intention to use lightweight applications based on vehicle configuration information and semantic parsing results.

[0119] As an example, vehicle configuration information can include vehicle model information, vehicle condition information, component configuration information, and ecosystem service configuration information. Vehicle model information can be used to associate the vehicle's hardware configuration, such as whether it has features like seat massage and power seat adjustment. Vehicle condition information can be associated with the vehicle's current state, such as whether the current state of the vehicle is suitable for activating specific functions. Component configuration information represents the available components in the vehicle configuration, and ecosystem service configuration information represents the available software ecosystem in the vehicle configuration, such as third-party music software.

[0120] In this embodiment, the lightweight application generative big data model identifies the occupant's intent to use the lightweight application based on semantic parsing results, contextual information, and vehicle configuration information. The contextual information may include in-vehicle status information and user identity information. In-vehicle status information may include current air conditioning temperature and seat status.

[0121] S303: Lightweight application generative large model is based on DSL scripts that generate lightweight applications based on usage intent.

[0122] The trained lightweight application generative large model has the ability to generate DSL scripts. After determining the usage intent for the lightweight application, it generates the corresponding DSL script.

[0123] For example, after a user gets into the car, they voice-input "I'm a little tired" accompanied by a yawning gesture. The in-vehicle system first collects and processes this voice and facial information in real time, performing data analysis and weighting to obtain multimodal semantics. This multimodal semantics is then input into a lightweight application generative model. The lightweight application generative model, through semantic understanding, contextual analysis, and configuration information, identifies the user's intent, obtains the passenger's intention to use the lightweight application, and then generates the lightweight application's DSL script. In this scenario, the target lightweight application could be "Nap Mode".

[0124] For example, the generated lightweight application interface includes interactive controls with graphical elements. For instance, for a lightweight application like "Rest Mode," the generated interface could display the seat adjustment angle, air conditioning temperature, and airflow direction. If the user finds the lightweight application satisfactory... To meet your needs, you can simply confirm with a click. Once the vehicle's infotainment system receives confirmation, it can adjust the seats and air conditioning according to the information displayed in the app's interface.

[0125] As can be seen, the generative lightweight application provided in this application has scalability and sustainability. Traditional methods require customized development and a large investment of human and material resources. This system only needs to access the corresponding ecosystem services (voice, navigation, vehicle control, etc.) to have the capabilities of most in-vehicle applications, which greatly reduces development costs.

[0126] In one embodiment of this application, the method further includes, before inputting multimodal data into a lightweight generative large model: Historical data within a second time window preceding the first time window is acquired. This historical data includes historical action information and / or historical vehicle condition information. The historical action information represents actions performed by the user on the vehicle. The first time window is the time window for acquiring multimodal data. The length of the second time window is a first preset duration.

[0127] Optionally, the first preset duration can be set according to actual needs. For example, the first preset duration can be the same as the duration corresponding to the first time window. This application does not impose specific restrictions on this.

[0128] For example, after collecting multimodal data within the first time window, intent recognition can be performed by combining historical action information and historical vehicle condition information from the second time window prior to the first time window with the multimodal data.

[0129] Optionally, there can be multiple second time windows, which means that historical action information and historical vehicle condition information within multiple second time windows can be obtained for intent recognition.

[0130] For example, historical action information refers to actions performed by the user themselves, not actions performed by the vehicle after intent recognition. Examples include the user opening a window or lowering the air conditioning temperature.

[0131] Correspondingly, the multimodal data is input into the lightweight application generative big model, specifically including: inputting multimodal data and historical data into the lightweight application generative big model, identifying the occupant's intention to use the lightweight application through the lightweight application generative big model, and generating the lightweight application's DSL script.

[0132] In one implementation, the aforementioned historical data can be mixed with a specific prompt word template and input into a mini-app. For example, if it is detected that the user just opened the car window, this information can be used for intent recognition.

[0133] As can be seen, in this embodiment of the application, the user's self-executed actions or vehicle condition information within the previous time window are input into the lightweight application generative model as additional information. Since the user's self-executed actions a short time ago can also reflect the current intent to some extent, the combination of the above additional information can help the large model to more accurately identify the user's intent and further ensure that the generated lightweight application meets the user's needs.

[0134] In one embodiment of this application, a lightweight application generative big model generates a DSL script for a lightweight application based on usage intent, including: the lightweight application generative big model generates a DSL script for a lightweight application based on usage intent and the personalized configuration information of the occupant; wherein, the personalized configuration information includes at least one of the following: interface layout preference information and in-vehicle service preference information.

[0135] For example, a user's personalized configuration information can be continuously optimized, and the in-vehicle system can collect key user interaction information to optimize user preferences, which may include interface layout preferences and in-vehicle service preferences.

[0136] As an example, personalized configuration information can be optimized through explicit feedback. For instance, a feedback entry point can be provided in the app's display interface, responding to user actions and receiving personalized configuration information entered by the user.

[0137] As an example, personalized configuration information can be optimized through implicit feedback. For instance, analyzing user behavior data can indirectly infer user preferences.

[0138] Therefore, when generating scripts for lightweight DSL applications, the system can further incorporate the personalized configuration information of passengers. For example, during the DSL script generation process, relevant preference settings are queried from the user preference library, and preference conflicts are handled. For instance, if a user prefers "light mode," and the current scene is nighttime, the system may automatically generate an interface for night mode according to the original rules. However, since this conflicts with the user's preference, the mode can be optimized and adjusted accordingly.

[0139] As can be seen, by applying the solution of this application embodiment, when generating DSL scripts, the user's usage habits and preferences are taken into account, and the interface layout and interaction methods are dynamically adjusted to provide a more personalized experience.

[0140] In one embodiment of this application, after displaying the interface of the lightweight application on the vehicle infotainment system, the method may further include: in response to the user's denial instruction of the lightweight application, obtaining new multimodal data of the occupant; inputting the new multimodal data into the lightweight application generative big model, identifying the deviation between the generated lightweight application and the occupant's usage needs through the lightweight application generative big model, and generating a lightweight application-adjusted DSL script.

[0141] For example, if a user believes the generated lightweight application does not meet their needs, they can enter a denial command. This can be done, for instance, through voice commands or by clicking on interactive controls on the vehicle's infotainment system.

[0142] Upon receiving a user's rejection command for the lightweight application, new multimodal data about the occupants can be reacquired. This new multimodal data is then processed, and the capabilities of the large model are used to identify discrepancies between the generated lightweight application and the occupants' usage needs, allowing for further adjustments to the DSL script.

[0143] In one embodiment of this application, after displaying the interface of the lightweight application on the vehicle infotainment system, the method may further include: in response to receiving a user's adjustment operation on the lightweight application, inputting the adjustment information corresponding to the adjustment operation into the lightweight application generative big model, generating the adjusted DSL script of the lightweight application through the lightweight application generative big model; and displaying the adjusted interface of the lightweight application on the vehicle infotainment system based on the adjusted DSL script of the lightweight application.

[0144] For example, if a user feels that the generated mini-app deviates from their needs, they can adjust the information displayed in the mini-app's editing interface. For instance, in the "Rest Mode" mini-app, if a user is dissatisfied with the seat angle or air conditioning blast angle displayed in the interface, they can manually change the seat angle. For example, they can slide their finger across the seat icon in the interface to change its angle.

[0145] The vehicle's infotainment system then receives the user's adjustment request and displays the adjusted interface of the lightweight application on the vehicle's screen. If the user is satisfied with the information to be adjusted displayed in the updated interface, they can confirm the change.

[0146] As can be seen, the solution provided by this application offers an intuitive and direct interactive interface, enabling users to generate lightweight applications in real time as needed. These real-time generated lightweight applications are not fixed or unchangeable; before they take effect, they can receive user adjustments to the application's configuration information. That is, when there are misunderstandings in the system's understanding, users can optimize multiple times through clarification, or customize the interface to obtain a lightweight application that meets their needs. This further enhances the flexibility of lightweight application generation and improves the user experience.

[0147] In some embodiments of this application, inputting multimodal data into a lightweight application generative large model includes: inputting multimodal data into a lightweight application generative large model when a triggering condition is met.

[0148] As an example, a trigger condition is set in advance, and multimodal data is input into the generative large model of the lightweight application only when the trigger condition is met, thereby triggering the generation of the lightweight application.

[0149] For example, the triggering conditions include: determining that the lightweight application generation function is enabled based on user instructions. For instance, the user issues an instruction to the vehicle system to enable the lightweight application generation function through vehicle operation or voice command. Subsequently, the vehicle system controls the lightweight application generation function to be enabled. Only in this state is the acquired multimodal data of the occupants input into the lightweight application generation model for intent recognition and subsequent lightweight application generation.

[0150] It is evident that if intent recognition and the generation of mini-applications are triggered based on multimodal data obtained from vehicle occupants every time, mini-applications may be generated frequently, affecting the normal use of the vehicle's infotainment system. By setting trigger conditions, the generation of mini-applications can be restricted, preventing the vehicle's infotainment system from frequently and automatically generating mini-applications, thus avoiding interference with the system's execution of direct user commands.

[0151] To facilitate understanding, the lightweight application generation method provided in this application embodiment will be further described below with reference to the accompanying drawings. See also... Figure 4 , Figure 4 The model training process and the model usage process are shown.

[0152] For the model training process, the dataset used for training is determined, including: component library metadata, DSL adaptation templates, attribute description documents, and historical records. The dataset is preprocessed, including structure transformation, data augmentation, and negative sample generation, and then iterative training is performed based on the dataset.

[0153] During model usage, multimodal data acquisition and processing are performed. The processed data is then input into the lightweight application generative large model, carrying personalized configurations. The lightweight application generative large model performs preprocessing, context association, intent recognition, and content generation to obtain the DSL script. The DSL script is then input into the generative rendering engine, which includes a DSL parsing engine and a rendering engine. The DSL parsing engine calls the scene engine service through the in-vehicle service call module. The scene engine is configured with a script parsing module, a decision-making module, an execution module, and a script management module. The rendering engine calls the intermediate server through the ecosystem service call module. The intermediate server is configured with a call parsing module, a service authentication module, an ecosystem service execution module, and a database.

[0154] After processing by the scene engine and intermediate server, the service call results are returned to the generative rendering engine. The generative rendering engine then generates an interactive, lightweight application interface based on the service call results.

[0155] The system then determines whether the user confirms the request; if so, the mini-application is executed. Furthermore, data is collected during the mini-application generation process, and this collected data is used as key interaction records and written into the dataset used for training the large model.

[0156] If the user does not confirm, for example, if the user clarifies the generated mini-application interface, the mini-application will not be executed, and the process will return to multimodal data collection.

[0157] In another embodiment, during the model training process, the lightweight application generation device can acquire a training sample set, i.e., a training dataset. Then, based on the training sample set, the lightweight application generation device can train the initial lightweight application generative large model to obtain the lightweight application generative large model.

[0158] The training sample set includes sample DSL scripts, and corresponding sample multimodal data and scores. The training samples also include historical sample data within a fourth time window preceding the third time window; the historical sample data includes historical action information and / or historical vehicle condition information. The third time window is the time window for acquiring the sample multimodal data; the length of the fourth time window is a second preset duration.

[0159] Optionally, the second preset duration can be set according to actual needs. For example, the second preset duration can be the same as the duration corresponding to the third time window. This application does not impose specific limitations in this regard.

[0160] Optionally, there can be multiple fourth time windows, which means that historical action information and historical vehicle condition information within multiple fourth time windows can be obtained for model training.

[0161] It is understood that the specific content of the sample multimodal data can be found in the aforementioned description of multimodal data, and the sample historical data can also be found in the aforementioned description of multimodal data. Further details will not be provided here.

[0162] In one possible implementation, the rating is an evaluation score of the sample DSL script's effectiveness. For example, the rating range could be set between 1 and 5 points, where a higher score indicates that the sample DSL script better conforms to human aesthetic standards. The rating can be obtained by human evaluation of the sample DSL script using the MOS (Mean Opinion Score) method, which provides a more intuitive measure of the user's actual experience with the DSL.

[0163] Specifically, the lightweight application generation device can use the sample multimodal data and the sample historical data as training input, the sample DSL script as training output, and the rating as the satisfaction level of the training output to train the initial lightweight application generative large model, thereby obtaining the lightweight application generative large model. It should be noted that the data fed to the initial lightweight application generative large model during training is solely intended to help the large model learn the syntax rules of the DSL and master appropriate layout methods for different scenarios, rather than restricting the model to select and adapt only within the range of fed data. The above mainly describes the solution provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the lightweight application generation device or electronic device includes hardware structures and / or software modules corresponding to perform each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] According to the above method, the exemplary lightweight application generation device or electronic device can divide functional modules. For example, the lightweight application generation device or electronic device may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0165] Figure 5 This is a block diagram illustrating a lightweight application generation apparatus according to an exemplary embodiment. (Refer to...) Figure 5 The lightweight application generation device includes: Acquisition module 501 is used to collect multimodal data of occupants inside the vehicle; The generation module 502 is used to input the multimodal data and vehicle configuration information into the lightweight application generative big model, identify the occupant's intention to use the in-vehicle service in the lightweight application through the lightweight application generative big model, and generate the DSL script of the lightweight application; the vehicle configuration information includes at least one of the following: vehicle model information, vehicle condition information, component configuration information, and ecosystem service configuration information; Display module 503 is used to display the interface of the lightweight application on the vehicle infotainment interface based on the DSL script; the interface includes the configuration of at least one in-vehicle service in the set of in-vehicle services covered by the lightweight application; the configuration is adapted to the usage intent.

[0166] As can be seen, by applying the scheme of this application, multimodal data of vehicle occupants, including at least two of the following: dialogue data, facial expression data, action data, physiological data, and body posture data, can be actively collected. This multimodal data is input into a lightweight application generative model, which identifies the occupant's intention to use the in-vehicle services of the lightweight application, i.e., the in-vehicle services the occupant might want to use, and generates a DSL script for the lightweight application. Based on the DSL script, the interface of the in-vehicle services that the occupant might want to use is displayed on the vehicle's infotainment system, recommending in-vehicle service interfaces to the user for selection. The interface includes the configuration of at least one in-vehicle service from the set of in-vehicle services covered by the lightweight application; the configuration is adapted to the usage intention. The DSL script for the lightweight application can be temporarily generated using real-time collected multimodal data, and then the lightweight application recommended to the user based on the DSL script can be actively generated to meet the user's usage needs. Pre-deployment of the lightweight application is not required, significantly saving system resources; only the relevant pre-stored components need to be reused when generating the lightweight application in real time. It can quickly generate lightweight application interfaces that adapt to the user's intent and include multiple in-vehicle services, and complete related configuration options, which greatly saves system resources and provides an immersive in-vehicle experience that meets the user's intent in real time.

[0167] In one possible embodiment, the DSL script includes a first DSL script for describing an in-vehicle service component and / or a second DSL script for describing a single in-vehicle service; wherein the in-vehicle service component includes a plurality of pre-configured in-vehicle services.

[0168] In one possible approach, the lightweight application generative large model generates the DSL script for the lightweight application through the following operations: The lightweight application generative large model performs semantic parsing on the multimodal data to obtain semantic parsing results; The generative big model for lightweight applications identifies the occupant’s intention to use the lightweight applications based on the vehicle configuration information and semantic parsing results. The lightweight application generative big model generates the DSL script of the lightweight application based on the usage intent.

[0169] In one possible approach, the vehicle configuration information includes at least one of the following: vehicle model information, vehicle condition information, component configuration information, and ecosystem service configuration information.

[0170] In one possible approach, the lightweight application generative big model generates the DSL script for the lightweight application based on the usage intent, including: The lightweight application generative big model generates the DSL script of the lightweight application based on the usage intent and the occupant's personalized configuration information; wherein, the personalized configuration information includes at least one of the following: interface layout preference information and in-vehicle service preference information.

[0171] In one possible approach, the acquisition module is also used for: Historical data within a second time window preceding the first time window is acquired; the historical data includes historical action information and / or historical vehicle condition information; wherein, the historical action information represents actions performed by the user on the vehicle; the first time window is the time window for acquiring the multimodal data; the length of the second time window is a first preset duration; The multimodal data is input into a lightweight application generative large model. The lightweight application generative large model identifies the occupant's intent to use the lightweight application and generates a DSL script for the lightweight application, including: The multimodal data and the historical data are input into the lightweight application generative big model. The lightweight application generative big model identifies the occupant's intention to use the lightweight application and generates the DSL script of the lightweight application.

[0172] In one possible embodiment, the device further includes: a training module; the training module is used for: Obtain a training sample set; the training samples in the training sample set include sample DSL scripts, as well as sample multimodal data and scores corresponding to the sample DSL scripts; Based on the training sample set, the initial lightweight application generative large model is trained to obtain the lightweight application generative large model.

[0173] In one possible approach, the training module is specifically used for: The initial lightweight application generative large model is trained by using the sample multimodal data and the sample historical data as training inputs, the sample DSL script as training outputs, and the rating as the satisfaction level of the training outputs, to obtain the lightweight application generative large model. In one possible approach, the display module is specifically used for: Parse the script code in the DSL script, wherein the script code includes at least one of service element description code, data source definition code, and UI layout description code; Based on the analysis results, the interface of the light application is visualized and rendered on the vehicle's infotainment system.

[0174] In one possible approach, the display module is specifically used for: Determine the in-vehicle service components associated with the parsing results; Determine the rendering order of the in-vehicle service components; Based on the rendering order, the vehicle service interfaces that connect to the vehicle service components are called sequentially. Based on the data returned by the vehicle service interfaces, the interface of the lightweight application is rendered on the vehicle infotainment system interface. The interface of the lightweight application includes a display interface for showing the service content of the vehicle service components.

[0175] In one possible approach, the apparatus further includes: a calling module, configured to, when the lightweight application described by the DSL script contains third-party ecosystem services, call an intermediate server associated with the third-party ecosystem services to access the API service interface for interfacing with the third-party ecosystem services. The display module is specifically used for: Based on the parsing results of the script code in the DSL script and the data returned by the API service interface, the interface of the lightweight application is rendered on the vehicle infotainment system interface; the interface of the lightweight application includes a display interface for showcasing the service content of the third-party ecosystem services.

[0176] In one possible approach, the acquisition module is also used for: In response to a user's denial command to the lightweight application, new multimodal data of the occupant is acquired; The generation module is further configured to: input the new multimodal data into the lightweight application generative large model, identify the deviation between the generated lightweight application and the occupant's usage needs through the lightweight application generative large model, generate the adjusted DSL script for the lightweight application, and display the adjusted interface of the lightweight application on the vehicle infotainment system based on the adjusted DSL script for the lightweight application.

[0177] In one possible embodiment, the device further includes: an adjustment module, configured to: In response to receiving user adjustments to the lightweight application, The adjustment information corresponding to the adjustment operation is input into the lightweight application generative large model, and the adjusted DSL script of the lightweight application is generated through the lightweight application generative large model. Based on the modified DSL script of the lightweight application, the modified interface of the lightweight application is displayed on the vehicle infotainment system.

[0178] In one possible embodiment, the apparatus further includes: an optimization module, configured to: Based on the DSL script corresponding to the lightweight application before adjustment and the DSL script corresponding to the lightweight application after adjustment, the generative large model of the lightweight application is optimized.

[0179] In one possible approach, the multimodal data includes at least two of the following: dialogue data, facial expression data, gesture data, physiological data, and body posture data.

[0180] In one possible approach, the generation module is specifically used for: The multimodal data is time-aligned; The time-aligned multimodal data is preprocessed and features are extracted to obtain multimodal feature information; The multimodal feature information is input into the lightweight application generative large model.

[0181] In one possible approach, a generation module is specifically used for: When the triggering conditions are met, the multimodal data is input into the lightweight application generative large model; the triggering conditions include: determining that the lightweight application generation function is enabled based on user instructions.

[0182] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Figure 6 As shown, the electronic device includes, but is not limited to, a processor 601 and a memory 602.

[0183] The memory 602 described above is used to store the executable instructions of the processor 601. It is understood that the processor 601 is configured to execute instructions to implement the lightweight application generation method in the above embodiments.

[0184] It should be noted that those skilled in the art will understand that Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device; the electronic device may include, but is not limited to, other electronic devices. Figure 6 This may indicate more or fewer components, or combinations of certain components, or different component arrangements.

[0185] Processor 601 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in memory 602, and by calling data stored in memory 602, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Processor 601 may include one or more processing units. Optionally, processor 601 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 601.

[0186] The memory 602 can be used to store software programs and various data. The memory 602 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required by at least one functional module (such as a determination unit, processing unit, etc.), etc. Furthermore, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0187] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, which can be executed by a processor 601 of an electronic device to implement the methods in the above embodiments.

[0188] Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a CD-ROM, magnetic tape, a floppy disk, and an optical data storage device.

[0189] In an exemplary embodiment, this application also provides a computer program product including one or more instructions, which can be executed by a processor of an electronic device to perform the methods described above.

[0190] It should be noted that when one or more instructions in the computer-readable storage medium or computer program product are executed by the processor of an electronic device, they implement the various processes of the above method embodiments and achieve the same technical effect as the above method. To avoid repetition, they will not be described again here.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0195] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0196] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating lightweight applications, characterized in that, The method includes: Collect multimodal data of occupants inside the vehicle; the multimodal data includes at least two of the following: dialogue data, facial expression data, motion data, physiological data, and body posture data; The multimodal data and vehicle configuration information are input into the lightweight application generative big model. The lightweight application generative big model identifies the occupant's intention to use the in-vehicle services in the lightweight application and generates the DSL script of the lightweight application. The vehicle configuration information includes at least one of the following: vehicle model information, vehicle condition information, component configuration information, and ecosystem service configuration information. Based on the DSL script, the interface of the lightweight application is displayed on the vehicle infotainment system; the interface includes the configuration of at least one in-vehicle service from the set of in-vehicle services covered by the lightweight application; the configuration is adapted to the intended use.

2. The method according to claim 1, characterized in that, The DSL script includes a first DSL script for describing in-vehicle service components and / or a second DSL script for describing a single in-vehicle service; wherein the in-vehicle service components include multiple pre-configured in-vehicle services.

3. The method according to claim 1, characterized in that, The lightweight application generative large model generates the DSL script of the lightweight application through the following operations: The lightweight application generative large model performs semantic parsing on the multimodal data to obtain semantic parsing results; The lightweight application generative big model identifies the usage intent based on the vehicle configuration information and semantic parsing results. The lightweight application generative big model generates the DSL script of the lightweight application based on the usage intent.

4. The method according to claim 3, characterized in that, The lightweight application generative big model generates the DSL script for the lightweight application based on the usage intent, including: The lightweight application generative big model generates the DSL script of the lightweight application based on the usage intent and the occupant's personalized configuration information; wherein, the personalized configuration information includes at least one of the following: interface layout preference information and in-vehicle service preference information.

5. The method according to claim 1, characterized in that, Before inputting the multimodal data into a lightweight generative large model, the method further includes: Historical data within a second time window preceding the first time window is acquired; the historical data includes historical action information and / or historical vehicle condition information; wherein, the historical action information represents actions performed by the user on the vehicle; the first time window is the time window for acquiring the multimodal data; the length of the second time window is a first preset duration; The multimodal data is input into a lightweight application generative large model. The lightweight application generative large model identifies the passenger's intent to use in-vehicle services within the lightweight application, and generates the lightweight application's DSL script, including: The multimodal data and the historical data are input into the lightweight application generative big model. The lightweight application generative big model identifies the usage intent and generates the DSL script of the lightweight application.

6. The method according to claim 5, characterized in that, The method includes: Obtain a training sample set; the training samples in the training sample set include sample DSL scripts, as well as sample multimodal data and scores corresponding to the sample DSL scripts; Based on the training sample set, the initial lightweight application generative large model is trained to obtain the lightweight application generative large model.

7. The method according to claim 6, characterized in that, The training samples also include: historical sample data within a fourth time window preceding the third time window; the historical sample data includes historical sample action information and / or historical sample vehicle condition information; wherein, the third time window is the time window for acquiring the multimodal sample data; and the length of the fourth time window is a second preset duration.

8. The method according to claim 7, characterized in that, The step of training the initial lightweight application generative large model based on the training sample set to obtain the lightweight application generative large model includes: The initial lightweight application generative large model is trained by using the sample multimodal data and the sample historical data as training inputs, the sample DSL script as training outputs, and the rating as the satisfaction level of the training outputs, to obtain the lightweight application generative large model.

9. The method according to claim 1, characterized in that, The step of displaying the lightweight application interface on the vehicle infotainment system based on the DSL script includes: Parse the script code in the DSL script, wherein the script code includes at least one of service element description code, data source definition code, and UI layout description code; Based on the analysis results, the interface of the light application is visualized and rendered on the vehicle's infotainment system.

10. The method according to claim 9, characterized in that, The visualization rendering based on the parsing results on the vehicle's infotainment interface includes: Determine the in-vehicle service components associated with the parsing results; Determine the rendering order of the in-vehicle service components; Based on the rendering order, the vehicle service interfaces that connect to the vehicle service components are called sequentially. Based on the data returned by the vehicle service interfaces, the interface of the lightweight application is rendered on the vehicle infotainment system interface. The interface of the lightweight application includes a display interface for showing the service content of the vehicle service components.

11. The method according to claim 9, characterized in that, The method further includes: In the case where the lightweight application described by the DSL script includes third-party ecosystem services, the intermediate server associated with the third-party ecosystem services is invoked to access the API service interface used to connect to the third-party ecosystem services. The visualization rendering based on the parsing results on the vehicle's infotainment interface to display the interface of the lightweight application includes: Based on the parsing results of the script code in the DSL script and the data returned by the API service interface, the interface of the lightweight application is rendered on the vehicle infotainment system interface; the interface of the lightweight application includes a display interface for showcasing the service content of the third-party ecosystem services.

12. The method according to claim 1, characterized in that, The method further includes: In response to a user's denial command to the lightweight application, new multimodal data of the occupant is acquired; The new multimodal data is input into the lightweight application generative big model. The lightweight application generative big model identifies the deviation between the generated lightweight application and the occupant's usage needs, and generates the adjusted DSL script for the lightweight application. Based on the modified DSL script of the lightweight application, the modified interface of the lightweight application is displayed on the vehicle infotainment system.

13. The method according to claim 1, characterized in that, The method further includes: In response to receiving a user's adjustment operation for the lightweight application, the adjustment information corresponding to the adjustment operation is input into the lightweight application generative big model, and the adjusted DSL script of the lightweight application is generated through the lightweight application generative big model. Based on the modified DSL script of the lightweight application, the modified interface of the lightweight application is displayed on the vehicle infotainment system.

14. The method according to claim 12 or 13, characterized in that, The method further includes: Based on the DSL script corresponding to the lightweight application before adjustment and the DSL script corresponding to the lightweight application after adjustment, the generative large model of the lightweight application is optimized.

15. The method according to claim 1, characterized in that, The step of inputting the multimodal data into a lightweight application generative large model includes: The multimodal data is time-aligned; The time-aligned multimodal data is preprocessed and features are extracted to obtain multimodal feature information; The multimodal feature information is input into the lightweight application generative large model.

16. The method according to claim 1, characterized in that, The step of inputting the multimodal data into a lightweight application generative large model includes: When the triggering conditions are met, the multimodal data is input into the lightweight application generative large model; The triggering conditions include: determining that the lightweight application generation function is enabled based on user instructions.

17. A lightweight application generation device, characterized in that, The device includes: The acquisition module is used to collect multimodal data of occupants inside the vehicle; the multimodal data includes at least two of the following: dialogue data, facial expression data, motion data, physiological data, and body posture data; The generation module is used to input the multimodal data and vehicle configuration information into the lightweight application generative big model, identify the occupant's intention to use the in-vehicle services in the lightweight application through the lightweight application generative big model, and generate the DSL script of the lightweight application; the vehicle configuration information includes at least one of the following: vehicle model information, vehicle condition information, component configuration information, and ecosystem service configuration information; The display module is used to display the interface of the lightweight application on the vehicle infotainment system based on the DSL script; the interface includes the configuration information of at least one in-vehicle service in the set of in-vehicle services covered by the lightweight application; the configuration information is adapted to the usage intent.

18. A vehicle, characterized in that, The vehicle includes the device as described in claim 17.

19. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1-16.

Images