Model training data generation method, device and equipment of in-vehicle infotainment system, storage medium and program product

Abstract

The invention relates to a model training data generation method, device and equipment of an in-vehicle infotainment system, a storage medium and a program product. The method comprises the following steps: acquiring operation record data of a human-computer interaction interface of the vehicle-mounted infotainment system; the operation record data comprises an operation track corresponding to a user operation target and an interface image associated with each interactive operation in the operation track; according to the operation record data, generating semantic description information of the interaction operation, and extracting a function hierarchical relationship among the interface elements from the semantic description information; based on the function hierarchical relationship, generating a target knowledge base of the in-vehicle infotainment system; the target knowledge base is used for recording each function module in the vehicle-mounted terminal system and the mutual relation among the function modules; generating model training data according to the target knowledge base; the model training data is used for training an artificial intelligence model carried by the in-vehicle infotainment system. By adopting the method, the model training data quality of the in-vehicle infotainment system can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating model training data for a vehicle infotainment system. Background Technology

[0002] With the rapid development of intelligent connected vehicle technology, the functions of vehicle systems (such as in-vehicle infotainment systems) are becoming increasingly complex, and the interaction methods are shifting from traditional physical buttons to touch, voice, and multimodal interaction. To train a vehicle system model that can understand user intent and automatically execute operations, massive amounts of training data are required.

[0003] Traditional methods for acquiring training data for in-vehicle infotainment systems are often quite simplistic. For example, they might involve collecting screenshots and establishing a simple correspondence between actual click locations, then using this correspondence to generate training data. In-vehicle infotainment system models trained with this method often exhibit poor generalization ability and struggle to handle complex user commands. Therefore, current methods for generating training data for in-vehicle infotainment systems suffer from poor model training data quality. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for generating model training data for vehicle infotainment systems that can improve the quality of model training data for vehicle infotainment systems, in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for generating model training data for an in-vehicle infotainment system, including:

[0006] The system acquires operation record data of the human-machine interface of the vehicle system; the operation record data includes the operation trajectory corresponding to the user's operation target, and the interface image associated with each interaction operation in the operation trajectory;

[0007] Based on the operation record data, semantic description information of the interaction operation is generated, and the functional hierarchy relationship between each interface element is extracted from the semantic description information; the interface element includes elements associated with the interface changes caused by the interaction operation.

[0008] Based on the aforementioned functional hierarchy, a target knowledge base for the vehicle infotainment system is generated; the target knowledge base is used to record each functional module in the vehicle infotainment system, as well as the relationships between the functional modules.

[0009] Based on the target knowledge base, model training data is generated; the model training data is used to train the artificial intelligence model mounted on the vehicle system.

[0010] In one embodiment, generating semantic description information of the interactive operation based on the operation record data includes:

[0011] The operation record data is input into a multimodal large model; wherein, the multimodal large model is used to determine the corresponding operation description information based on the interaction operation; the multimodal large model is also used to determine interface change information based on the differences between the interface images associated with the interaction operation; the multimodal large model is also used to determine the progress description information of the interaction operation relative to the user operation target based on the interaction operation and the user operation target;

[0012] The semantic description information is generated by integrating the operation description information, the interface change information, and the progress description information.

[0013] In one embodiment, extracting the functional hierarchy relationship between interface elements from the semantic description information includes:

[0014] The multimodal large model extracts relationship tuples between the interface elements from the semantic description information; the relationship tuples include the parent-child functional affiliation relationships between the interface elements.

[0015] Based on the relationship tuples between the interface elements, a functional hierarchy relationship between the interface elements is generated.

[0016] In one embodiment, the target knowledge base of the vehicle system is generated based on the functional hierarchy relationship:

[0017] An initial knowledge base with a directed graph structure is constructed; the nodes in the directed graph structure of the initial knowledge base are used to represent the interface elements of the vehicle system, and the edges in the directed graph structure are used to represent the jump paths between the nodes.

[0018] Based on the functional hierarchy, the directed graph structure of the initial knowledge base is updated to obtain the target knowledge base.

[0019] In one embodiment, generating model training data based on the target knowledge base includes:

[0020] The jump paths between the nodes in the target knowledge base are reorganized to generate new operation trajectories;

[0021] The model training data is generated based on the user operation target associated with the new operation trajectory, the interactive operations included in the new operation trajectory, and the interface images associated with each interactive operation.

[0022] In one embodiment, generating the model training data based on the user operation target associated with the new operation trajectory, each interactive operation included in the new operation trajectory, and the interface image associated with each interactive operation includes:

[0023] Based on the user operation target associated with the new operation trajectory, the interactive operations included in the new operation trajectory, and the interface images associated with each interactive operation, new operation record data is generated.

[0024] Based on the new operation record data, semantic description information for each interactive operation is generated;

[0025] The model training data is generated based on the new operation record data and the semantic description information of each interaction operation.

[0026] Secondly, this application also provides a model training data generation device for a vehicle infotainment system, comprising:

[0027] The data acquisition module is used to acquire operation record data of the human-machine interaction interface of the vehicle system; the operation record data includes the operation trajectory corresponding to the user's operation target, and the interface image associated with each interaction operation in the operation trajectory;

[0028] The relationship extraction module is used to generate semantic description information of the interactive operation based on the operation record data, and to extract the functional hierarchy relationship between each interface element from the semantic description information; the interface element includes elements associated with the interface changes caused by the interactive operation.

[0029] A knowledge construction module is used to generate a target knowledge base for the vehicle system based on the functional hierarchy; the target knowledge base is used to record each functional module in the vehicle system and the relationships between the functional modules.

[0030] The training data generation module is used to generate model training data based on the target knowledge base; the model training data is used to train the artificial intelligence model mounted on the vehicle system.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0032] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0033] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0034] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for generating model training data for in-vehicle infotainment systems acquire operation record data of the human-machine interface of the in-vehicle infotainment system. This operation record data includes the operation trajectory corresponding to the user's operation target and the interface image associated with each interaction operation in the operation trajectory. Semantic description information of the interaction operations is generated based on the operation record data, and the functional hierarchy relationships between various interface elements are extracted from the semantic description information. Then, a target knowledge base for the in-vehicle infotainment system is generated based on the functional hierarchy relationships, and finally, model training data for the in-vehicle infotainment system is generated using the knowledge base. Thus, from collecting operation record data to semantic parsing and relational reasoning of single-step interaction operations, and then to the construction of a hierarchical knowledge base, training data with rich functional semantics and operational logic can be generated, improving the quality and diversity of model training data for in-vehicle infotainment systems. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is an application environment diagram of a model training data generation method for a vehicle infotainment system in one embodiment.

[0037] Figure 2 This is a flowchart illustrating a method for generating model training data for a vehicle infotainment system in one embodiment.

[0038] Figure 3 This is a logic diagram of a method for generating model training data for a vehicle infotainment system in one embodiment;

[0039] Figure 4 This is a flowchart illustrating a method for generating model training data for a vehicle infotainment system in another embodiment.

[0040] Figure 5 This is a structural block diagram of a model training data generation device for a vehicle infotainment system in one embodiment.

[0041] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] The model training data generation method for vehicle infotainment systems provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0044] Terminal 102 acquires operation record data of the human-machine interface of the vehicle system; the operation record data includes the operation trajectory corresponding to the user's operation target, and the interface image associated with each interaction operation in the operation trajectory; based on the operation record data, terminal 102 generates semantic description information of the interaction operation, and extracts the functional hierarchy relationship between each interface element from the semantic description information; the interface element includes elements associated with the interface changes caused by the interaction operation; based on the functional hierarchy relationship, terminal 102 generates a target knowledge base for the vehicle system; the target knowledge base is used to record each functional module in the vehicle system, and the interrelationship between each functional module; based on the target knowledge base, terminal 102 generates model training data; the model training data is used to train the artificial intelligence model installed in the vehicle system.

[0045] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and in-vehicle devices. The server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0046] In one exemplary embodiment, such as Figure 2 As shown, a method for generating model training data for a vehicle infotainment system is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes:

[0047] Step S202: Obtain operation record data of the human-machine interface of the vehicle system.

[0048] The in-vehicle infotainment system refers to the system installed in a vehicle, providing users with functions such as navigation, entertainment, communication, and vehicle control. An in-vehicle infotainment system typically includes a display screen, a user interface, and various functional modules. The human-machine interface (HMI) is the interface provided by the in-vehicle infotainment system for users to interact with, allowing them to exchange information and operate functions.

[0049] The operation log data can include a series of operation records generated when performing a specific task on the human-machine interface of the vehicle system, including the operation trajectory corresponding to the user's operation target, and the interface image associated with each interaction operation in the operation trajectory. In specific implementation, the operation log data can be collected in a real vehicle environment or in a simulated vehicle system environment, wherein the simulated vehicle system environment can be the operating environment of the vehicle system simulated by software.

[0050] The operation log data includes several components: user operation target, operation trajectory, and interface image.

[0051] In this context, the user's operation goal refers to the ultimate purpose that the user expects to achieve by operating the vehicle's infotainment system. User operation goals can take the form of high-level commands, such as "turn on sport mode," "adjust screen brightness," or "play music." User operation goals describe the user's intent without specifying the exact steps involved.

[0052] In this context, an operation trajectory refers to a sequence of interactive operations performed to achieve a user's operational goal. The operation trajectory records the entire process from start to finish, chronologically. In practical applications, each interactive operation in the operation trajectory can be an atomic-level action.

[0053] Interactive operations refer to the basic operational units by which users interact with the vehicle's infotainment system interface. Interactive operations can include clicks, swipes, long presses, and multi-finger gestures. For clicks, the interactive operation records the screen coordinates of the click. For swipes, it records the swipe vector. Interactive operations can also include button events, such as pressing a physical or virtual button.

[0054] Here, "interface image" refers to a screenshot of the interface displayed on the vehicle's infotainment system screen during an interactive operation. The interface image associated with each interactive operation in the operation trajectory includes the interface image before and after the operation. Before and after each interactive operation, screenshots of the vehicle's interface can be taken and saved as interface images. By comparing the interface images before and after the interactive operation, the interface changes caused by the interactive operation can be observed.

[0055] For example, the user's operation goal is "turn on sports mode," and the operation trajectory includes a first interaction, a second interaction, and a third interaction. The first interaction is clicking the "Settings" icon at coordinates (100, 200). The terminal takes a screenshot of the vehicle's interface before performing the first interaction to obtain the image before the first operation, and takes a screenshot of the interface after performing the first interaction to obtain the image after the first operation. The second interaction is clicking the "Driving" menu item at coordinates (150, 300). The terminal takes a screenshot of the vehicle's interface before performing the second interaction to obtain the image before the second operation, and takes a screenshot of the interface after performing the second interaction to obtain the image after the second operation. The third interaction is clicking the "Sports Mode" option at coordinates (200, 400). The terminal takes a screenshot of the vehicle's interface before performing the third interaction to obtain the image before the third operation, and takes a screenshot of the interface after performing the third interaction to obtain the image after the third operation. The terminal can combine the user's operation goal, the operation trajectory, and the interface images associated with each interaction to form a structured operation record data.

[0056] Optionally, the operation log data may also include a completion status flag, which is used to record whether each interactive operation leads to the end of the task (such as completing the user's operation goal) or to an intermediate state. For example, when an interactive operation is only an intermediate step in the process of achieving the user's operation goal, the completion status flag of the interactive operation can be set to an intermediate state.

[0057] Step S204: Based on the operation record data, generate semantic description information of the interactive operation, and extract the functional hierarchy relationship between each interface element from the semantic description information.

[0058] In practice, semantic description information of interactive operations is generated based on operation record data. This can be done by generating semantic description information of interactive operations based on interface images, interactive operations, and user operation goals.

[0059] Semantic description information refers to the high-level semantic interpretation of interactive operations. It includes not only the literal description of the interactive operation, but also its functional meaning, its impact on the interface, and its role in achieving the user's operational goals. Semantic description information transforms low-level operational coordinates into human-understandable semantic expressions.

[0060] For example, semantic description information can be represented as a quadruple (I t I t+1 A t , G), where I t Image before operation, I t+1 The image after the operation, together with the image after the operation, constitutes the interface image associated with the interactive operation; A tFor interactive operations, the coordinates (x, y) on the interface can be used to represent the interactive operation; G is the user's target operation.

[0061] Interface elements include elements associated with interface changes caused by interactive operations. In specific implementations, interface elements include various interactive or displayable components in the vehicle infotainment system interface. For example, interface elements can include the entire page, buttons, menu items, icons, text boxes, sliders, etc.

[0062] Among them, the elements associated with the interface changes caused by interactive operations refer to the interface elements that change in the interface image or participate in the interaction before and after the interactive operation is performed.

[0063] In practical applications, semantic description information can be generated for each interactive operation in the operation trajectory, and semantic description information for each interactive operation can be generated sequentially according to the execution order of the interactive operations in the operation trajectory.

[0064] In this context, functional hierarchy refers to the subordinate or containment relationships between interface elements. Functional hierarchy reflects the organizational structure between different functional modules in the vehicle infotainment system. For example, functional hierarchy can be represented as a parent-child relationship, where a parent function contains child functions, and child functions belong to the parent function.

[0065] In practical implementation, extracting the functional hierarchy between interface elements from the semantic description information of interactive operations can involve analyzing descriptions of interface transitions, menu expansions, and option selections to identify the parent-child relationships between different interface elements. For example, if the semantic description information is: clicking the "Driving" menu item leads to an interface containing the "Sports Mode" option, the functional hierarchy relationship of the "Sports Mode" function belonging to the "Driving" menu can be extracted from this semantic description information. Optionally, the extracted functional hierarchy relationship can be represented as a relational tuple in the form of (child function, parent function), such as (Sports Mode, Driving) or (Driving, Settings).

[0066] Optionally, when generating semantic description information, the progress stage of the interactive operation in achieving the user's operation goal can also be considered. The terminal analyzes the relationship between the current interactive operation and the user's operation goal to determine whether the interactive operation is an initial step, an intermediate step, or a final step. Then, the progress information is integrated into the semantic description information, for example, "This interactive operation has completed the user's operation goal."

[0067] Step S206: Generate the target knowledge base of the vehicle system based on the functional hierarchy.

[0068] The target knowledge base is used to record the functional modules of the vehicle infotainment system and the relationships between them. Specifically, the target knowledge base is a structured database that stores the functional modules and their relationships within the vehicle infotainment system. The target knowledge base organizes data using a specific data structure, recording the overall functional architecture and operational logic of the vehicle infotainment system. The target knowledge base supports operations such as querying, updating, and reasoning about functional modules and relationships.

[0069] In this context, a functional module refers to a component or interface in the vehicle's infotainment system that provides a specific function. A functional module can be a menu, such as a settings menu or display settings menu; it can also be a specific function option, such as a sport mode or brightness adjustment; or it can be a specific function interface. Functional modules are represented in the target knowledge base as nodes or entries.

[0070] The interrelationships between functional modules refer to the ways in which functional modules are associated. Interrelationships can include inclusion, hierarchical, jump, and dependency relationships.

[0071] In practice, a target knowledge base is generated based on the functional hierarchy extracted from the operation log data. For example, each interface element can be treated as a functional module in the target knowledge base. For instance, a "Home Screen" module, a "Settings" module, a "Driving" module, and a "Sports Mode" module can be created, and then relationships can be established between these modules. For example, a hierarchical relationship can be established between the "Driving" module and the "Settings" module.

[0072] In one embodiment, the target knowledge base can be implemented using a graph database, relational database, or knowledge graph. Graph databases use nodes and edges to store data, making them suitable for representing complex relational networks. Relational databases use tables to store data, making them suitable for structured data management. Knowledge graphs are semantic networks that use entities and relations to represent knowledge, making them suitable for semantic reasoning and knowledge retrieval.

[0073] In one embodiment, the target knowledge base can be represented using a directed graph structure. Each functional module of the vehicle system can be treated as a node in the target knowledge base, and directed edges can be established between nodes based on functional hierarchy. These edges represent the membership or navigation relationships between functions, such as "driving belongs to settings" or "clicking settings navigates to driving". For example, for a functional hierarchy represented as a relational tuple (driving, settings), a directed edge can be established between the "driving" node and the "settings" node, indicating that the "driving" function belongs to the "settings" function.

[0074] In practice, the extracted functional hierarchy can be integrated into the target knowledge base as knowledge entries. Through deduplication and conflict resolution, a unified hierarchical functional tree diagram covering all software functions in the vehicle system can be gradually constructed.

[0075] In practice, the terminal extracts functional hierarchy relationships from multiple operation records and generates a target knowledge base based on these relationships. When functional hierarchy relationships extracted from different operation records are duplicated, the terminal performs deduplication, retaining only one set of relationships in the knowledge base. When functional hierarchy relationships extracted from different operation records conflict, the terminal uses a conflict resolution mechanism. Optionally, the conflict resolution mechanism can employ a voting mechanism. When multiple operation records extract different functional hierarchy relationships for the same pair of interface elements, the frequency of each functional hierarchy relationship can be counted, and the functional hierarchy relationship with the highest frequency can be selected as the final result.

[0076] In practice, the content of the target knowledge base can be continuously iterated and updated during its construction. When new operation record data is acquired, functional hierarchy relationships can be extracted from this data and integrated into the existing knowledge base. If the newly extracted functional hierarchy relationships involve interface elements that do not yet exist in the knowledge base, corresponding new nodes can be created. If the newly extracted functional hierarchy relationships involve new relationships between existing interface elements in the knowledge base, corresponding new edges can be created. Through iterative updates, the knowledge base gradually covers all functional modules of the vehicle infotainment system and their interrelationships.

[0077] Step S208: Generate model training data based on the target knowledge base.

[0078] Among them, the model training data is used to train the artificial intelligence model installed in the vehicle system.

[0079] In practice, the terminal can generate model training data by utilizing the built target knowledge base in two ways: the first is to annotate and enhance the original operation record data, and the second is to synthesize new training data based on the target knowledge base.

[0080] In the first approach, the interactive operations and their coordinate information from the original operation record data can be obtained, and then the original interactive operation A can be replaced or supplemented using the previously generated semantic description information. t A t This provides the coordinate representation (x, y) of interactive operations on the interface, thereby generating high-quality training data (I). t I t+1 A tLet G, T), where T represents semantic description information. This allows for the generation of highly semantic action descriptions from the original operation record data.

[0081] For example, the original operation log data records the interaction as "click coordinates (100, 200)", and the semantic description is "click the settings icon located at coordinates (100, 200) to enter the settings menu". Then, the user operation target, interface image, interaction, and semantic description are combined to form new operation log data, which is used to generate model training data. This model training data includes instructions (user operation target), images (interface images), actions (interactions), and thought processes (semantic description information).

[0082] In the second approach, if the target knowledge base is represented using a directed graph structure, new operation paths can be generated based on the nodes and edges in the target knowledge base. Specifically, a starting node and a target node can be selected in the target knowledge base, and by traversing the edges between the nodes, a jump path from the starting node to the target node can be found. This jump path constitutes a new operation trajectory. For example, if the target knowledge base contains nodes such as "Home Screen," "Settings," "Display," and "Brightness," as well as edges connecting these nodes, these nodes and edges can be combined to generate an operation path from "Home Screen" to "Brightness": Home Screen -> Settings -> Display -> Brightness. Therefore, based on the rich jump paths in the target knowledge base, new and unseen high-level instructions can be automatically generated. Even if there are no instructions to directly adjust brightness in the original operation record data, a new instruction to "adjust screen brightness" and its corresponding operation trajectory can be automatically synthesized based on the target knowledge base. This allows for reverse reasoning to synthesize new high-level instruction data and infer the various interactive operations and interface images it contains to generate new training data. Thus, new and unseen instruction-operation sequence data can be automatically generated using the paths in the target knowledge base, greatly enhancing the diversity of data and the generalization ability of the model.

[0083] Furthermore, semantic description information can be generated for each interactive operation in the newly synthesized operation trajectory. Specifically, based on the jump relationships between nodes in the operation path and the user's operation target, the semantic description information of each interactive operation is inferred. Then, the newly synthesized user operation target, each interactive operation, the interface image associated with each interactive operation, and the semantic description information of each interactive operation are combined to generate new model training data.

[0084] Therefore, by collecting structured operation records, performing single-step operation semantic parsing and relational reasoning, and then constructing and iteratively updating a hierarchical knowledge base, a complete process of knowledge base-based data synthesis and enhancement is achieved. This process transforms the original, low-semantic operation coordinate sequences into high-semantic, logic-rich training data pairs, improving the quality of model training data generation. Furthermore, it automatically constructs high-quality data covering all interfaces and operation paths, eliminating the need for extensive manual instruction writing and annotation, thus improving the efficiency and intelligence of model training data generation.

[0085] In the aforementioned method for generating model training data for the vehicle infotainment system, operation record data, including user operation targets, operation trajectories, and interface images, is acquired from the human-machine interface of the vehicle infotainment system. This operation record data includes the operation trajectory corresponding to the user operation target and the interface image associated with each interaction operation in the operation trajectory. Semantic description information of the interaction operations is generated based on the interface images, interaction operations, and user operation targets in the operation record data. The functional hierarchy relationships between various interface elements are extracted from the semantic description information, and a target knowledge base for the vehicle infotainment system is constructed based on these functional hierarchy relationships. Finally, the model training data for the vehicle infotainment system is generated using this knowledge base. Thus, from collecting operation trajectory data and operation record data, to semantic parsing and relational reasoning of single-step interaction operations, and then to the construction of the hierarchical knowledge base, training data with rich functional semantics and operational logic can be generated, improving the quality and diversity of model training data for the vehicle infotainment system.

[0086] In another embodiment, semantic description information of interactive operations is generated based on operation record data, including: inputting operation record data into a multimodal large model; and fusing operation description information, interface change information, and progress description information to generate semantic description information.

[0087] Among them, the multimodal large model is used to determine the corresponding operation description information based on the interaction operation; the multimodal large model is also used to determine the interface change information based on the differences between the interface images associated with the interaction operation; the multimodal large model is also used to determine the progress description information of the interaction operation relative to the user's operation goal based on the interaction operation and the user's operation goal.

[0088] In practical implementation, a multimodal large model refers to a deep learning model capable of processing data from multiple modalities. A multimodal large model can receive image and text data as input and output analysis results in text form. Having undergone large-scale pre-training, a multimodal large model possesses the ability to understand image content and generate text semantics.

[0089] For example, suppose the semantic description information is a quadruple (I t I t+1 At G).

[0090] The operation description information can refer to a description of the interactive operation itself. It specifies what type of operation was performed, such as a click; it also specifies the object of the operation, for example, in I... t In the diagram, what UI element corresponds to the coordinate (x, y) position, such as a driving button?

[0091] The interface change information can include what changes in the interface state were caused by the interactive operation. This information describes the appearance, disappearance, movement, and state changes of interface elements. For example, interface change information could be "the interface jumps from the home screen to the settings menu" or "the motion mode option changes from unselected to selected."

[0092] In its implementation, the multimodal large model analyzes the differences between the interface images associated with the interactive operation, i.e., it compares and analyzes the interface images before and after the operation. It is not limited to simple image descriptions, but explicitly infers the implicit hierarchy and jump relationship between interface elements to generate semantically rich operation descriptions.

[0093] The progress description information refers to the current stage of an interactive operation in achieving the user's goal. It explains the role of the current interactive operation in ultimately achieving the user's goal. For example, the progress description information could be "This operation is the first step in achieving the goal" or "This operation has completed the user's goal."

[0094] In one embodiment, the input to a multimodal large model can be optimized using prompting engineering techniques. Prompt engineering refers to designing specific prompt templates to guide the model in generating the desired output. Specifically, prompt templates containing specific instructions can be designed, such as "Please analyze the differences between the following two images, and explain the object of the operation, the action performed, the interface changes, and the progress of the operation in achieving the goal, based on the coordinate positions." The prompt template, user operation target, interaction coordinates, before-operation image, and after-operation image are input together into the multimodal large model. Guided by the prompt template, the multimodal large model can analyze the input data and generate structured output results in the expected manner.

[0095] In specific implementation, when generating semantic description information, operation description information, interface change information, and progress description information are fused in a logical order. Optionally, semantic description information can be generated through a multimodal large model. Specifically, the multimodal large model first describes what operation was performed, obtaining operation description information; then it describes the interface changes caused by the operation, obtaining interface change information; finally, it describes the role of the operation in achieving the user's operation goal, obtaining progress description information; finally, the operation description information, interface change information, and progress description information are fused to obtain semantic description information. The semantic description information generated by the multimodal large model has a clear logical structure, facilitating the subsequent extraction of functional hierarchy relationships.

[0096] The technical solution of this embodiment inputs operation record data into a multimodal large model, and uses the image understanding and text generation capabilities of the multimodal large model to determine operation description information, interface change information and progress description information respectively. Then, these three parts of information are integrated to generate semantic description information containing rich semantic content, which improves the reliability and accuracy of subsequent extraction of functional hierarchy relationships.

[0097] In another embodiment, extracting the functional hierarchy relationship between interface elements from the semantic description information includes: extracting the relationship tuple between interface elements from the semantic description information through a multimodal large model; the relationship tuple includes the parent-child functional attribution relationship between interface elements; and generating the functional hierarchy relationship between interface elements based on the relationship tuple between interface elements.

[0098] In practical applications, after determining the semantic description information, the multimodal large model can be further guided to perform reasoning and extract the membership relationship between interface elements, that is, the functional hierarchy relationship.

[0099] In this context, a relational tuple is a data structure that represents the relationship between two UI elements. Relational tuples are typically represented as binary tuples, such as (child function, parent function), indicating that the child function belongs to the parent function. Relational tuples clearly define the dependency or containment relationship between UI elements.

[0100] In this context, the parent-child function relationship refers to the hierarchical relationship between interface elements. A parent function is a higher-level functional module, and a child function is a lower-level functional module. Child functions belong to and can be accessed through the parent function. For example, "Sports Mode" is a child function, and "Driving" is the parent function; a parent-child function relationship exists between them.

[0101] For example, the semantic description information is "click the driving menu item in the settings menu". It can be inferred that the "driving" menu belongs to the "settings" menu. Therefore, the relation tuple (driving, settings) can be extracted.

[0102] In its implementation, the multimodal large model analyzes keywords and phrases in semantic descriptions when extracting relation tuples. It identifies keywords indicating inclusion relationships, such as "contains," "belongs to," "located in," and "in." It also identifies keywords indicating interface navigation, such as "jump to," "enter," and "open." Based on these keywords, the multimodal large model determines the relationship type between interface elements and extracts the corresponding relation tuples.

[0103] When extracting relational tuples, multimodal large models can use specific prompt templates to guide the model output. For example, a prompt template can be designed such as "Please extract the parent-child function relationship between interface elements from the following description and output it in the form of (child function, parent function)". Then, the prompt template and semantic description information are input into the multimodal large model. The multimodal large model outputs structured relational tuples according to the requirements of the prompt template.

[0104] For example, the extracted relational tuples include: (Sports Mode, Driving), (Driving, Settings), (Settings, Home Screen), etc., and then a functional hierarchy relationship is constructed based on these relational tuples: Home Screen -> Settings -> Driving -> Sport Mode.

[0105] In practical applications, a tree structure can be constructed based on relation tuples to represent the functional hierarchy. Specifically, the root node of the tree structure is the top-level interface element, such as "Home Screen". The internal nodes of the tree structure are intermediate-level interface elements, such as "Settings" and "Driving". The leaf nodes of the tree structure are the bottom-level interface elements, such as "Sports Mode". Connections are established between parent and child nodes based on relation tuples to form a complete functional hierarchy tree.

[0106] Optionally, when different relational tuples specify different parent functions for the same sub-function, a conflict resolution strategy can be adopted. The conflict resolution strategy can be to prioritize the parent-child relationship that occurs most frequently.

[0107] The technical solution of this application embodiment extracts relational tuples representing the parent-child functional attribution relationship from semantic description information through a multimodal large model, and then generates functional hierarchical relationships between interface elements based on these relational tuples, making the hierarchical relationship implicit in the semantic description explicit, constructing a clear functional hierarchical structure, and providing a comprehensive and high-quality data foundation for subsequent knowledge base construction and training data generation.

[0108] In another embodiment, a target knowledge base for the vehicle system is generated based on the functional hierarchy: an initial knowledge base with a directed graph structure is constructed; the nodes in the directed graph structure of the initial knowledge base are used to represent the interface elements of the vehicle system, and the edges in the directed graph structure are used to represent the jump paths between nodes; the directed graph structure of the initial knowledge base is updated according to the functional hierarchy to obtain the target knowledge base.

[0109] The initial knowledge base refers to the basic knowledge base built before integrating the functional hierarchy. The initial knowledge base provides the basic structural framework of the knowledge base, which is subsequently updated and improved through the integration of functional hierarchy relationships. The initial knowledge base can be empty or it can contain some predefined basic nodes and edges.

[0110] A directed graph structure is a graph data structure composed of nodes and directed edges. Edges in a directed graph are directional, pointing from one node to another. Directed graphs can represent directional relationships, such as navigation from one interface to another, or hierarchical relationships from sub-functions to parent functions. For example, a node can be created for each interface element in a vehicle infotainment system. Interface elements can represent the functional modules of the system, and each node stores information such as the name, type, and attributes of the interface element. The edges in the directed graph structure represent the navigation paths between nodes.

[0111] The navigation path refers to the route a user takes when navigating from one interface to another within the vehicle's infotainment system. A navigation path consists of a series of interface elements and the relationships between them. For example, the navigation path from the home screen to Sport mode could be: Home screen -> Settings -> Driving -> Sport mode.

[0112] In practical implementation, the directed graph structure of the initial knowledge base is updated based on the functional hierarchy. This can be achieved by merging the functional hierarchy into the directed graph structure of the initial knowledge base. Assuming the functional hierarchy is represented by a relation tuple (Sports Mode, Driving), we can check if the "Sports Mode" and "Driving" nodes already exist in the directed graph. If the nodes exist, they can be used directly; if they do not exist, new nodes can be created. A directed edge is created between the "Driving" and "Sports Mode" nodes, indicating that the "Driving" menu includes the "Sports Mode" function. By continuously merging new functional hierarchy relationships, the directed graph gradually expands, eventually covering all functional modules and navigation paths of the vehicle's infotainment system.

[0113] For example, during the process of updating the directed graph structure of the initial knowledge base according to the functional hierarchy, the relationship types of the edges can be labeled. Relationship types can include attribution relationships and navigation relationships. An attribution relationship represents a static inclusion relationship where a sub-function belongs to a parent function. A navigation relationship represents a dynamic navigation relationship where a user navigates from one interface to another. The relationship type of the edges can be determined and labeled based on the semantic meaning of the functional hierarchy.

[0114] For example, during the process of updating the directed graph structure of the initial knowledge base according to the functional hierarchy, if a jump path corresponding to a certain functional hierarchy already exists in the directed graph, the weight or confidence of that jump path can be updated. The weight of the jump path indicates the frequency or importance of the path being used. When multiple operation record data involve the same jump path, the weight of that path increases, indicating that it is a frequently used path.

[0115] For example, during the process of updating the directed graph structure of the initial knowledge base according to the functional hierarchy, conflict detection and resolution can also be performed. When a new functional hierarchy conflictes with an existing relationship in the directed graph, such as assigning different parent functions to the same sub-function, a voting mechanism or a confidence mechanism can be used to resolve the conflict. The voting mechanism counts the frequency of various relationships and selects the relationship with the most occurrences as the final result. The confidence mechanism assigns a confidence score to each relationship and selects the relationship with the highest confidence score as the final result.

[0116] Finally, the directed graph structure of the initial knowledge base is updated based on the functional hierarchy to obtain the target knowledge base. The target knowledge base comprehensively represents the functional architecture and operational logic of the vehicle infotainment system in graph form. The nodes in the target knowledge base cover every major interface element of the vehicle infotainment system, and the edges cover every major navigation path and relationship between these interface elements. The target knowledge base provides high-quality training data for subsequent model training.

[0117] The technical solution of this application embodiment constructs a generalizable target knowledge base, elevating discrete operation instances into a structured hierarchical knowledge graph. This target knowledge base is an abstract representation of the functions of the vehicle system. The reasoned relation tuples are integrated to construct a unified hierarchical functional tree graph knowledge base of the vehicle system represented by a directed graph structure, enabling the vehicle system to have reasoning and planning capabilities, and laying a solid foundation for data synthesis and model generalization.

[0118] In another embodiment, generating model training data based on the target knowledge base includes: reorganizing the jump paths between nodes in the target knowledge base to generate new operation trajectories; and generating model training data based on the user operation targets associated with the new operation trajectories, the interactive operations included in the new operation trajectories, and the interface images associated with each interactive operation.

[0119] Jump path reorganization refers to connecting existing nodes and edges in the target knowledge base in a new combination to form new paths that do not exist in the original operation record data. Jump path reorganization generates diverse operation sequences based on the interface elements and jump relationships recorded in the target knowledge base through different path selections and combinations.

[0120] The new operation trajectory refers to the sequence of operations not collected in the original data, which is reconstructed through jump paths. The new operation trajectory has the same structure as the original operation trajectory, including the starting interface elements, a series of interactive operations, and the final target interface elements.

[0121] For example, a starting node and a target node can be selected in the target knowledge base. The starting node could be "home screen," and the target node could be "brightness adjustment." A path from the starting node to the target node is searched in the directed graph of the target knowledge base, using graph traversal algorithms such as depth-first search or breadth-first search, to find the node sequence connecting the starting and target nodes. For instance, if the jump path is found as: Home screen -> Settings -> Display -> Brightness adjustment, and this jump path is a path combination not found in the original operation record data, a new operation trajectory can be generated based on this jump path. Furthermore, based on the new operation trajectory, each interactive operation, associated interface images, and user operation targets can be determined. These data can then be combined to generate new model training data, thereby enriching the diversity of training data and improving the generalization ability of subsequent model training.

[0122] The technical solution of this application embodiment utilizes the rich interface elements and jump paths in the target knowledge base to automatically synthesize a large number of training samples that do not exist in the original data, which greatly expands the scale and diversity of the training data, enabling the vehicle system model to learn more comprehensive and generalized operation strategies, and improving the decision planning and instruction execution capabilities of the vehicle system model.

[0123] In another embodiment, based on the user operation target associated with the new operation trajectory, the various interactive operations included in the new operation trajectory, and the interface images associated with each interactive operation, semantic description information of each interactive operation is generated, including: generating new operation record data based on the user operation target associated with the new operation trajectory, the various interactive operations included in the new operation trajectory, and the interface images associated with each interactive operation; generating semantic description information of each interactive operation based on the new operation record data; and generating model training data based on the new operation record data and the semantic description information of each interactive operation.

[0124] The new operation record data may include a new operation trajectory corresponding to the user's operation target, as well as the interface image associated with each interactive operation in the new operation trajectory.

[0125] The method of generating semantic description information for each interactive operation based on the new operation record data can refer to the implementation method of generating semantic description information for interactive operations based on the operation record data described above, and will not be repeated here.

[0126] In practice, model training data is generated based on the new operation record data and the semantic description information of each interaction operation. This can be done by creating a training data record, which includes an instruction field, an image field, an action field, and a thought process field. The instruction field can store the user's operation target; the image field can store the interface image associated with the interaction operation; the action field can store the interaction operation; and the thought process field can store semantic description information.

[0127] Optionally, data can be merged according to a specific data format, such as JSON. For example, fields can be organized as properties of a JSON object, and the JSON representation of training data can be as follows:

[0128] {

[0129] "instruction": "Adjust screen brightness"

[0130] "images": ["Home screen image", "Settings menu image", "Display settings image", "Brightness adjustment image"],

[0131] Actions: ["Click the settings icon", "Click to show menu item", "Click the brightness adjustment option"]

[0132] "thoughts": ["First interactive operation semantic description", "Second interactive operation semantic description", "Third interactive operation semantic description"]

[0133] }

[0134] The technical solution implemented in this application can ensure that the generated training data contains complete task information, operation sequences, interface images, and high-level semantic descriptions, providing rich and comprehensive training data for training high-quality vehicle system models. By incorporating semantic description information into the training data, the annotation enhancement of operation record data is achieved.

[0135] For the convenience of those skilled in the art, Figure 3 A logic diagram of a method for generating model training data for a vehicle infotainment system is provided as an example.

[0136] Step S301: Structured operation trajectory data acquisition. Specifically, this can collect user operation targets to reflect user intent, operation action sequences to record specific interaction details, screen image sequences to display visual feedback before and after the operation, and completion status markers to reflect task progress.

[0137] Step S302 involves single-step semantic parsing and relational reasoning. Specifically, the data collected in step S301 can be intelligently processed, including: multimodal difference perception and action interpretation, using a large model to analyze image differences and coordinate actions to generate structured semantic descriptions; and extracting hierarchical relationships and further reasoning out functional subordination relationships.

[0138] Step S303: Construction and iterative update of the hierarchical knowledge base. Specifically, based on the functional book relationships deduced in step S302, a directed graph structure is defined through knowledge representation, and the scattered relation tuples are integrated through knowledge fusion to construct a unified hierarchical knowledge base covering the functions of the vehicle system.

[0139] Step S304: Data synthesis and enhancement based on the knowledge base. Specifically, the knowledge base constructed in step S303 can be used for data generation. On the one hand, trajectory data can be enhanced by annotation, supplementing the original operation trajectory data with high-quality semantic descriptions; on the other hand, new high-level instructions can be synthesized through reverse reasoning, using path combinations in the knowledge base to generate entirely new and unseen instruction and operation sequences, thereby achieving large-scale synthesis and diversity enhancement of training data.

[0140] In another embodiment, such as Figure 4 As shown, a method for generating model training data for a vehicle infotainment system is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0141] Step S402: Obtain operation record data of the human-machine interface of the vehicle system.

[0142] The operation record data includes the operation trajectory corresponding to the user's operation target, as well as the interface image associated with each interactive operation in the operation trajectory.

[0143] Step S404: Input the operation record data into the multimodal large model, and integrate the operation description information, interface change information and progress description information generated by the multimodal large model to generate semantic description information.

[0144] Among them, the multimodal large model is used to determine the corresponding operation description information based on the interaction operation, and to determine the interface change information based on the differences between the interface images associated with the interaction operation, and to determine the progress description information of the interaction operation relative to the user operation goal based on the interaction operation and the user operation goal.

[0145] Step S406: Extract the relationship tuples between interface elements from the semantic description information using the multimodal large model, and generate the functional hierarchy relationship between interface elements based on the relationship tuples between interface elements.

[0146] Among them, the relation tuple includes the parent-child function affiliation between various interface elements.

[0147] Interface elements include elements that are associated with interface changes caused by interactive operations.

[0148] Step S408: Construct an initial knowledge base using a directed graph structure. Update the directed graph structure of the initial knowledge base according to the functional hierarchy to obtain the target knowledge base.

[0149] In the initial knowledge base, the nodes in the directed graph structure are used to represent the interface elements of the vehicle system, and the edges in the directed graph structure are used to represent the jump paths between nodes.

[0150] Step S410: Reorganize the jump paths between nodes in the target knowledge base to generate new operation trajectories.

[0151] Step S412: Generate model training data based on the user operation target associated with the new operation trajectory, the interactive operations contained in the new operation trajectory, and the interface images associated with each interactive operation.

[0152] In one embodiment, generating the model training data based on the user operation target associated with the new operation trajectory, each interactive operation included in the new operation trajectory, and the interface image associated with each interactive operation includes: generating new operation record data based on the user operation target associated with the new operation trajectory, each interactive operation included in the new operation trajectory, and the interface image associated with each interactive operation; generating semantic description information for each interactive operation based on the new operation record data; and generating the model training data based on the new operation record data and the semantic description information for each interactive operation.

[0153] It should be noted that the specific limitations of the above steps can be found in the specific limitations of the model training data generation method for a vehicle infotainment system described above.

[0154] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0155] Based on the same inventive concept, this application also provides a vehicle system model training data generation device for implementing the above-mentioned vehicle system model training data generation method. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more vehicle system model training data generation device embodiments provided below can be found in the limitations of the vehicle system model training data generation method described above, and will not be repeated here.

[0156] In one exemplary embodiment, such as Figure 5 As shown, a model training data generation device for a vehicle infotainment system is provided, comprising:

[0157] The data acquisition module 510 is used to acquire operation record data of the human-machine interaction interface of the vehicle system; the operation record data includes the operation trajectory corresponding to the user's operation target, and the interface image associated with each interactive operation in the operation trajectory.

[0158] The relationship extraction module 520 is used to generate semantic description information of the interactive operation based on the operation record data, and to extract the functional hierarchy relationship between each interface element from the semantic description information; the interface element includes elements associated with the interface changes caused by the interactive operation.

[0159] The knowledge construction module 530 generates a target knowledge base for the vehicle system based on the functional hierarchy relationship; the target knowledge base is used to record each functional module in the vehicle system and the interrelationships between the functional modules.

[0160] The training data generation module 540 is used to generate model training data based on the target knowledge base; the model training data is used to train the artificial intelligence model mounted on the vehicle system.

[0161] In one embodiment, the relationship extraction module 520 is specifically used to input the operation record data into a multimodal large model; wherein, the multimodal large model is used to determine corresponding operation description information based on the interaction operation; the multimodal large model is also used to determine interface change information based on the differences between the interface images associated with the interaction operation; the multimodal large model is also used to determine the progress description information of the interaction operation relative to the user operation target based on the interaction operation and the user operation target; and to generate the semantic description information by fusing the operation description information, the interface change information, and the progress description information.

[0162] In one embodiment, the relationship extraction module 520 is specifically used to extract relationship tuples between the interface elements from the semantic description information through the multimodal large model; the relationship tuples include the parent-child functional attribution relationships between the interface elements; and generate functional hierarchical relationships between the interface elements based on the relationship tuples between the interface elements.

[0163] In one embodiment, the knowledge construction module 530 is specifically used to construct an initial knowledge base using a directed graph structure; the nodes in the directed graph structure of the initial knowledge base are used to represent the interface elements of the vehicle system, and the edges in the directed graph structure are used to represent the jump paths between the nodes; the directed graph structure of the initial knowledge base is updated according to the functional hierarchy relationship to obtain the target knowledge base.

[0164] In one embodiment, the training data generation module 540 is specifically used to reorganize the jump paths between the nodes in the target knowledge base to generate new operation trajectories; and to generate the model training data based on the user operation target associated with the new operation trajectory, the interactive operations included in the new operation trajectory, and the interface images associated with the interactive operations.

[0165] In one embodiment, the training data generation module 540 is specifically configured to generate new operation record data based on the user operation target associated with the new operation trajectory, each interactive operation included in the new operation trajectory, and the interface image associated with each interactive operation; generate semantic description information of each interactive operation based on the new operation record data; and generate the model training data based on the new operation record data and the semantic description information of each interactive operation.

[0166] Each module in the model training data generation device of the aforementioned vehicle infotainment system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0167] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for generating model training data for a vehicle-mounted system. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0168] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0169] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0170] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0171] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0172] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0173] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0174] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0175] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating model training data for a vehicle infotainment system, characterized in that, The method includes: The system acquires operation record data of the human-machine interface of the vehicle system; the operation record data includes the operation trajectory corresponding to the user's operation target, and the interface image associated with each interaction operation in the operation trajectory; Based on the operation record data, semantic description information of the interaction operation is generated, and the functional hierarchy relationship between each interface element is extracted from the semantic description information; the interface element includes elements associated with the interface changes caused by the interaction operation. Based on the aforementioned functional hierarchy, a target knowledge base for the vehicle infotainment system is generated; the target knowledge base is used to record each functional module in the vehicle infotainment system, as well as the relationships between the functional modules. Based on the target knowledge base, model training data is generated; the model training data is used to train the artificial intelligence model mounted on the vehicle system.

2. The method according to claim 1, characterized in that, The step of generating semantic description information of the interactive operation based on the operation record data includes: The operation record data is input into a multimodal large model; wherein, the multimodal large model is used to determine the corresponding operation description information based on the interaction operation; the multimodal large model is also used to determine interface change information based on the differences between the interface images associated with the interaction operation; the multimodal large model is also used to determine the progress description information of the interaction operation relative to the user operation target based on the interaction operation and the user operation target; The semantic description information is generated by integrating the operation description information, the interface change information, and the progress description information.

3. The method according to claim 2, characterized in that, The step of extracting the functional hierarchy relationship between interface elements from the semantic description information includes: The multimodal large model extracts relationship tuples between the interface elements from the semantic description information; the relationship tuples include the parent-child functional affiliation relationships between the interface elements. Based on the relationship tuples between the interface elements, a functional hierarchy relationship between the interface elements is generated.

4. The method according to claim 1, characterized in that, Based on the aforementioned functional hierarchy, the target knowledge base of the vehicle system is generated: An initial knowledge base with a directed graph structure is constructed; the nodes in the directed graph structure of the initial knowledge base are used to represent the interface elements of the vehicle system, and the edges in the directed graph structure are used to represent the jump paths between the nodes. Based on the functional hierarchy, the directed graph structure of the initial knowledge base is updated to obtain the target knowledge base.

5. The method according to claim 4, characterized in that, The step of generating model training data based on the target knowledge base includes: The jump paths between the nodes in the target knowledge base are reorganized to generate new operation trajectories; The model training data is generated based on the user operation target associated with the new operation trajectory, the interactive operations included in the new operation trajectory, and the interface images associated with each interactive operation.

6. The method according to claim 5, characterized in that, The step of generating the model training data based on the user operation target associated with the new operation trajectory, each interactive operation included in the new operation trajectory, and the interface image associated with each interactive operation includes: Based on the user operation target associated with the new operation trajectory, the interactive operations included in the new operation trajectory, and the interface images associated with each interactive operation, new operation record data is generated. Based on the new operation record data, semantic description information for each interactive operation is generated; The model training data is generated based on the new operation record data and the semantic description information of each interaction operation.

7. A model training data generation device for a vehicle infotainment system, characterized in that, The device includes: The data acquisition module is used to acquire operation record data of the human-machine interaction interface of the vehicle system; the operation record data includes the operation trajectory corresponding to the user's operation target, and the interface image associated with each interaction operation in the operation trajectory; The relationship extraction module is used to generate semantic description information of the interactive operation based on the operation record data, and to extract the functional hierarchy relationship between each interface element from the semantic description information; the interface element includes elements associated with the interface changes caused by the interactive operation. A knowledge construction module is used to generate a target knowledge base for the vehicle system based on the functional hierarchy; the target knowledge base is used to record each functional module in the vehicle system and the relationships between the functional modules. The training data generation module is used to generate model training data based on the target knowledge base; the model training data is used to train the artificial intelligence model mounted on the vehicle system.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images