Model training and intelligent cockpit visible method

Abstract

The application discloses a kind of multi-modal wisdom cockpit interactive large model training and wisdom cockpit visible method can say. Utilize first sample set to the multi-modal basic large model that has multi-modal data processing ability and natural language understanding ability is trained, make it transform into multi-modal wisdom cockpit interactive large model. This process makes model accurately identify the function meaning of interface element contained in each interface in target wisdom cockpit system, lay a solid foundation for subsequent interaction with user. Based on second sample set, the multi-modal wisdom cockpit interactive large model is trained, and the multi-modal wisdom cockpit interactive large model can learn the mapping relationship between voice intent and interface element in each interface image in target wisdom cockpit system. Subsequently, when user issues voice command, the multi-modal wisdom cockpit interactive large model can accurately understand user intent, and find corresponding interface element to execute operation, greatly enhance the accuracy and effectiveness of voice interaction, improve user experience.

Description

Technical Field

[0001] This application relates to the fields of intelligent driving and deep learning technology, and in particular to a training method for a large-scale interactive model of a multimodal smart cockpit and a method for making the smart cockpit visible and talkable. Background Technology

[0002] With the rapid development of smart cockpit technology, providing drivers with a safer, more efficient, and natural interactive experience has become a core issue that the industry urgently needs to address. Traditional interaction methods, such as manually touching screens or operating physical buttons, while meeting basic operational needs to some extent, inevitably distract drivers, thereby increasing potential risks during driving. Therefore, developing a new interaction method that can reduce driver distraction and improve driving safety is particularly important.

[0003] The "See-and-Speak" functionality is a promising interactive solution that emerged in this context. This feature allows drivers to directly manipulate elements on the screen using voice commands, resulting in a more intuitive and natural interactive experience. However, existing methods for implementing "See-and-Speak" functionality face numerous challenges.

[0004] On the one hand, while the Android Assistive Services-based approach can leverage Android's built-in mechanisms to monitor screen rendering changes and register all text on the current screen, this method suffers from issues such as passive information acquisition, high resource consumption, and limited interface compatibility. Because it relies on system-level Assistive Services, it cannot actively acquire or control interface element information, potentially leading to compatibility problems in certain interfaces or applications. Furthermore, this approach consumes significant system resources when processing a large number of interface elements, impacting overall performance.

[0005] On the other hand, while third-party application registration solutions allow developers to manually enter text for registration, enabling more flexible interactive control, this approach is relatively costly and complex. Developers need to deeply customize each application and manually register the corresponding text on different interfaces, which not only increases the workload but may also lead to poor application compatibility. Since different applications may have different interface layouts and elements, developers need to perform individual adaptation for each application, which undoubtedly increases development costs and time. Summary of the Invention

[0006] This application provides a training method for a large-scale multimodal smart cockpit interaction model and a smart cockpit "see-and-talk" method, which solves the problems of passive and uncontrollable information acquisition, poor interface compatibility, high cost of third-party application development, and high resource consumption in existing smart cockpit "see-and-talk" function implementation methods.

[0007] Firstly, this application provides a method for training a large-scale multimodal intelligent cockpit interaction model, the method comprising:

[0008] Obtain a first sample set containing attribute information samples corresponding to multiple interface elements; wherein, the attribute information sample of any interface element includes one or more of the following: operation-related information of the interface element, the self-features of the interface element, text descriptions related to the interface element, and context information of the interface element, and each attribute information sample corresponds to a label with a real functional meaning.

[0009] Based on the attribute information samples in the first sample set and the corresponding real functional meaning labels, the pre-trained multimodal basic model is trained to obtain the trained multimodal smart cockpit interaction model. The multimodal basic model is a large model that already has multimodal data processing and natural language understanding capabilities, and the multimodal smart cockpit interaction model has the ability to identify the functional meaning of the interface elements contained in each interface of the target smart cockpit system.

[0010] Obtain a second sample set containing multiple voice command samples related to interaction with the target smart cockpit system; wherein each voice command sample corresponds to a voice intent tag and an interface element tag;

[0011] Based on the voice command samples in the second sample set, and the voice intent labels and interface element labels of each voice command sample, the multimodal smart cockpit interaction model is trained so that the multimodal smart cockpit interaction model learns the mapping relationship between voice intent and interface elements of each interface image in the target smart cockpit system.

[0012] Secondly, this application also provides a smart cockpit "see-and-say" method based on a model trained using the above method, the method comprising:

[0013] Obtain the voice commands to be processed;

[0014] By using a pre-trained multimodal smart cockpit interaction model, the target interface elements corresponding to the voice intent of the voice command to be processed are determined based on the voice command to be processed.

[0015] Based on the pre-saved coordinate information corresponding to each interface element, determine the target coordinate information corresponding to the target interface element;

[0016] Based on the target coordinate information, operate the target interface elements in the current interface of the target smart cockpit system.

[0017] Thirdly, this application provides a training device for a large-scale multimodal intelligent cockpit interaction model, the device comprising:

[0018] The first acquisition unit is used to acquire a first sample set containing attribute information samples corresponding to multiple interface elements; wherein, the attribute information sample of any interface element includes one or more of the following: operation-related information of the interface element, the self-features of the interface element, text descriptions related to the interface element, and context information of the interface element, and each attribute information sample corresponds to a label with a real functional meaning.

[0019] The first training unit is used to train a pre-trained multimodal basic model based on the attribute information samples in the first sample set and the real functional meaning labels corresponding to each attribute information sample, so as to obtain a trained multimodal smart cockpit interaction model; wherein, the multimodal basic model is a large model that already has multimodal data processing capabilities and natural language understanding capabilities, and the multimodal smart cockpit interaction model has the ability to identify the functional meaning of the interface elements contained in each interface of the target smart cockpit system.

[0020] The second acquisition unit is used to acquire a second sample set containing multiple voice command samples related to interaction with the target smart cockpit system; wherein each voice command sample corresponds to a voice intent tag and an interface element tag;

[0021] The second training unit is used to train the multimodal smart cockpit interaction model based on each of the voice command samples in the second sample set, and the voice intent labels and interface element labels of each of the voice command samples, so that the multimodal smart cockpit interaction model learns the mapping relationship between voice intent and interface elements of each interface image in the target smart cockpit system.

[0022] Fourthly, this application provides a smart cockpit device that allows for both visibility and speech, the device comprising:

[0023] The acquisition module is used to acquire the voice commands to be processed.

[0024] The training module is used to determine the target interface elements corresponding to the voice intent of the voice command to be processed based on the pre-trained multimodal smart cockpit interaction model.

[0025] The determination module is used to determine the target coordinate information corresponding to the target interface element based on the pre-saved coordinate information corresponding to each interface element.

[0026] The operation module is used to operate the target interface elements in the current interface of the target smart cockpit system according to the target coordinate information.

[0027] Fifthly, this application provides a computer device including a processor, which executes a computer program stored in a memory to implement the steps of the training method for the multimodal smart cockpit interaction model described above, or to implement the steps of the smart cockpit visible-as-speakable method described above.

[0028] Sixthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the training method for the multimodal smart cockpit interaction model described above, or the steps of the smart cockpit visible-as-speakable method described above.

[0029] The beneficial effects of this application are as follows:

[0030] 1. By acquiring a first sample set of interface images based on the target smart cockpit system, where each attribute information sample corresponds to a label with the actual functional meaning, rich and accurate raw materials are provided for the multimodal smart cockpit interaction model. The multimodal smart cockpit interaction model can learn the functional meaning of various interface elements in different scenarios, constructing a comprehensive smart cockpit interface knowledge system. This allows for accurate identification of the functional meaning of interface elements contained in each interface of the target smart cockpit system, laying a solid foundation for subsequent user interaction.

[0031] 2. Using the first sample set, the existing multimodal basic model with multimodal data processing and natural language understanding capabilities is trained, transforming it into a multimodal smart cockpit interaction model. This process allows the model to deeply understand the characteristics of the smart cockpit system and user interaction needs, significantly improving its intelligent interaction level. It can respond quickly and accurately based on user operations or commands, achieving more efficient and intelligent human-computer interaction.

[0032] 3. A second sample set containing voice command samples and their corresponding voice intent labels and interface element labels was obtained, further enriching the learning data of the multimodal smart cockpit interaction model. Based on this second sample set, the multimodal smart cockpit interaction model was trained, enabling it to learn the mapping relationship between voice intents and interface elements in the target smart cockpit system. Subsequently, when a user issues a voice command, the multimodal smart cockpit interaction model can accurately understand the user's intent and find the corresponding interface element to execute the operation, greatly enhancing the accuracy and effectiveness of voice interaction and improving the user experience. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A schematic diagram illustrating the training process of a large-scale multimodal smart cockpit interaction model provided in this application embodiment;

[0035] Figure 2 A schematic diagram illustrating the training process of a large multimodal smart cockpit interaction model provided for embodiments of this application;

[0036] Figure 3 This application provides a schematic diagram illustrating the visible and speakable process of a smart cockpit.

[0037] Figure 4 A schematic diagram illustrating the process of a smart cockpit that is visible and speakable, provided as an embodiment of this application;

[0038] Figure 5 A schematic diagram of the structure of a training device for a large-scale multimodal smart cockpit interaction model provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of the structure of a smart cockpit device that allows users to speak as soon as they see it, provided in an embodiment of this application.

[0040] Figure 7 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] To enhance the intelligence level of smart cockpit system interaction with users and achieve a more accurate and efficient human-computer interaction experience, this application provides a training method for a multimodal smart cockpit interaction model and a smart cockpit "see-and-talk" approach.

[0043] Example 1:

[0044] This application provides a method for training a large-scale multimodal intelligent cockpit interaction model. Figure 1This application provides a schematic diagram of the training process for a large-scale multimodal smart cockpit interaction model, which includes:

[0045] S101: Obtain a first sample set containing attribute information samples corresponding to multiple interface elements; wherein, the attribute information sample of any interface element includes one or more of the following: operation-related information of the interface element, the self-features of the interface element, text descriptions related to the interface element, and contextual information of the interface element, and each attribute information sample corresponds to a label with a real functional meaning.

[0046] The training method for the multimodal smart cockpit interaction model provided in this application is applied to computer equipment, which can be a smart terminal, such as an in-vehicle central control unit or an in-vehicle processor, or a server, such as an in-vehicle business server.

[0047] To enhance the intelligence level of the smart cockpit system's interaction with users and achieve a more precise and efficient human-computer interaction experience, this application requires pre-training a multimodal smart cockpit interaction model for the target smart cockpit system. To train this multimodal smart cockpit interaction model, a sample set (denoted as the first sample set) of relevant information for each interface element in the target smart cockpit system needs to be pre-acquired for the model to learn. The first sample set contains attribute information samples corresponding to multiple interface elements. Interface elements are visual components with independent functions or interactive roles on the target smart cockpit system interface. The types of interface elements include one or more of the following: buttons, text boxes (such as text boxes in a car navigation interface, play / pause buttons in a music playback interface, etc.). Attribute information samples are a series of feature information extracted for each interface element. For example, the attribute information sample of a volume control slider includes the slider's coordinates, size, and the current volume value. This information accurately describes the state and characteristics of the interface element within the interface. At the same time, the attribute information samples also have corresponding labels with the true functional meaning, which are used to accurately describe the functions of interface elements. For example, the true functional meaning label of the route planning button in the navigation interface is "plan navigation route".

[0048] In one possible implementation, the attribute information sample of any interface element includes one or more of the following: operation-related information of the interface element, the interface element's own characteristics, the environment information of the interface element, and the context information of the interface element. The operation-related information guides how to operate the interface element. For example, button-type interface elements typically correspond to click operations; clicking the play button plays music. Slider-type interface elements correspond to swiping operations; sliding the volume slider up and down changes the volume. Long-pressing a point on a map interface brings up a special function to set the destination. Text box-type interface elements typically correspond to click and text input operations; clicking the text box allows you to enter text content. The interface element's own characteristics encompass features such as its shape, color, texture, text content, and coordinate information. The context information of the interface element involves rich information such as the preceding and following operation flows associated with the interface element, closely related functions, and the specific page scene in which it is located.

[0049] These attribute information samples were all obtained from interface images in the target smart cockpit system.

[0050] There are several ways to obtain interface images:

[0051] 1. Take a screenshot

[0052] This can be achieved by taking screenshots of the target smart cockpit system's interface. For example, screenshots can be taken at preset intervals or time points. For instance, by periodically taking screenshots, various images of the target smart cockpit system's interface can be obtained. For example, a suitable time period can be set, such as every 100 milliseconds, to automatically trigger a screenshot command and capture a screen image. This allows capturing the interface state of the smart cockpit system at different stages of use, including system startup, user operation, and stable operation.

[0053] It's important to note that the screenshot interval can be set differently depending on the scenario. If you want to comprehensively capture the interface state of the smart cockpit system at different stages of use, including system startup, user operation, and stable operation, the screenshot interval can be set shorter. However, screenshotting consumes system resources, including CPU, memory, and storage read / write. If the screenshot interval is too short, such as every second or even less, it may slow down the system and affect the normal functions of the smart cockpit system, such as real-time navigation route calculation and smooth multimedia playback. Therefore, if you want screenshotting to consume as few system resources as possible and not affect system operation, the screenshot interval can be set longer.

[0054] 2. Screen recording

[0055] Screen recording software can be used to continuously record the interface display of the target intelligent cockpit system under different operating scenarios, and key frames can be extracted from the recorded video according to time nodes as interface images.

[0056] As a possible implementation, to reduce the resources consumed in subsequent processing of interface images, in this application, after obtaining the interface image of the target smart cockpit system based on the above embodiments, the interface image can be preprocessed. This image preprocessing includes one or more of the following: grayscale processing and image denoising. Grayscale processing converts a color image to a grayscale image, retaining only the brightness information and removing color information. After grayscale processing, the data volume is reduced, simplifying the calculation process of the subsequent element recognition model, improving processing efficiency, and highlighting the structure and contours of the image, which is beneficial for element recognition. After denoising, the image provides clearer and more accurate image data for subsequent element recognition, reducing the interference of noise on the recognition results. Using data augmentation techniques (such as random cropping, rotation, and flipping of button images) can increase the diversity of attribute data samples, improving the ability of the multimodal smart cockpit interaction model to recognize and understand interface elements in different scenarios.

[0057] In one example, attribute information samples can be obtained from interface images of the target smart cockpit system through manual collection.

[0058] In another example, attribute information samples can also be obtained automatically based on interface images from the target smart cockpit system in the following manner:

[0059] For information related to user interface element interactions, an interaction log module can be embedded in the target smart cockpit system. When a user interacts with an interface element, the system records the interaction in real time, including the timestamp of clicks and the start and end positions of swipes. For example, for a volume control slider, the log module records the time of each swipe and the range of changes in the slider's position. By analyzing this log data and combining it with interface images, relevant information about the volume control slider's operation (such as the swipe operation, its direction, and amplitude) can be obtained.

[0060] For the characteristics of interface elements themselves, different methods can be used to obtain them depending on the type of interface element.

[0061] For example, for any of the interface images, various types of interface elements contained in the interface image are determined by an image recognition algorithm (such as the YOLO algorithm or the Faster R-CNN algorithm).

[0062] For button-type interface elements in the interface image, a pre-trained button recognition algorithm is used to determine the visual features (e.g., button shape, color, texture, etc.) and bounding box coordinates of each button in the interface image. This pre-trained button recognition algorithm is trained on a dataset constructed from a large number of image samples containing various types of buttons. This dataset includes interface images from different brands and models of smart cockpit systems, ensuring button diversity and covering various shapes (circles, squares, polygons, etc.), colors (solid colors, gradient colors, etc.), and textures (smooth, matte, patterned, etc.). These image samples are labeled to accurately mark the category, bounding box coordinates, and corresponding visual feature attributes of each button.

[0063] For the text box type interface element in the interface image, optical character recognition technology is used to convert the text box in the interface image into an editable text string, and the bounding box coordinates of the area where the text box is located are recorded.

[0064] For text descriptions related to interface elements, the bounding box coordinates of any given element can be determined from the interface image. Then, the image area containing these bounding box coordinates is identified using optical character recognition (OCR) technology to determine the text contained within the image, which is then identified as the text description related to that interface element.

[0065] For the contextual information of interface elements, the operation logs and user behavior data of the target intelligent cockpit system can be combined. The operation logs record the order in which users operate on various interface elements at different times. By analyzing this data, the preceding and following operation processes associated with that interface element can be reconstructed. Simultaneously, based on the system's functional architecture and page navigation logic, the specific page scenario in which the interface element exists and its closely related functions can be determined. For example, in a navigation page, the contextual information of a map interface element includes its association with elements such as the destination input box and route planning buttons, as well as information on which stage of the navigation function is currently in.

[0066] To improve the model's generalization ability, the first sample set should cover as many different types of interfaces and interface elements as possible in the target smart cockpit system. This includes not only common navigation interfaces and multimedia playback interfaces, but also vehicle settings interfaces and driving information display interfaces. Furthermore, for each interface type, multiple interface images under different states should be collected, such as images of the navigation interface under different zoom levels and different route planning conditions, to ensure that the model can learn comprehensive and accurate information related to interface elements.

[0067] S102: Based on the attribute information samples in the first sample set and the real functional meaning labels corresponding to each attribute information sample, the pre-trained multimodal basic large model is trained to obtain the trained multimodal smart cockpit interaction large model; wherein, the multimodal basic large model is a large model that already has multimodal data processing capabilities and natural language understanding capabilities, and the multimodal smart cockpit interaction large model has the ability to identify the functional meaning of the interface elements contained in each interface of the target smart cockpit system.

[0068] After obtaining the first sample set, it can be used to train the pre-trained multimodal basic model to obtain the trained multimodal smart cockpit interaction model. This multimodal basic model is built on a deep learning framework and already possesses multimodal data processing and natural language understanding capabilities. It can process various types of data, including images and text, and understand the semantics of natural language. For example, some neural network models based on the Transformer architecture, trained on large amounts of image and text data, already possess image feature extraction and text semantic understanding capabilities.

[0069] For example, attribute information samples and their corresponding true functional meaning labels from the first sample set are input into a multimodal basic large model. Using this multimodal basic large model, the functional meaning of the interface elements corresponding to the input attribute information samples can be predicted. The stochastic gradient descent algorithm and its variants are employed to minimize the error between the model's prediction and the true functional meaning labels. Regularization techniques (such as L1 and L2 regularization) are used to prevent overfitting, and hyperparameters such as the number of training epochs and batch size are appropriately set.

[0070] After multiple rounds of training, the multimodal basic model gradually learns the functional meaning of each interface element in the target smart cockpit system, thus transforming into a trained multimodal smart cockpit interaction model, which has the ability to identify the functional meaning of the interface elements contained in each interface of the target smart cockpit system.

[0071] The resulting multimodal smart cockpit interaction model, trained on the first sample set, is specifically designed for interaction within smart cockpit systems. It not only possesses the natural language understanding and multimodal data processing capabilities of the multimodal foundation model, but also accurately identifies various interface elements within the target smart cockpit system interface and understands their functional meaning. For example, when this multimodal smart cockpit interaction model receives a navigation interface image, it can accurately determine the function of each button and area, recognizing that the text box is for inputting the destination and the route planning button is for planning the trip route.

[0072] S103: Obtain a second sample set containing multiple voice command samples related to interaction with the target smart cockpit system; wherein each voice command sample corresponds to a voice intent tag and an interface element tag.

[0073] To further enhance the interaction capabilities of the multimodal smart cockpit interaction model and enable it to better understand the voice intent of users interacting with the target smart cockpit system, a second sample set is needed. Based on this second sample set, the multimodal smart cockpit interaction model can learn the mapping relationship between voice intent and interface elements of various interface images in the target smart cockpit system. The second sample set contains multiple voice command samples related to interaction with the target smart cockpit system. These voice command samples represent the voice content uttered by the user when interacting with the target smart cockpit system, such as "Find me a nearby hot pot restaurant," "Play the song by xxx," and "Open the car window." Each voice command sample corresponds to a voice intent label and an interface element label. The voice intent label clarifies the purpose of the user's voice command; for example, the voice intent label for "Find me a nearby hot pot restaurant" is "Search for nearby hot pot restaurants"; the voice intent label for "Play the song by xxx" is "Play the song by the specified artist." The interface element label identifies the specific interface element in the target smart cockpit system interface involved in the voice command. For example, in the voice command "Find me a nearby hot pot restaurant", the interface element label might be a text box in the navigation interface, because you need to enter the relevant information of the hot pot restaurant in this interface element to perform the search operation; for "Play the song of xxx", the interface element label might be the song search input box or the song list area in the music playback interface.

[0074] In this application, voice command samples can be obtained by collecting voice commands in actual vehicle usage scenarios, inviting users to issue voice commands through simulation experiments and recording relevant information simultaneously, or uploaded by users during the use of the target smart cockpit.

[0075] In one example, considering that the acquired speech signal is often affected by the complex environment inside the vehicle, with significant environmental noise and echo interference, which severely impacts the quality and accuracy of voice commands, speech preprocessing is necessary. This preprocessing includes, but is not limited to, one or more of the following: speech enhancement, speech signal framing, and windowing. Speech enhancement can remove environmental noise and echo interference. Speech signal framing and windowing can convert continuous speech signals into a series of short, high-quality speech frames.

[0076] In one possible implementation, the voice command sample can be either a collected voice signal or a preprocessed voice signal. Alternatively, it can be a voice feature vector, such as Mel-frequency cepstral coefficients (MFCC) or linear predictive cepstral coefficients (LPCC). By converting the voice signal into a set of feature vectors that characterize the voice content, these feature vectors contain information such as the frequency and amplitude of the voice, reflecting its acoustic characteristics and providing foundational data for subsequent voice intent understanding, thus reducing the computational burden of subsequent model training. In specific implementations, the settings can be flexibly configured according to requirements, and no specific limitations are imposed here.

[0077] As one possible implementation, when acquiring voice command samples, the diversity and comprehensiveness of the voice command samples can be ensured as much as possible. This includes not only common voice interaction scenarios such as navigation, music playback, and vehicle control, but also some more special or personalized voice command scenarios, such as users interacting with the smart cockpit system using dialects or custom voice commands. Representing these situations in the sample set as much as possible can improve the adaptability of the multimodal smart cockpit interaction model to various voice interaction situations.

[0078] S104: Based on each of the voice command samples in the second sample set, and the voice intent labels and interface element labels of each of the voice command samples, the multimodal smart cockpit interaction model is trained so that the multimodal smart cockpit interaction model learns the mapping relationship between voice intent and interface elements of each interface image in the target smart cockpit system.

[0079] After obtaining the second sample set, it is used to further train the multimodal smart cockpit interaction model already trained with the first sample set. For example, during training, voice command samples from the second sample set are input into the multimodal smart cockpit interaction model in an orderly manner. The multimodal smart cockpit interaction model calculates the prediction results through forward propagation based on the input data, then uses the cross-entropy loss function to calculate the differences between the prediction results and the actual voice intent labels and interface element labels. Subsequently, it updates the parameters of the multimodal smart cockpit interaction model through the backpropagation algorithm, enabling the model to continuously learn the mapping relationship between voice intents and interface elements in the interface image. For example, for the voice intent of "zoom in on the map," the multimodal smart cockpit interaction model will learn the corresponding interface element (such as a zoom button) in the navigation interface to perform the map zoom-in operation.

[0080] In one possible implementation, the step of training the multimodal smart cockpit interaction model based on each of the voice command samples in the second sample set, and the voice intent labels and interface element labels of each of the voice command samples, so that the multimodal smart cockpit interaction model learns the mapping relationship between voice intent and interface elements of each interface image in the target smart cockpit system, includes:

[0081] Obtain any of the aforementioned voice command samples and their corresponding voice intent tags and interface element tags;

[0082] Based on the multimodal smart cockpit interaction model, and the learned functional meanings of the interface elements contained in each interface of the target smart cockpit system, the model outputs the first probability distribution of the predicted voice intent of the voice command sample and the second probability distribution of the interface elements corresponding to the predicted voice intent.

[0083] Based on the first probability distribution and the voice intent label, and the second probability distribution and the interface element label, the multimodal smart cockpit interaction model is trained so that the multimodal smart cockpit interaction model learns the mapping relationship between voice intent and interface elements.

[0084] During any round of training the multimodal smart cockpit interaction model, firstly, any voice command sample and its corresponding voice intent label and interface element label are obtained from the second sample set. For example, the voice command sample "Open music, play my favorite playlist" has the corresponding voice intent label "Play music from the specified playlist," and the interface element label might be the playlist selection button and play button on the music playback interface. Next, the obtained voice command sample is input into the multimodal smart cockpit interaction model. Based on the voice command sample and the functional meanings of the interface elements in the target smart cockpit system learned from the first sample set, the multimodal smart cockpit interaction model performs in-depth analysis and reasoning. The multimodal smart cockpit interaction model outputs the first probability distribution of the predicted voice intent for the voice command sample. For example, for "Open music, play my favorite playlist," the multimodal smart cockpit interaction model might output a probability of 0.85 for the predicted voice intent "Play music from the specified playlist," a probability of 0.1 for "Open music," and a probability of 0.05 for other irrelevant intents, etc. Simultaneously, the multimodal intelligent cockpit interaction model also outputs a second probability distribution of the interface elements corresponding to the predicted voice intent. For example, for the predicted voice intent "play music from the specified playlist," the probability of the playlist selection button is output as 0.9, the play button as 0.08, and other interface elements as 0.02, etc. Then, by calculating the difference between the first probability distribution and the voice intent label, and the difference between the second probability distribution and the interface element label, the loss value is determined. Based on this loss value, the parameters of the multimodal intelligent cockpit interaction model are adjusted so that in subsequent predictions, the model can more accurately output probability distributions that match the true labels, thus gradually learning the mapping relationship between voice intents and interface elements.

[0085] Since the second sample set contains several voice command samples, the above operations are performed on each voice command sample to train the multimodal smart cockpit interaction model. When the preset convergence condition is met, the training of the multimodal smart cockpit interaction model is complete.

[0086] The preset convergence conditions can be: the sum of the loss values ​​determined based on each voice command sample is less than a pre-configured loss threshold; the sum of the determined loss values ​​is consistently decreasing and tending to level off; or the number of iterations for training the multimodal smart cockpit interaction model reaches the set maximum number of iterations. These conditions can be flexibly set in practice and are not specifically limited here.

[0087] As one possible implementation, when training the multimodal smart cockpit interaction model, the voice command samples can be divided into training samples and test samples. First, the multimodal smart cockpit interaction model is trained based on the training samples, and then the reliability of the trained multimodal smart cockpit interaction model is verified based on the test samples.

[0088] After multiple rounds of training and optimization, the multimodal smart cockpit interaction model can more accurately understand and respond to voice commands. When a user issues a relatively complex voice command such as "Find me a Chinese restaurant," the multimodal smart cockpit interaction model can quickly locate the text box element in the navigation interface or the life service recommendation interface.

[0089] The beneficial effects of this application are as follows:

[0090] 1. By acquiring a first sample set of interface images based on the target smart cockpit system, where each attribute information sample corresponds to a label with the actual functional meaning, rich and accurate raw materials are provided for the multimodal smart cockpit interaction model. The multimodal smart cockpit interaction model can learn the functional meaning of various interface elements in different scenarios, constructing a comprehensive smart cockpit interface knowledge system. This allows for accurate identification of the functional meaning of interface elements contained in each interface of the target smart cockpit system, laying a solid foundation for subsequent user interaction.

[0091] 2. Using the first sample set, the existing multimodal basic model with multimodal data processing and natural language understanding capabilities is trained, transforming it into a multimodal smart cockpit interaction model. This process allows the model to deeply understand the characteristics of the smart cockpit system and user interaction needs, significantly improving its intelligent interaction level. It can respond quickly and accurately based on user operations or commands, achieving more efficient and intelligent human-computer interaction.

[0092] 3. A second sample set containing voice command samples and their corresponding voice intent labels and interface element labels was obtained, further enriching the learning data of the multimodal smart cockpit interaction model. Based on this second sample set, the multimodal smart cockpit interaction model was trained, enabling it to learn the mapping relationship between voice intents and interface elements in the target smart cockpit system. Subsequently, when a user issues a voice command, the multimodal smart cockpit interaction model can accurately understand the user's intent and find the corresponding interface element to execute the operation, greatly enhancing the accuracy and effectiveness of voice interaction and improving the user experience.

[0093] Example 2:

[0094] The training method for the multimodal intelligent cockpit interaction model provided in this application will be described below through specific embodiments. Figure 2The flowchart illustrating the training process of the multimodal smart cockpit interaction model provided in this application embodiment mainly includes a sample collection stage, a function meaning recognition stage, and a voice command and interface element debugging stage. Each stage is described in detail below:

[0095] Phase 1: Sample Collection Phase (including the first and second sample sets).

[0096] For the first sample set:

[0097] S201: Obtain images of each interface of the target smart cockpit system by periodically taking screenshots.

[0098] S202: Perform image preprocessing on each interface image; wherein, image preprocessing includes one or more of the following: grayscale processing, image noise reduction, and data enhancement.

[0099] S203: For each preprocessed interface image, obtain attribute information samples of each interface element in the preprocessed interface image.

[0100] The attribute information sample of any interface element includes one or more of the following: operation-related information of the interface element, the characteristics of the interface element itself, textual descriptions related to the interface element, and contextual information of the interface element.

[0101] S204: Determine the actual functional meaning labels corresponding to the attribute information samples of each interface element, and save the attribute information samples of each interface element and the actual functional meaning labels corresponding to each attribute information sample.

[0102] For the second sample set:

[0103] S205: Collects voice signals from the in-vehicle interaction with the target smart cockpit system.

[0104] S206: Perform speech preprocessing on the acquired speech signal.

[0105] The speech preprocessing includes speech enhancement, framing, and windowing operations.

[0106] S207: Based on the preprocessed speech signal, obtain the speech feature vector of each speech signal, and determine the speech feature vector of each speech signal as a speech command sample.

[0107] S208: Determine the voice intent label and interface element label corresponding to each voice command sample, and save each voice command sample, as well as the voice intent label and interface element label corresponding to each voice command sample.

[0108] It should be noted that the order of the steps for collecting the first and second sample sets is not fixed. S201 to S204 can be before S205 to S208, or S201 to S204 can be after S205 to S208, or S201 to S204 and S205 to S208 can be executed simultaneously.

[0109] Phase Two: Functional Meaning Identification Phase.

[0110] S209: Obtain a pre-trained multimodal base model.

[0111] Among them, the multimodal basic large model is a large model that already has multimodal data processing capabilities and natural language understanding capabilities.

[0112] S210: Obtain any attribute information sample and its corresponding true functional meaning label from the first sample set.

[0113] S211: Using a pre-trained multimodal basic large model, the meaning of the prediction function is obtained based on the attribute information sample.

[0114] S212: Based on the predicted functional meaning and the actual functional meaning labels, train the multimodal basic large model to obtain the trained multimodal smart cockpit interaction large model.

[0115] The multimodal smart cockpit interaction model possesses the ability to identify the functional meanings of interface elements within each interface of the target smart cockpit system. In Phase Three, based on the multimodal smart cockpit interaction model trained in Phase Two, the model is further optimized to learn the mapping relationship between voice commands and interface elements.

[0116] Phase 3: Voice command and interface element debugging phase.

[0117] S213: Obtain any voice command sample and its corresponding voice intent label and interface element label.

[0118] S214: Using a multimodal smart cockpit interaction model, based on voice command samples and the functional meanings of interface elements in each interface of the target smart cockpit system that have been learned, output the first probability distribution of the predicted voice intent of the voice command samples and the second probability distribution of the interface elements corresponding to the predicted voice intent.

[0119] S215: Based on the first probability distribution and voice intent labels, and the second probability distribution and interface element labels, train the multimodal smart cockpit interaction model so that the multimodal smart cockpit interaction model can learn the mapping relationship between voice intent and interface elements.

[0120] Example 3:

[0121] This application also provides a method for making a smart cockpit visible and speakable based on the model trained according to the above embodiments. Figure 3 This application provides a schematic diagram of a smart cockpit process that is both visible and speakable, which includes:

[0122] S301: Obtain the voice command to be processed.

[0123] S302: Based on the voice command to be processed, determine the target interface element corresponding to the voice intent of the pre-trained multimodal smart cockpit interaction model.

[0124] S303: Determine the target coordinate information corresponding to the target interface element based on the pre-saved coordinate information corresponding to each interface element.

[0125] S304: Based on the target coordinate information, operate the target interface elements in the current interface of the target smart cockpit system.

[0126] The "seen and speakable" smart cockpit method provided in this application is applied to computer equipment, which can be a smart terminal, such as an in-vehicle central control unit or in-vehicle processor, or a server, such as an in-vehicle business server. The computer equipment used in this application for "seen and speakable" smart cockpit operation can be the same as, or different from, the computer equipment used for training the large-scale multimodal smart cockpit interaction model described above.

[0127] In one possible implementation, the training of the multimodal smart cockpit interaction model is generally carried out offline. After obtaining the trained multimodal smart cockpit interaction model, it can be deployed to the computer device used for the smart cockpit's "see-and-talk" functionality.

[0128] In this application, sensors (such as microphones and speakers) for collecting voice signals are pre-arranged in an array and installed in the vehicle equipped with the target intelligent cockpit system, based on the size and shape of the vehicle, to ensure accurate capture of sound signals from all directions inside the vehicle. For example, microphones are installed around the driver's seat, in the passenger area, and even in the trunk to achieve comprehensive monitoring of all important sound sources inside the vehicle. Subsequently, during the operation of the vehicle equipped with the target intelligent cockpit system, voice signals from various areas inside the vehicle can be continuously collected through microphones in different areas of the vehicle's interior.

[0129] In one possible implementation, the acquired speech signal can be preprocessed. This preprocessing includes, but is not limited to, one or more of the following: speech enhancement, speech signal framing, and windowing. Speech enhancement can remove environmental noise and echo interference. Speech signal framing and windowing can convert continuous speech signals into a series of short, high-quality speech frames.

[0130] As one possible implementation, the speech signal can be directly identified as the speech command to be processed, or the preprocessed speech signal can be identified as the speech command to be processed. The preprocessing includes, but is not limited to, one or more of the following: speech enhancement processing, speech signal framing, and windowing operations.

[0131] As another possible implementation, speech feature vectors (e.g., Mel-frequency cepstral coefficients (MFCC), linear predictive cepstral coefficients (LPCC), etc.) can be determined as the speech commands to be processed, thereby reducing the computational load of subsequent multimodal smart cockpit interaction models. These speech feature vectors can be the speech feature vectors of the speech signal itself or the speech feature vectors of the preprocessed speech signal. For example, the acquired speech signal is preprocessed. This preprocessing includes, but is not limited to, one or more of the following: speech enhancement processing, speech signal framing, and windowing operations. Based on each preprocessed speech frame, the speech feature vector of each speech signal is obtained, and the speech feature vector of each speech signal is determined as the speech command to be processed.

[0132] After obtaining the voice command to be processed based on the above embodiments, the voice command can be input into a pre-trained multimodal smart cockpit interaction model. Through this multimodal smart cockpit interaction model, the target interface element corresponding to the voice intent of the voice command is determined. In this process, the multimodal smart cockpit interaction model performs semantic analysis on the voice command, combining the previously learned mapping relationship between voice intent and interface elements of various interface images in the target smart cockpit system, and comprehensively judges the target interface element that best matches the voice intent of the voice command.

[0133] After identifying the target interface elements and key information, the target coordinates of the target interface element are determined based on the pre-saved coordinate information corresponding to each interface element. For example, for the play / pause button in a music playback interface, the target coordinates of the target interface element can be obtained by querying the pre-saved coordinate information corresponding to each interface element.

[0134] With the target coordinates information, target interface elements can be manipulated within the current interface of the target smart cockpit system. For example, an automated operation simulation tool can be used to manipulate target interface elements at the target coordinates. In an Android-based target smart cockpit system, Android's Instrumentation framework or Accessibility Service can be used to simulate click events, triggering corresponding interface transitions or functional operations.

[0135] In one possible implementation, after determining the target coordinate information corresponding to the target interface element based on the pre-saved coordinate information corresponding to each interface element, and before operating the target interface element in the current interface of the target smart cockpit system based on the target coordinate information, the method further includes:

[0136] The target coordinate information is confirmed to be valid in the current interface.

[0137] To ensure operational accuracy, this application, after determining the target coordinate information, determines whether the target coordinate information is valid within the current interface of the target smart cockpit system. Only if the target coordinate information is determined to be valid within the current interface is the subsequent step of operating the target interface elements within the current interface of the target smart cockpit system based on the target coordinate information executed, to avoid coordinate errors of interface elements due to interface changes or other abnormal situations. Here, "valid" means that the target coordinate information is within the valid range of the current interface. For example, after obtaining the target coordinate information, it can be determined whether the target coordinate information is within the valid range of the current interface. If it is determined that the target coordinate information is within the valid range of the current interface, then the target coordinate information is determined to be valid; if it is determined that the target coordinate information is not within the valid range of the current interface, then the target coordinate information is determined to be invalid.

[0138] As one possible implementation, if it is determined that the target coordinate information is invalid in the current interface, the method further includes:

[0139] Get the latest interface image of the current interface;

[0140] The latest coordinate information corresponding to each interface element in the latest interface image is redefined;

[0141] Based on the latest coordinate information corresponding to each interface element in the newly determined latest interface image, update the coordinate information corresponding to each interface element.

[0142] Based on the updated coordinate information of each interface element, the target coordinate information corresponding to the target interface element is determined.

[0143] Based on the target coordinate information, operate the target interface element in the current interface.

[0144] If the target coordinate information is determined to be invalid based on the above embodiments, the latest interface image of the current interface is obtained. For example, the latest interface image can be obtained by taking a screenshot or recording the screen.

[0145] Then, the latest coordinate information corresponding to each interface element in the latest interface image is re-determined. Specifically, in determining the latest coordinate information corresponding to each interface element in the latest interface image, each interface element must first be re-identified, and then its coordinate information is determined.

[0146] For example, the newly acquired interface image is first preprocessed. This preprocessing includes one or more of the following: grayscale processing, image noise reduction, and data augmentation. After preprocessing, object detection and recognition algorithms, such as deep learning-based convolutional neural network (CNN) algorithms, are used to identify various interface elements in the image. These algorithms can learn and extract the features of interface elements, thereby accurately determining the type of each element and its latest coordinate information in the latest interface image.

[0147] When updating the coordinate information of each interface element based on the latest coordinate information of each interface element in the newly determined interface image, the newly acquired coordinate information can be compared and replaced with the original coordinate information. If the original coordinate information table is built on a relational database, then SQL statements can be used to update the data to ensure that the coordinate information of each interface element is up-to-date and accurate.

[0148] After updating the coordinate information, the system determines the target coordinates of the target interface element based on the updated coordinates of each interface element. Then, the system uses the multimodal smart cockpit interaction model to understand the voice commands and the location of the target interface element to retrieve the accurate coordinates of the target interface element from the updated coordinate information database.

[0149] Finally, based on the determined target coordinates, the target interface elements are manipulated within the current interface.

[0150] As one possible implementation, after operating the target interface elements in the interface of the target smart cockpit system based on the target coordinate information, the method further includes:

[0151] Based on the feedback information from the current interface, determine whether the operation on the target interface element was successful;

[0152] If the operation is successful, record the successful operation log;

[0153] If the operation fails, determine the cause of failure; determine and execute the pre-configured remedial measures corresponding to the cause of failure; and record the failure operation log.

[0154] To ensure the accuracy and stability of the operation, the system monitors the feedback information of the current interface in real time during the operation to determine whether the operation on the target interface element was successful. This feedback information may include changes in the state of interface elements and system prompts. For example, in the case of adjusting the air conditioner temperature, if the displayed temperature value changes to 26 degrees Celsius, or if the system displays a "Temperature adjusted to 26 degrees Celsius" prompt, the operation can be considered successful.

[0155] If the operation is successful, a success operation log is recorded. The log entry can include the operation time, the voice command, the corresponding target interface element, and the specific operation details. For example, it might record, "[Specific time], the user issued the command 'Adjust the air conditioning temperature to 26 degrees,' and operated the temperature adjustment slider on the air conditioning control interface to set the temperature to 26 degrees." The success operation log can be used for further optimization of the multimodal smart cockpit interaction model.

[0156] If the operation fails, determine the cause of failure. There may be various reasons for failure, such as the target interface element being obscured and unable to be operated, a temporary system malfunction, or a network connection problem. The specific cause is determined by analyzing the feedback information from the current interface, system logs, and the status data of relevant hardware devices. For example, if the system prompts "Network connection abnormal, operation cannot be completed," the cause of failure is determined to be a network problem. Identify and execute the pre-configured remedial measures corresponding to the failure cause. If the failure cause is a network problem, try reconnecting to the network, such as automatically switching the network connection mode from 4G to Wi-Fi, or restarting the network module; if the target interface element is obscured, guide the user to manually adjust the interface display through interface prompts, or try automatically adjusting the interface layout to display the obscured element. Record the failed operation log. The log content can record the failure time, the voice command of the operation, the corresponding target interface element, the specific operation content, the cause of failure, and the remedial measures executed. For example, record "[Specific time], the user issued the command 'Adjust the air conditioner temperature to 26 degrees', operated the temperature adjustment slider on the air conditioner control interface, the operation failed due to a network connection abnormality, and an attempt has been made to reconnect to the network." Failure operation logs help in the subsequent troubleshooting and repair of the target intelligent cockpit system, and continuously improve the stability and reliability of the target intelligent cockpit system.

[0157] In practical applications, to further improve the performance and user experience of the "see-and-talk" approach in smart cockpits, actual user feedback data is collected during use. The model's processing effectiveness in different scenarios is analyzed, addressing issues such as voice command recognition errors and inaccurate interface element positioning. The multimodal smart cockpit interaction model is regularly optimized and updated, with targeted adjustments to its parameters and structure to improve accuracy and robustness. Simultaneously, as vehicle hardware is upgraded and new application scenarios emerge, the coordinate information database and related knowledge of interface elements are continuously updated to adapt to the dynamic changes in the smart cockpit system.

[0158] Example 4:

[0159] The "smart cockpit that is visible and speakable" method provided in this application will be described below through specific embodiments. Figure 4 This application provides a schematic diagram of a smart cockpit process that is visible and speakable, comprising the following steps:

[0160] S401: Acquired voice signal to be processed.

[0161] S402: Perform speech preprocessing on the speech signal to be processed.

[0162] Among them, speech preprocessing includes, but is not limited to, one or more of the following: speech enhancement processing, speech signal framing, and windowing operation.

[0163] S403: Determine the speech command to be processed based on the speech feature vector of the preprocessed speech signal to be processed.

[0164] S404: Using a pre-trained multimodal smart cockpit interaction model, the target interface elements corresponding to the voice intent of the voice command to be processed are determined based on the voice command to be processed.

[0165] S405: Determine the target coordinate information corresponding to the target interface element based on the pre-saved coordinate information corresponding to each interface element.

[0166] S406: Determine whether the target coordinate information is valid in the current interface of the target intelligent cockpit system. If yes, execute S412; otherwise, execute S407.

[0167] S407: Capture the latest screen image of the current interface by taking a screenshot.

[0168] S408: Perform image preprocessing on the latest interface image.

[0169] The image preprocessing includes one or more of the following: grayscale processing, image noise reduction, and data augmentation.

[0170] S409: Re-identify each interface element and then determine its coordinate information.

[0171] S410: Update the coordinate information corresponding to each interface element based on the latest coordinate information corresponding to each interface element in the newly determined interface image.

[0172] S411: Based on the updated coordinate information of each interface element, determine the target coordinate information of the target interface element again, and execute S406.

[0173] S412: Based on the target coordinate information, manipulate the target interface elements in the current interface.

[0174] S413: Based on the feedback information from the current interface, determine whether the operation on the target interface element was successful. If yes, execute S414; otherwise, execute S415.

[0175] S414: Record successful operation logs.

[0176] S415: Determine the cause of failure, identify and execute the pre-configured remedial measures corresponding to the cause of failure, and record the failure operation log.

[0177] Example 5:

[0178] Based on the same inventive concept, this application also provides a training device for a large-scale multimodal smart cockpit interaction model. Figure 5 This is a schematic diagram of the structure of a training device for a multimodal smart cockpit interaction model provided in an embodiment of this application. The device includes:

[0179] The first acquisition unit 51 is used to acquire a first sample set containing attribute information samples corresponding to multiple interface elements respectively; wherein, the attribute information sample of any interface element includes one or more of the following: operation-related information of the interface element, the self-features of the interface element, text descriptions related to the interface element, and context information of the interface element, and each attribute information sample corresponds to a real functional meaning label.

[0180] The first training unit 52 is used to train a pre-trained multimodal basic large model based on each attribute information sample in the first sample set and the real functional meaning label corresponding to each attribute information sample, so as to obtain a trained multimodal smart cockpit interaction large model; wherein, the multimodal basic large model is a large model that already has multimodal data processing capabilities and natural language understanding capabilities, and the multimodal smart cockpit interaction large model has the ability to identify the functional meaning of the interface elements contained in each interface of the target smart cockpit system.

[0181] The second acquisition unit 53 is used to acquire a second sample set containing multiple voice command samples related to interaction with the target smart cockpit system; wherein each voice command sample corresponds to a voice intent tag and an interface element tag;

[0182] The second training unit 54 is used to train the multimodal smart cockpit interaction model based on each of the voice command samples in the second sample set, and the voice intent labels and interface element labels of each of the voice command samples, so that the multimodal smart cockpit interaction model learns the mapping relationship between voice intent and interface elements of each interface image in the target smart cockpit system.

[0183] In this embodiment, the training device for the multimodal smart cockpit interaction model is presented in the form of functional modules. Here, a module refers to an application-specific integrated circuit (ASIC), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0184] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0185] Example 6:

[0186] Based on the same inventive concept, this application also provides a smart cockpit device that allows users to speak as soon as they see it. Figure 6 This application provides a schematic diagram of the structure of a smart cockpit device that allows users to see and speak freely, the device comprising:

[0187] Acquisition module 61 is used to acquire the voice command to be processed;

[0188] The interface element determination module 62 is used to determine the target interface element corresponding to the voice intent of the voice command to be processed based on the pre-trained multimodal smart cockpit interaction model.

[0189] The coordinate determination module 63 is used to determine the target coordinate information corresponding to the target interface element based on the pre-saved coordinate information corresponding to each interface element.

[0190] The operation module 64 is used to operate the target interface elements in the current interface of the target smart cockpit system according to the target coordinate information.

[0191] In this embodiment, the smart cockpit device is presented in the form of functional modules. Here, a module refers to an application-specific integrated circuit (ASIC), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0192] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0193] Example 7:

[0194] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application, such as... Figure 7 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0195] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0196] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0197] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0198] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0199] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0200] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0201] Example 8:

[0202] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to perform the following steps:

[0203] Obtain a first sample set containing attribute information samples corresponding to multiple interface elements; wherein, the attribute information sample of any interface element includes one or more of the following: operation-related information of the interface element, the self-features of the interface element, text descriptions related to the interface element, and context information of the interface element, and each attribute information sample corresponds to a label with a real functional meaning.

[0204] Based on the attribute information samples in the first sample set and the corresponding real functional meaning labels, the pre-trained multimodal basic model is trained to obtain the trained multimodal smart cockpit interaction model. The multimodal basic model is a large model that already has multimodal data processing and natural language understanding capabilities, and the multimodal smart cockpit interaction model has the ability to identify the functional meaning of the interface elements contained in each interface of the target smart cockpit system.

[0205] Obtain a second sample set containing multiple voice command samples related to interaction with the target smart cockpit system; wherein each voice command sample corresponds to a voice intent tag and an interface element tag;

[0206] Based on the voice command samples in the second sample set, and the voice intent labels and interface element labels of each voice command sample, the multimodal smart cockpit interaction model is trained so that the multimodal smart cockpit interaction model learns the mapping relationship between voice intent and interface elements of each interface image in the target smart cockpit system.

[0207] Since the principle of the computer-readable storage medium in solving the problem is similar to the training method of the multimodal intelligent cockpit interaction model, the implementation of the computer-readable storage medium can be found in Examples 1-2 of the method, and the repeated parts will not be described again.

[0208] Example 9:

[0209] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to perform the following steps:

[0210] Obtain the voice commands to be processed;

[0211] By using a pre-trained multimodal smart cockpit interaction model, the target interface elements corresponding to the voice intent of the voice command to be processed are determined based on the voice command to be processed.

[0212] Based on the pre-saved coordinate information corresponding to each interface element, determine the target coordinate information corresponding to the target interface element;

[0213] Based on the target coordinate information, operate the target interface elements in the current interface of the target smart cockpit system.

[0214] Since the principle of the computer-readable storage medium in solving the problem is similar to that of the smart cockpit "see and say" method, the implementation of the computer-readable storage medium can be found in embodiments 3-4 of the method, and the repeated parts will not be described again.

[0215] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for training a multi-modal intelligent cockpit interaction large model, characterized in that, The method comprises: obtaining a first sample set containing attribute information samples corresponding to a plurality of interface elements respectively; wherein the attribute information sample of any interface element comprises one or more of the following: operation-related information of the interface element, self-characteristics of the interface element, text description related to the interface element, and context information of the interface element, and each attribute information sample corresponds to a real function meaning label; based on each attribute information sample in the first sample set and the real function meaning label corresponding to each attribute information sample, a pre-trained multi-modal base large model is trained to obtain a trained multi-modal intelligent cockpit interaction large model; wherein the multi-modal base large model is a large model that has multi-modal data processing capability and natural language understanding capability, and the multi-modal intelligent cockpit interaction large model has the capability to identify the function meaning of each interface element contained in each interface of a target intelligent cockpit system; obtaining a second sample set containing a plurality of voice instruction samples related to the target intelligent cockpit system; wherein any voice instruction sample corresponds to a voice intent label and an interface element label; based on each voice instruction sample in the second sample set and the voice intent label and the interface element label of each voice instruction sample, the multi-modal intelligent cockpit interaction large model is trained to enable the multi-modal intelligent cockpit interaction large model to learn the mapping relationship between voice intent and interface elements of each interface image in the target intelligent cockpit system.

2. The method of claim 1, wherein, The self-characteristics of each interface element in the interface image are obtained in the following manner: for any interface image, various types of interface elements contained in the interface image are determined by an image recognition algorithm; for interface elements of the button type in the interface image, the visual characteristics and bounding box coordinates of each button in the interface image are determined by a pre-trained button recognition algorithm; for interface elements of the text box type in the interface image, the text box in the interface image is converted into an editable text string by optical character recognition technology, and the bounding box coordinates of the region where the text box is located are recorded.

3. The method of claim 1 or 2, wherein, obtaining interface images in the target intelligent cockpit system comprises: obtaining interface images in the target intelligent cockpit system by periodic screenshot or screen recording.

4. The method of claim 3, wherein, The method further comprises: image preprocessing is performed on the interface image; wherein the image preprocessing comprises one or more of the following: grayscale processing, image noise reduction, and data enhancement.

5. The method of claim 1, wherein, The training of the multi-modal intelligent cockpit interaction large model based on each voice instruction sample in the second sample set and the voice intent label and the interface element label of each voice instruction sample to enable the multi-modal intelligent cockpit interaction large model to learn the mapping relationship between voice intent and interface elements of each interface image in the target intelligent cockpit system comprises: obtaining any voice instruction sample and its corresponding voice intent label and interface element label; The multi-modal intelligent cockpit interaction large model is used to output a first probability distribution of a predicted voice intent of the voice instruction sample and a second probability distribution of an interface element corresponding to the predicted voice intent based on the voice instruction sample and the function meanings of the interface elements contained in the target intelligent cockpit system. The multi-modal intelligent cockpit interaction large model is trained based on the first probability distribution and the voice intent label and the second probability distribution and the interface element label, so that the multi-modal intelligent cockpit interaction large model learns the mapping relationship between the voice intent and the interface element.

6. The method of claim 1, wherein, Any of the voice instruction samples is obtained by the following method: The obtained voice signal is subjected to voice preprocessing, wherein the voice preprocessing includes one or more of the following: voice enhancement processing, voice signal framing, and windowing operation; The voice feature vector of the preprocessed voice signal is obtained, and the voice feature vector is determined as the voice instruction sample.

7. An intelligent cockpit visible-to-say method based on a model trained by the method of any one of claims 1-6. The method comprises: Obtaining a voice instruction to be processed; Based on the voice instruction to be processed, a target interface element corresponding to a voice intent of the voice instruction to be processed is determined by using a pre-trained multi-modal intelligent cockpit interaction large model; Target coordinate information corresponding to the target interface element is determined according to pre-stored coordinate information respectively corresponding to each interface element; The target interface element is operated in the current interface of the target intelligent cockpit system according to the target coordinate information.

8. The method of claim 7, wherein, After the target coordinate information corresponding to the target interface element is determined according to the pre-stored coordinate information respectively corresponding to each interface element, before the target interface element is operated in the current interface of the target intelligent cockpit system according to the target coordinate information, the method further comprises: It is determined that the target coordinate information is valid in the current interface.

9. The method of claim 8, wherein, If it is determined that the target coordinate information is invalid in the current interface, the method further comprises: Obtaining a latest interface image of the current interface; Redetermining the latest coordinate information respectively corresponding to each interface element in the latest interface image; The coordinate information respectively corresponding to each interface element is updated according to the re-determined latest coordinate information respectively corresponding to each interface element in the latest interface image; Based on the updated coordinate information respectively corresponding to each interface element, the target coordinate information corresponding to the target interface element is determined; The target interface element is operated in the current interface according to the target coordinate information. 10.The smart cabin seeable-say method of claim 7 or 9, wherein, After the target interface element is operated in the interface of the target intelligent cockpit system according to the target coordinate information, the method further comprises: It is determined whether the operation on the target interface element is successful according to the feedback information of the current interface; If the operation is successful, a successful operation log is recorded; If the operation fails, the failure reason is determined, a pre-configured repair measure corresponding to the failure reason is determined and executed, and a failed operation log is recorded.

Images