Vehicle control method and device based on user intention recognition and electronic equipment

Abstract

The embodiment of the invention provides a vehicle control method and device based on user intention recognition and electronic equipment. The method comprises the following steps: scanning and classifying processed to-be-identified data by the electronic equipment, and searching to obtain a corresponding large model combination according to a preset search strategy; the large model combination can comprise a plurality of large models. The electronic device can input the to-be-recognized data into the large model combination, deep recognition of the data of multiple data modalities in the to-be-recognized data is achieved, and the user intention is obtained through extraction. The user intent may include vehicle control instructions. The electronic device can parse and execute the vehicle control instructions in the user intention. The method is used for achieving the effect of improving the recognition precision of the user intention.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a vehicle control method, device and electronic equipment based on user intention recognition. Background Art

[0002] With the rapid development of intelligent technology, in-vehicle intelligent interaction systems have gradually become an important part of the driving experience. However, existing in-vehicle intelligent interaction systems have the problem of inaccurate intention prediction. Summary of the invention

[0003] The embodiments of the present application provide a vehicle control method, device and electronic device based on user intention recognition, so as to achieve the effect of improving the accuracy of intention prediction.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, an embodiment of the present application provides a vehicle control method based on user intention recognition, comprising:

[0006] In response to obtaining the data to be identified, determining a large model combination corresponding to the data to be identified; wherein the large model combination includes at least one large model; the large model corresponds to the data modality of the data to be identified;

[0007] Based on the large model in the large model combination, the data of the data modality corresponding to the large model in the data to be identified is identified to obtain the user intention; wherein the user intention represents the vehicle control task for the vehicle;

[0008] Execute the vehicle control task represented by the user intention.

[0009] In a second aspect, an embodiment of the present application provides a vehicle control device based on user intention recognition, comprising:

[0010] A processing module, configured to, in response to obtaining data to be identified, determine a large model combination corresponding to the data to be identified; wherein the large model combination includes at least one large model; the large model corresponds to the data modality of the data to be identified; based on the large model in the large model combination, identify the data of the data modality corresponding to the large model in the data to be identified to obtain a user intention; wherein the user intention represents a vehicle control task for the vehicle;

[0011] An execution module is used to execute the vehicle control task represented by the user intention.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0013] The memory stores computer-executable instructions;

[0014] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0017] Beneficial effects of the present invention:

[0018] This application determines the business category by acquiring identification data, and then determines the large model combination based on the business category. The large model combination corresponding to the business category can accurately identify the user's intention, and on the basis of ensuring the accuracy of intention recognition, it reduces the use of unnecessary large models and improves the prediction efficiency of user intention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 Schematic diagram of the process of using the large model provided by the existing technology Figure 1 ;

[0021] Figure 2 Schematic diagram of the process of using the large model provided by the existing technology Figure 2 ;

[0022] Figure 3 Schematic diagram of the process used for the large model provided for this application Figure 1 ;

[0023] Figure 4 Schematic diagram of the process used for the large model provided for this application Figure 2 ;

[0024] Figure 5 Schematic diagram of the process of the vehicle control method based on user intention recognition provided in this application Figure 1 ;

[0025] Figure 6 A schematic diagram of an example of vehicle control based on user intention recognition provided by this application;

[0026] Figure 7 Schematic diagram of the process of the vehicle control method based on user intention recognition provided in this application Figure 2 ;

[0027] Figure 8 Module diagram of the vehicle control method based on user intention recognition provided in this application Figure 1 ;

[0028] Fig. 9 Module diagram of the vehicle control method based on user intention recognition provided in this application Figure 2 ;

[0029] Fig.10 A schematic diagram of the structure of vehicle control based on user intention recognition provided by this application;

[0030] Fig.11 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0031] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0032] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0033] With the development of intelligent technology, in-vehicle intelligent interaction systems are gradually becoming an indispensable core part of the driving experience. How to improve the accuracy of vehicle intention prediction during intelligent interaction and improve user experience has become an important direction of current vehicle research and development.

[0034] At present, agilely capturing and processing user voice commands is a basic requirement for vehicles in the process of predicting user intentions. In addition, the vehicle can also deeply integrate various sensor data inside and outside the vehicle, vehicle interface information, and high-definition image data inside and outside the cockpit to achieve accurate insight into and immediate response to user needs. However, the challenges faced in this process cannot be ignored: the complex semantics and potential multiple intentions of user voice commands, coupled with the ubiquitous noise interference in the vehicle environment, the richness and diversity of sensor data, and the ever-changing characteristics of driving scenarios, together constitute a severe test of the system's ability to accurately understand and efficiently execute. Therefore, how to further improve the comprehensive performance of the in-vehicle intelligent interaction system has become an urgent problem to be solved.

[0035] The emergence of big models has made it possible to predict user intentions through artificial intelligence methods based on a large amount of data such as various sensor data inside and outside the car, vehicle interface information, and high-definition image data inside and outside the cockpit. In the existing technology, the use of big models can be as follows: Figure 1 As shown in FIG. 1 , the electronic device can use the application programming interface (API) to implement the call of a large model, thereby using the capabilities of the large model to parse the business-related data and obtain the final recognition result. The electronic device can use the recognition result as the return result of the business to complete a business request. Specifically, the execution process of the large model can be as follows: Figure 2 As shown. After the user inputs voice information, the electronic device can distribute the voice information to the corresponding semantic processing model through semantic distribution, business distribution and other modes, and then obtain the corresponding user intention and complete the business execution. However, the process of the big model and the business has a strong binding relationship, and there is a problem of a single distribution strategy. In addition, the user intention identified by the big model is usually not usable.

[0036] In order to solve these problems, this application introduces a vertical domain large model, a multimodal large model and an intention prediction large model. Among them, the vertical domain large model is used to process complex contexts, semantic parsing and understanding of multiple intentions in one sentence. This type of model can deeply analyze voice commands based on the user's historical commands and current environment, and generate relevant operation recommendations through the intention prediction model. At the same time, the multimodal large model further enhances the system's understanding of complex scenes and its ability to accurately convert commands by integrating various information such as the car screen, environmental perception data inside and outside the cabin, and driving sensor signals. These multiple large models form a generative human-computer interaction algorithm model fusion architecture. This architecture can support multimodal data processing, complex dialogues, intent prediction, multimodal cabin-driving collaborative control, etc.

[0037] In order to further improve the execution efficiency of the vehicle computer in intention prediction, this application also introduces a large model combination. After obtaining the data to be identified, the electronic device can determine the large model combination required to process the data to be identified based on the data to be identified. The electronic device can implement the optimal processing solution for the data to be identified based on the large model combination, thereby improving the processing efficiency of the data to be identified and the accuracy of the predicted user intention.

[0038] Specifically, in this application, the calling process of the large model can be as follows Figure 3 After acquiring the data to be identified, the electronic device can distribute the data to be identified to the corresponding large model through the distribution module. Optionally, the electronic device can distribute the data to be identified to one or more large models. Optionally, as Figure 3 As shown, the big model may include at least a conversation vertical domain big model, an intention prediction big model, and a multimodal big model. Optionally, the distribution model may determine the big model to be distributed according to the modality of the data to be identified and the corresponding business type. The big model may obtain the corresponding processing result after processing the data. Optionally, the processing result of the conversation vertical domain big model may be the result of intelligent speech after language analysis. The processing result of the intention prediction big model may be the result of intelligent recommendation after intention analysis. The processing result of the multimodal big model may be the result of intelligent perception after perception analysis. The processing result may continue to be distributed to different business terminals to realize the processing of different businesses. For example, business 1 only needs to process the intelligent speech through the conversation vertical domain big model. For another example, business 2 needs to use the processing results of the above three big models. For another example, business 3 needs to use the processing results of the intention prediction big model and the multimodal big model. That is, in the present application, different big model combinations may be set according to different business types. For example, the big model combination of business 1 may include a conversation vertical domain big model. The big model combination of business 2 includes the big model of vertical domain dialogue, the big model of intention prediction, and the big model of multimodal. The big model combination of business 3 includes the big model of intention prediction and the big model of multimodal. The architecture of this big model combination supports complex dialogue, intention prediction, and multimodal cockpit cooperative control, further improving the accuracy of user intentions.

[0039] like Figure 4 As shown, taking user voice input as an example, the electronic device can send voice information to the data distribution module. Data distribution can select an interface based on the data source, data combination, text semantics, data type and other information of the voice information. Each interface can correspond to at least one large model. According to the mapping relationship between the interface and the large model, the electronic device can call and execute the large model by calling the interface. This method can achieve the effect of dynamically calling the large model according to the input data to be identified. The flexible calling of the large model can better expand the task continuity.

[0040] Therefore, compared with the use of large models in the prior art, the present application avoids the problem of single use of large models in the prior art by combining large models, improves the flexibility of using large models, improves the processing efficiency of data to be identified, and improves the accuracy and effectiveness of identification results. The present application determines the method of combining large models according to the business type, improves the convenience of using large models, makes the use of large models more in line with business needs, and further improves the accuracy and effectiveness of identification results. In addition, the setting method facilitates users to optimize the large model combination when business needs change, increase or decrease, thereby ensuring the use effect of the large model combination. The present application determines the business type corresponding to the data to be identified through the distribution module, thereby determining the API to be called and the method of using the large model, improving the completion of scene tasks in the vehicle, realizing the flexible combination of different model resources, and expanding business continuity. In addition, the use of large model combinations based on business scenarios in the present application also extends the use link of large models and improves the utilization rate of large models.

[0041] In addition, the present application can also use the information security check module as a pre-processing and post-processing module. The use of the information security check module ensures the security of the algorithm content in the entire process.

[0042] Figure 5 Schematic diagram of the process of the vehicle control method based on user intention recognition provided in this application Figure 1 ,like Figure 5 As shown, the method includes:

[0043] S101: In response to obtaining data to be identified, determine a large model combination corresponding to the data to be identified, wherein the large model combination includes at least one large model. The large model corresponds to the data modality.

[0044] In this embodiment, when the electronic device receives the data to be identified, it first starts the built-in preprocessing module to perform preliminary cleaning, formatting and feature extraction on the data to ensure that the data meets the requirements of subsequent processing. Subsequently, the electronic device will use the preset preliminary recognition model to quickly scan and preliminarily classify the processed data to be identified to determine its approximate data modal range. Based on the output results of the preliminary recognition model, the electronic device will further refine its search strategy and screen out large model combinations that highly match the data modality to be identified from the pre-trained large model library. These large model combinations are carefully designed and optimized by electronic devices for different types of data modalities based on historical data and experience, aiming to provide the best recognition effect and performance.

[0045] Optionally, when the electronic device determines the calling interface that needs to be called, the electronic device can distribute the data to be identified to the corresponding calling interface. Optionally, the electronic device can send the data to the found calling interface for processing through communication protocols such as HTTP requests, gRPC or WebSocket. Optionally, the electronic device can also use Python's requests library, gRPC library or other related libraries to achieve efficient data transmission.

[0046] In one example, the process of obtaining the data to be identified may specifically include:

[0047] Step 11: In response to the trigger instruction, obtain the interaction information input by the user.

[0048] In this step, when the electronic device receives the trigger instruction, the electronic device can quickly enter the response state and prepare to obtain the interactive information input by the user. Optionally, the trigger instruction can be generated when the user clicks the trigger button, touches the screen, inputs a preset trigger voice, etc. The electronic device can be built with a highly sensitive voice sensor and recognition algorithm to instantly capture and recognize the user's voice. In response to the trigger instruction, the electronic device can obtain interactive information such as voice, gestures, text, case selection operations, etc. input by the user through sensors set in the vehicle cabin.

[0049] Step 12: Obtain the trigger command response time, the vehicle's operating data, and the vehicle's external data.

[0050] In this step, while responding to the trigger command, the electronic device can also obtain the vehicle's operating data and external data in real time through the vehicle's internal sensor network and external sensor array. The operating data can include vehicle speed, acceleration, steering angle, engine status, etc., reflecting the vehicle's current dynamic behavior. The external data covers the location and dynamics of obstacles, road signs, other vehicles and pedestrians in the surrounding environment, providing the vehicle with comprehensive environmental perception capabilities.

[0051] Step 13: Generate data to be identified based on the interactive information, operating data and external data.

[0052] In this step, after obtaining the interactive information input by the user, the operating data of the vehicle at the time of the response to the trigger command, and the external data, the electronic device begins to integrate these data to obtain the data to be identified. Optionally, the data integration process may include the electronic device performing data preprocessing on the interactive information, operating data, and external data. The data preprocessing may include operations such as security verification and data cleaning. Optionally, the data integration process may also include the electronic device performing format conversion, data combing, information extraction, and other operations on the interactive information, operating data, and external data, so that the data to be identified meets the format requirements and content requirements of subsequent processing.

[0053] In one example, the electronic device may determine the corresponding large model combination according to the data to be identified. The process may specifically include:

[0054] Step 21: Identify the data to be identified and obtain the service category corresponding to the data to be identified, wherein the service category represents the category of existing functions in the vehicle.

[0055] In this step, the electronic device can obtain the data to be identified, use advanced machine learning, deep learning and other algorithm models to deeply identify the data to be identified, and determine the business category corresponding to the data to be identified. Optionally, the business category may include categories of key functions in the vehicle such as navigation, entertainment, communication, and safety assistance. Alternatively, the business category can also be a more detailed category determined based on navigation, entertainment, communication, and safety assistance functions. The advanced machine learning, deep learning and other algorithm models can classify the data to be identified based on information such as the data structure, content characteristics, and metadata identifiers of the data to be identified.

[0056] Specifically, the process of the electronic device identifying the service category pair of the data to be identified may include the following steps:

[0057] Step 211: determine the data source of the data to be identified, and determine the feature extraction model corresponding to the data source.

[0058] In this step, before processing the data to be identified, the electronic device first needs to determine the exact source of the data. Because different data sources are often accompanied by different data formats, structures and quality characteristics. The electronic device may trace the original source of the data by checking the metadata of the data, file header information, network protocol identification or interaction records with other systems. The electronic device can select or construct a feature extraction model that matches the data source according to the data source. After determining the feature extraction model, the electronic device can use the feature extraction model to extract features from the data to be identified. Optionally, the feature extraction model can be used for the electronic device to use a preset algorithm library, and flexibly select or combine feature extraction models according to the different characteristics of the data source. For example, the preset algorithm library may include text processing algorithms such as inverse document frequency (IDF) and word embedding models, and image processing algorithms such as convolutional neural networks (CNN) and image pyramids.

[0059] Optionally, a preset mapping table may be stored in the electronic device. The preset mapping table is used to indicate the correspondence between the data source and the feature extraction model. After determining the data source, the electronic device may determine the feature extraction model corresponding to the data source of the current data to be identified according to the preset mapping table. Optionally, the feature extraction model may be a preset large model.

[0060] Step 212: Based on the feature extraction model corresponding to the data source, feature processing is performed on the data to be identified to obtain data features of the data to be identified.

[0061] In this step, after the data source is identified and the corresponding feature extraction model is determined, the electronic device can input the data to be identified into the feature extraction model to obtain the data features of the data to be identified. The execution of this feature extraction can realize the extraction of information from the original data and improve the validity of the data.

[0062] Step 213: Input the data features into a preset classification model for classification processing to obtain the business category corresponding to the data to be identified.

[0063] In this step, after obtaining the data features of the data to be identified, the electronic device will use these features as input and pass them to the preset classification model for classification processing. Optionally, the preset classification model is a multi-classification model that is pre-trained by the electronic device based on historical data and business needs and has powerful pattern recognition and classification capabilities. The preset classification model can identify the user's intention in the data features and determine the category of the function that the user is most likely to need to perform. The category of this function is the business category. That is, during the classification process, the electronic device will interact with the internal algorithm and data features of the preset classification model to determine the business category to which the data to be identified belongs.

[0064] Step 22: Determine a large model combination corresponding to the data to be identified based on the business category corresponding to the data to be identified.

[0065] In this step, after determining the business category corresponding to the data to be identified, the electronic device will further determine the large model combination that matches the data to be identified based on the category information of this business category. Among them, the large model combination is an ordered combination of at least one large model. Among them, different large model combinations can include different large models. And, different large model combinations can include different execution orders of large models. The selection of large models in the large model combination and the setting of the execution order can be determined according to the processing requirements of the business category, so as to achieve accurate analysis, prediction or decision support of data to be identified of different business types. The use of this large model combination can not only improve the efficiency and accuracy of data processing, but also ensure that the electronic device can flexibly call and combine different model resources according to different business needs to meet the complex and changeable functional requirements in the vehicle.

[0066] Specifically, the process of determining the large model combination may include the following steps:

[0067] Step 221: Determine a calling interface corresponding to the business category according to the business category corresponding to the data to be identified, wherein the calling interface corresponds to the large model combination.

[0068] In this step, the electronic device will further determine the corresponding calling interface based on the business category determined after the data to be identified is classified. Specifically, a mapping relationship table between business categories and calling interfaces can be maintained inside the electronic device. The relationship mapping table can clearly indicate the interface that should be called for different business categories. By querying this mapping table, the electronic device can quickly lock the calling interface that matches the current business category, indicating the direction for subsequent data processing or business operations. Optionally, each business category can correspond to a calling interface. Each calling interface can correspond to one or more business categories.

[0069] Optionally, the relationship mapping table can use a hash table or dictionary structure to accurately map the correspondence between business categories and calling interfaces. Optionally, the relationship mapping table can be in the form of a key-value pair. For example, {"voice text":"semantic analysis API","vehicle sensor data":"driving behavior analysis API"}. When the vehicle system is started, the electronic device will automatically load and parse the relationship mapping table into the memory during the initialization process for real-time query and access. After the data to be identified has been classified and processed, the electronic device can quickly find and lock the calling interface that matches the business category through the relationship mapping table. After testing, the search process can be O(1) time complexity. Therefore, the use of the relationship mapping table in this step ensures the efficiency and smoothness of the data processing process, laying a solid foundation for subsequent business operations.

[0070] Step 222: Based on the calling interface corresponding to the business category, determine the large model combination corresponding to the calling interface to obtain the large model combination corresponding to the data to be identified.

[0071] In this step, the electronic device will further determine the big model combination corresponding to the interface based on the determined calling interface. In this scenario, there can be a close correspondence between the calling interface and the big model combination. That is, each calling interface presets a big model combination that matches it. The calling interface can implement the data processing or analysis tasks of the business category by calling the big model combination. Optionally, the electronic device can automatically identify and load the corresponding big model combination by parsing the metadata or configuration information of the calling interface. Optionally, the big model combination can include a combination of multiple big models such as a dialogue vertical domain big model, an intent prediction big model, and a multimodal big model.

[0072] Among them, the training data of the large model of the dialogue vertical domain can be information collected and processed. Optionally, the electronic device can obtain high-quality input-output data pairs through steps such as dirty data cleaning, complex data mining, one language with multiple intentions, and multi-round dialogue data construction. The data constitutes the large model of the dialogue vertical domain. In addition, the present application also specially designs the corresponding prompt data as an input prefix. The large model of the dialogue vertical domain can adopt an advanced text generation architecture, based on the Transformer Decoder architecture, and integrates innovative modules such as SwiGLU activation, attention to QKV deviation, and group query attention. The large model of the dialogue vertical domain significantly improves the understanding accuracy of complex semantic corpus. In addition, during the use of the large model of the dialogue vertical domain, the present application also optimizes the large model of the dialogue vertical domain based on the GPU environment, improves the reasoning speed and reduces the memory usage. The large model of the dialogue vertical domain of the present application is the optimal model obtained after model evaluation. The model evaluation can be specifically carried out around the three indicators of downstream task accuracy, reasoning speed, and memory usage. With the continuous evolution of product requirements and the deepening of scene exploration, the large model of the dialogue vertical domain can also continue to iterate the model effect to meet the increasingly complex dialogue needs.

[0073] Among them, the intention prediction model is based on user log data and sensor data, and an efficient training data set is constructed through fine atomic function division and data association. The intention prediction model adopts an advanced text generation architecture, and is based on the Transformer Decoder architecture, integrating advanced modules such as SwiGLU activation, attention QKV deviation, and group query attention to achieve the effect of enhancing the learning and prediction capabilities of the model. The intention prediction model of this application significantly improves the recall rate of intention prediction, while ensuring the relevance, diversity and efficiency of the recall results. In addition, the intention prediction model is also optimized for the reasoning performance in the GPU environment to achieve faster reasoning speed and lower video memory occupancy. The intention prediction model of this application is the optimal model obtained after model evaluation. Model evaluation can be specifically carried out around the three core indicators of intention prediction recall rate, reasoning speed and video memory occupancy. With the evolution of product demand and the deepening of scenario exploration, it is also possible to continuously promote the continuous optimization and improvement of model effects by implementing a flexible update and iteration strategy for the intention prediction model.

[0074] Among them, the multimodal large model is trained based on massive image and text data. These massive image and text data are pre-processed by image and text data alignment and instruction coordinate annotation to achieve accurate correspondence of multimodal information. The model integrates the advanced architecture of visual coding, text coding and large language model, and improves the model's understanding and reasoning accuracy of complex multimodal data. The multimodal model of this application improves the accuracy of model prediction by ensuring that the model can deeply understand and accurately reason about data of different modalities, and by optimizing the model reasoning speed, it meets the requirements of real-time or near-real-time application scenarios, enhances the real-time performance of the model, and reduces the consumption of computing resources by the model, reduces the memory usage and storage space requirements, and improves resource efficiency. This application uses the F1 value, reasoning speed and memory usage of downstream tasks as key evaluation indicators to achieve performance evaluation of the multimodal large model. In addition, with the development of technology and changes in demand, the multimodal large model can also be iteratively trained for continuous optimization of supplementary data sets, and explore large model compression technology to further reduce resource usage and reasoning latency, and promote the wide deployment and efficient operation of the model in practical applications.

[0075] S102: Based on the large model in the large model combination, identify the data of the data modality corresponding to the large model in the data to be identified to obtain the user intention, wherein the user intention represents the vehicle control task for the vehicle.

[0076] In this embodiment, at least one large model in the large model combination can be used to identify data from different data models. Therefore, the electronic device can use the large model in the large model combination to process the data of different data models in the data to be identified. For each data modality, the corresponding large model will play its unique advantages. For example, the image recognition large model can accurately analyze the vehicle status or surrounding environment information in the picture, the text analysis large model is good at extracting key instructions from the text input by the user, and the voice recognition large model can quickly convert the user's voice instructions into understandable text form. Through this series of efficient recognition and processing processes, the electronic device can accurately capture the user's true intentions. Subsequently, these intention information is further parsed and converted into specific vehicle control tasks. The vehicle control task is the final user intention. For example, adjusting the vehicle temperature, navigating to a designated location, opening or closing the windows, etc.

[0077] In one example, the process of the electronic device using the large model in the large model combination to serially process the data to be recognized may include the following steps:

[0078] Step 31: Repeat the following steps until the preset conditions are met.

[0079] In this step, the electronic device starts a loop processing flow. The electronic device can obtain the data to be identified from the preset data source and set an initial large model index i. The initial value of the index i is 1, which means that the processing will start from the first large model in the large model combination. Optionally, the large models in the large model combination can be preset with an execution order. The loop processing flow can end when a preset condition is met. Optionally, the preset condition can be that all large models in the large model combination have completed execution.

[0080] Step 32: Input the data to be identified into the ith large model in the large model combination. Based on the ith large model in the large model combination, identify the data modality corresponding to the ith large model in the data to be identified, and output the first identification information of the ith large model.

[0081] In this step, the electronic device can input the current data to be identified into the i-th large model in each cycle. Since each large model is usually used to identify and process data with a data modality. Therefore, the electronic device can perform format conversion and data screening on the data to be identified according to the data modality processed by the i-th large model. The electronic device can input data that conforms to the data modality processed by the i-th large model into the i-th large model. Through complex algorithms and powerful computing power, the i-th large model can extract key information from it and generate the first identification information. This information is not only an in-depth interpretation of the data corresponding to the data modality of the i-th large model in the data to be identified, but also provides an important basis for subsequent processing.

[0082] Step 33: add the first identification information of the i-th large model to the data to be identified, obtain updated data to be identified, and determine that the value of i is plus 1. Wherein, i is a positive integer greater than or equal to 1. Wherein, the identification information output by the last large model in the large model combination is the user intention.

[0083] In this step, the electronic device can add the first identification information obtained in this round of processing to the data to be identified to form an updated set of data to be identified. The execution of this step realizes the accumulation and fusion of information, so that the subsequent large model can be identified based on more comprehensive information. Subsequently, the electronic device increases the value of i by 1 so that the next large model in the large model combination can be called for processing in the next cycle. At the same time, the electronic device can return to step 31 to implement the execution of the above-mentioned loop processing flow.

[0084] When the electronic device obtains the identification information output by the last large model in the large model combination, the preset condition of the loop processing flow is met and the loop processing flow ends. The electronic device outputs the identification information output by the last large model. The identification information output by the last large model will be regarded as the final user intention. In the process of this serial processing, the output data of the previous large model can be analyzed and processed more deeply in the next large model, thereby improving the accuracy of the final user intention.

[0085] In another example, the process of the electronic device using the large model in the large model combination to perform parallel processing on the data to be recognized may include the following steps:

[0086] Step 41: Based on the data modality corresponding to the large model in the large model combination, obtain the data of the data modality from the data to be identified. Use the large model in the large model combination to identify the data of the data modality corresponding to the large model, and obtain the second identification information output by the large model in the large model combination.

[0087] In this step, at least one large model in the large model combination can be used to identify data from different data models. Therefore, after obtaining the large model combination, the electronic device can classify the data of different data models in the data to be identified to obtain data from multiple different data models. The electronic device can input the data of different data models into the corresponding large models in the large model combination respectively. The process of inputting data of different data modalities into different large models respectively is a parallel processing process of multiple large models. Each large model can process the data of the data model and identify the second identification information of the data modality. The second identification information can be used to reflect the content of the data of the data model itself, as well as the deep meaning and relationship. The parallel processing process improves the processing effectiveness of multimodal data and improves the recognition accuracy of the second identification information.

[0088] Step 42: Determine the user intention based on the second identification information output by each large model in the large model combination.

[0089] In this step, the electronic device can collect the second identification information output by all the large models after all the large models have completed their respective identification tasks. The electronic device can use a preset decision model to process these second identification information to obtain the final user intention. These second identification information can complement and verify each other, and together constitute a comprehensive interpretation of the user's intention.

[0090] The process of the electronic device deciding to obtain the final second identification information according to the plurality of second identification information may include the following process:

[0091] Step 421: Determine a preliminary intention based on the second identification information output by the large model. The preliminary intention includes at least one function to be controlled and / or at least one control information. The function to be controlled is an existing function in the vehicle, and the control information is used to control the existing function in the vehicle.

[0092] In this step, after receiving the second identification information output by each large model in the large model combination, the electronic device can refine the data in the second identification new information to obtain the user's preliminary intention. These preliminary intentions may cover the functions to be controlled that the user wants to adjust. And / or, these preliminary intentions may also cover the control information that the user wants to execute. Among them, the functions to be controlled are existing functions in the vehicle. For example, adjusting the seat position, adjusting the air-conditioning temperature, playing music, etc. Among them, the control information can be a specific control of the existing functions in the vehicle. For example, "move the seat back two spaces", "set the air-conditioning temperature to 24 degrees", etc.

[0093] Since the electronic device can obtain multiple second identification information, the electronic device can extract multiple preliminary intentions. Each preliminary intention may include at least one function to be controlled and / or at least one control information. That is, the multiple second identification information may include multiple functions to be controlled and multiple control information.

[0094] Step 422: Determine the user's intention by integrating the preliminary intentions corresponding to the major models.

[0095] In this step, the electronic device may include a decision model. The decision model can be used to comprehensively consider the preliminary intentions corresponding to the various models, and fuse the decisions to obtain the final user intention. Specifically, in the decision-making process, the electronic device can realize the fusion of the preliminary intentions according to the logical relationship and priority between the preliminary intentions.

[0096] Optionally, a fusion process of the preliminary intention may be:

[0097] Step 4221: Summarize the functions to be controlled and the control information included in the preliminary intentions corresponding to each major model.

[0098] In this step, the electronic device can first comprehensively summarize the functions to be controlled and the control information involved in each preliminary intention. Specifically, the electronic device can construct a detailed summary table based on the functions to be controlled and the control information in each preliminary intention. The summary table may include two fields, wherein the first field may be the function to be controlled and the second field may be the control information. That is, the electronic device can organize a function to be controlled and its corresponding control information into a row of data.

[0099] Step 4222: If there are multiple control information corresponding to the function to be controlled, and there is a conflict between the multiple control information, use a preset decision algorithm to determine the control information corresponding to the function to be controlled.

[0100] In this step, when a function to be controlled corresponds to multiple control information and there are conflicts between these control information, the multiple control information of the function to be controlled cannot be executed at the same time. At this time, the electronic device needs to make a decision to determine the control information that the user is most likely to need as the final control information of the function to be controlled. The electronic device can comprehensively consider factors such as the priority, timeliness, interdependence of the control information and the current state of the vehicle to make the most reasonable and safe decision.

[0101] In one implementation, the electronic device may obtain multiple control information corresponding to the function to be controlled, and count the number of identical instructions therein, and the electronic device may use the control information corresponding to the instruction with the largest number as the control information corresponding to the function to be controlled.

[0102] In another implementation, when the number of identical instructions in the control information is obtained by counting, the electronic device can calculate the product of each large model weight and the number, and select the control information corresponding to the instruction with the largest product as the control information corresponding to the function to be controlled.

[0103] In another implementation, the electronic device may determine the degree of matching between the multiple control information corresponding to the function to be controlled and the current scene based on the control information or execution information of other functions to be controlled related to the function to be controlled. The electronic device may select the control information with the highest matching degree as the control information corresponding to the function to be controlled.

[0104] Step 4223: Generate a vehicle control task intended by the user based on the vehicle's functions to be controlled and the control information corresponding to each function to be controlled.

[0105] In this step, after resolving all possible conflicts and determining the control information for each function to be controlled, the electronic device will generate the vehicle control tasks intended by the user based on this information, combined with the actual capabilities and current status of the vehicle. These tasks not only clearly indicate the functions to be controlled that need to be executed, but also specify in detail the control information corresponding to the functions to be controlled. The control information may include the specific parameters and steps required for execution.

[0106] In another example, the electronic device may also process the data to be identified according to a serial and parallel combination process of multiple large models preset in the large model combination, thereby achieving more accurate identification of the data to be identified and achieving more accurate prediction of the user's intention.

[0107] S103: Execute the vehicle control task represented by the user intention.

[0108] In this embodiment, after successfully parsing and determining the user's intention, the electronic device immediately enters the execution phase to implement the vehicle control task represented by the user's intention. The electronic device will generate corresponding control instructions based on the functions to be controlled specified in the user's intention and their corresponding control information. Optionally, these control instructions may include adjusting specific parameters of the seat position, setting specific values ​​of the air conditioning temperature, playing specific music tracks, etc.

[0109] In the vehicle control method based on user intention recognition provided by the embodiment of the present application, the electronic device scans and classifies the processed data to be recognized, and searches for the corresponding large model combination according to the preset search strategy. The large model combination may include multiple large models. The electronic device can input the data to be recognized into the large model combination to realize in-depth recognition of data of multiple data modes in the data to be recognized, and extract the user intention. The user intention may include vehicle control instructions. The electronic device can parse and execute the vehicle control instructions in the user intention. In the present application, accurate recognition of the data to be recognized is realized by obtaining the large model combination corresponding to the data to be recognized, the recognition accuracy of the user intention is improved, and the user experience is improved.

[0110] In one implementation, the present application may use three large models, namely, a large model for vertical domain conversation, a large model for intent prediction, and a large model for multimodal perception. Among them, the large model for vertical domain conversation can be recorded as model 1, the large model for intent prediction can be recorded as model 2, and the large model for multimodal perception can be recorded as model 3. In the present application, the electronic device may include three large model combinations, corresponding to three calling interfaces respectively. Among them, the three calling interfaces may be a conversation assistant API, a conversation perception API, and a conversation perception prediction API. Among them, the large model combination corresponding to the conversation assistant API may be model 1+model 2, the large model combination corresponding to the conversation perception API may be model 1+model 3, and the large model combination corresponding to the conversation perception prediction API may be model 1+model 2+model 3. Optionally, multiple large models in the large model combination may be processed serially or in parallel. The electronic device may judge the theme of the user conversation scene based on the data to be identified, and give a business category, and determine the corresponding calling interface by the business category.

[0111] For example, Figure 6As shown, in a scenario with a complex road environment and music playing in the car, when the user triggers a query (Query) through voice "How should I drive the vehicle in this situation?", the electronic device can obtain the execution process through Query-->Distribution Model-->Driving Suggestions-->Select API (Dialogue Perception Prediction API)-->Get the image outside the car-->Multimodal Perception Big Model-->Traffic jam ahead, pedestrians crossing the road-->Dialogue Vertical Domain Big Model-->Suggestion to reduce speed, turn on hazard lights to keep distance, observe pedestrian movements-->Intention Prediction Big Model-->[Lower the music volume, turn off the music], thereby realizing the recognition of the user's intention and the vehicle controller.

[0112] Specifically, when the user triggers the inquiry by voice "How should I drive the vehicle in this situation?", the electronic device can obtain the data to be identified. The electronic device can determine that the business type corresponding to the data to be identified is driving advice through the distribution model. The electronic device can determine the corresponding calling interface according to the business type of the driving advice, and then select the API. The API selected in this application can be a dialogue perception prediction API. The large model combination corresponding to the API can include three large models: a dialogue vertical domain large model, an intention prediction large model, and a multimodal perception large model. The electronic device can obtain an image outside the vehicle from the data to be identified, and input the image outside the vehicle into the multimodal perception large model. The multimodal perception large model can identify the traffic jam ahead and the road condition information that pedestrians are crossing the road. The electronic device can input the traffic jam ahead and the road condition information that pedestrians are crossing the road into the dialogue vertical domain large model. The dialogue vertical domain large model can analyze and obtain driving advice that recommends reducing the speed, turning on the double flash to keep the distance, and observing the movement of pedestrians. The electronic device can input the driving advice that recommends reducing the speed, turning on the double flash to keep the distance, and observing the movement of pedestrians into the intention prediction large model. The intent prediction model can generate the user intent of turning down the music volume and turning off the music, and execute it.

[0113] On the basis of the above embodiment, the electronic device can also receive the processing result of the large model combination. The processing result is the user intention. The electronic device can also send the user intention to the vehicle computer and execute it. The electronic device can also monitor the status and feedback of the vehicle in real time during the execution of the vehicle computer to ensure the smooth progress of the control task. Once any abnormal situation or execution deviation is found, the electronic device will immediately start the emergency response mechanism and take corresponding corrective measures to ensure the safety of the vehicle and passengers. The electronic device can also return an error message or call the default exception handling process when determining the corresponding large model combination according to the data to be identified, if the corresponding large model group is not found according to the data to be identified. Optionally, the exception handling process can be used to capture and handle abnormal situations through a try-except exception handling mechanism.

[0114] Based on the above embodiments, taking the three large models of the conversation vertical domain large model, the intention prediction large model and the multimodal perception large model as examples, the input and output data of each large model can be as follows. Table 1 is the input data of the conversation vertical domain large model, Table 2 is the output data of the conversation vertical domain large model, Table 3 is the input data of the intention prediction large model, Table 4 is the output data of the intention prediction large model, Table 5 is the input data of the multimodal large model, and Table 6 is the output data of the multimodal large model.

[0115] Table 1

[0116]

[0117] Table 2

[0118]

[0119] Table 3

[0120]

[0121] Table 4

[0122]

[0123] Table 5

[0124]

[0125] Table 6

[0126]

[0127] Figure 7 Schematic diagram of the process of the vehicle control method based on user intention recognition provided in this application Figure 2 ,exist Figures 3 to 6 Based on the embodiment shown, Figure 7 As shown, the electronic device may also verify the data to be verified and / or the user's intention, and the verification process includes:

[0128] S201. The data to be identified and / or the user's intention are used as data to be verified, and the data to be verified is input into a preset security verification program for verification to obtain a verification result.

[0129] In this embodiment, the electronic device can also perform security verification on the data throughout the whole process. When receiving the data to be identified or capturing the user's intention, the electronic device can use the data to be identified or the user's intention as the data to be verified. The electronic device inputs the data to be verified into a preset security verification program. The security verification program can perform a detailed verification analysis through a series of complex algorithms and rules to obtain a verification result. Optionally, the security verification program not only checks the integrity and consistency of the data, but also analyzes the potential risks or abnormal patterns that may be hidden behind the data.

[0130] In one example, the process of verifying the data to be verified may specifically include:

[0131] Step 51: Determine the modality of the data to be verified.

[0132] In this step, the electronic device first determines the data modality of the data to be verified, that is, the type, format or source of the data. The accurate identification of the data modality pair can facilitate the electronic device to automatically call the corresponding security verification model. The security verification model can be a verification task that has been trained and is specifically used to process specific modality data, and can efficiently and accurately identify potential risks or anomalies in the data.

[0133] Step 52: Call the security verification model corresponding to the modality of the data to be verified to identify the data to be verified and determine the verification result of the data to be verified. The security verification model is a binary classification model corresponding to the modality. The classification result of the security verification model is safe or abnormal.

[0134] In this step, the electronic device can input the data to be verified into the selected security verification model, and the model will use the built-in algorithms and rules to conduct in-depth analysis of the data to obtain the verification result. The security verification model can be used to implement at least one of the verification methods such as data integrity verification, abnormal pattern detection, and behavior pattern matching, aiming to comprehensively evaluate the legitimacy and security of the data. After identification and analysis of the model, the electronic device will eventually obtain the verification result of the data to be verified. This result will clearly indicate whether the data has passed the verification. And, if there is an anomaly, the verification result can also include the specific nature of the anomaly and its possible impact.

[0135] S202: If the verification result indicates that the data to be verified is abnormal, generate and send an alarm message. The alarm message indicates that the data to be verified is abnormal.

[0136] In this embodiment, if the execution result of the security verification program shows that the data to be verified is abnormal, that is, it does not meet the preset security standards or triggers the early warning mechanism, the electronic device will immediately respond to the abnormality and automatically generate an alarm message containing detailed abnormal information. This alarm message clearly indicates which data to be verified has abnormalities, and may be accompanied by information such as the specific type of abnormality, the time of occurrence, and the possible scope of impact. The output of this abnormal information can facilitate administrators or users to quickly understand the problem and take corresponding measures. The electronic device can send this alarm message to the preset recipient in a timely manner through the network or other communication means. The recipient can be a system administrator, a security team, or a related user. Optionally, based on the verification result, the electronic device can further perform corresponding security measures. For example, allowing data to pass, intercepting abnormal data, generating alarm information, etc. Optionally, the security verification step can also process any data in the intention recognition process of this application.

[0137] The vehicle control method based on user intention recognition provided in the embodiment of the present application realizes verification of the data to be verified by acquiring the data to be verified and inputting the verification data into a security verification model corresponding to the data modality of the data to be verified, thereby achieving the effect of improving data security and improving system stability.

[0138] In the above Figure 7On the basis of the illustrated embodiment, the security verification process can be completed by the information security filtering and inspection module. The information security filtering and inspection module is intended to achieve effective filtering of illegal and bad information in the algorithm service through automated scripts and Python's Pandas, SQL and other data processing tools, combined with manually set hard strategies. The information security filtering and inspection module takes effect immediately when the user requests data to the server during the operation of the algorithm service. The information security filtering and inspection module can cover the data security in the whole process from algorithm acquisition to input data until the algorithm sends output data. According to the data security control process of the information security filtering and inspection module, this application can use algorithms such as keyword extraction and malicious guided sentence matching to realize abnormal mining. Specifically, this application can introduce a natural language processing (NLP) classifier in the information security filtering and inspection module, and select the bidirectional encoder representation (Bidirectional Encoder Representations from Transformers, BERT) and the generative pre-trained transformer (Generative Pre-trained Transformer, GPT) to perform in-depth analysis of text data, accurately identify prohibited words, sensitive topics and malicious intent. For image data inside and outside the cockpit, the information security filtering and inspection module can use convolutional neural networks (CNN) to classify images and screen for non-compliant content such as violence and sensitive logos. In addition, by building a multimodal classifier and integrating multi-source information such as voice, images, and sensors, the information security filtering and inspection module can also use multimodal fusion technology to achieve all-round and high-precision filtering of complex input data to ensure the compliance and security of data at all stages. The policy automatically becomes invalid after the information security module completes the output, reflecting an efficient and accurate information security management system.

[0139] Based on the above embodiment, the data format of the information security filtering module input data may be as shown in Table 7, and the data format of the information security filtering module output data may be as shown in Table 8.

[0140] Table 7

[0141]

[0142] Table 8

[0143]

[0144] Based on the above embodiments, Figure 8 and Fig. 9The figure shows the architecture diagram of the vehicle intelligent interaction system that needs to be implemented when the vehicle control method based on user intention recognition of the present application is executed. Figure 8 and Fig. 9 As shown in the figure, the system integrates data from multiple sources and multiple large models to accurately understand and respond to user commands. The following is a detailed description of each model of the in-vehicle intelligent interaction system:

[0145] Among them, the information security filtering module is the front module of the system, which is responsible for preliminary filtering of all input data to ensure that the data does not contain sensitive information, malicious content or other factors that may threaten the security of the system before entering the core processing flow. Among them, the input data may include interactive information, vehicle operation data and vehicle external data. Furthermore, the electronic device can generate data to be identified based on the interactive information, vehicle operation data and vehicle external data. The information security filtering module can also process the data before it enters other core modules to ensure the compliance and security of the input data and prevent unsafe content from affecting subsequent semantic parsing and instruction execution.

[0146] The data distribution module plays the role of a scheduler in the entire system, responsible for distributing data to a suitable large model combination according to the current scenario and the type and characteristics of the input data, thereby realizing data processing. Figure 1 Step S101 in the illustrated embodiment. The large model combination includes at least one of a dialogue vertical domain large model, an intention prediction large model and a multimodal perception large model, as well as the order in which each large model is used. The data distribution module can send the data to be identified to the large model combination after determining the large model combination to be used. The use of the large model combination can ensure the accuracy and efficiency of user intention recognition. The data distribution module also monitors the status of the vehicle environment and input data in real time, dynamically allocates resources and selects appropriate processing models according to actual needs.

[0147] Among them, the dialogue vertical domain large model focuses on processing complex semantic parsing, contextual understanding, and understanding of multiple intentions in one sentence in interaction with users. The dialogue vertical domain large model can analyze the user's current voice command, and at the same time combine the dialogue history and context information to accurately understand the user's intention and generate corresponding instructions. After receiving the user's voice text and related context information, the dialogue vertical domain large model can perform in-depth analysis to identify potential intentions and semantics. Especially in the case of multiple intentions or ambiguous instructions, the role of the dialogue vertical domain large model is particularly important.

[0148] The intention prediction model analyzes the user's historical commands, current commands, and vehicle sensor data to predict the user's potential needs and provide highly relevant intention recommendations. The intention prediction model mainly predicts and analyzes the relevance of user commands. The intention prediction model can analyze the historical and real-time data input by the system in combination with the current user commands to generate multiple possible intention recommendations and pass them to the subsequent processing module for decision-making.

[0149] The multimodal perception model integrates information from multiple data sources such as the vehicle screen, the environment perception inside and outside the cabin, and the driving sensor signals, and combines it with the user's voice commands to generate more accurate operation instructions. The model can understand and process complex vehicle environment information to ensure that the system fully responds to user needs. The multimodal perception model can integrate all relevant multimodal data, and then combine it with voice information to analyze the specific situation of the current driving scene, and finally convert it into executable operation instructions or suggestions.

[0150] The information security check module is located at the back end of the system processing flow and is responsible for performing security checks on the generated output results. The output results include semantic analysis results, intent recommendation results, operation instructions, etc. The execution of the information security check module ensures the security of the output content and prevents any unsafe operation instructions or sensitive information from being executed or displayed. The information security check module performs strict security checks on all the final output data and filters out potential security risks or sensitive information to protect user privacy and system security.

[0151] The output module summarizes the processing results of each model and generates the final semantic analysis results, intention recommendations or operation instructions. These results will be passed to the vehicle's execution system or displayed to the user. After passing all safety checks, the output module can present the results to the user or perform related operations according to the system's goals.

[0152] Through the collaborative work of the above modules, the system can accurately, quickly and safely process and execute user commands in complex in-vehicle environments, improving the human-computer interaction experience of smart vehicles.

[0153] On the basis of the above embodiment, the data format of the input data of the distribution module may be as shown in Table 9, and the data format of the output data of the distribution module may be as shown in Table 10.

[0154] Table 9

[0155]

[0156] Table 10

[0157]

[0158] Based on the above embodiment, the electronic device can also obtain training data to train and optimize at least one large model. The data format of the cyclic data of the large model can be as shown in Table 11.

[0159] Table 11

[0160]

[0161] Fig.10 A schematic diagram of the structure of the vehicle control based on user intention recognition provided by this application, such as Fig.10 As shown, the vehicle control 500 based on user intention recognition provided by this embodiment includes:

[0162] A processing module, configured to determine a large model combination corresponding to the data to be identified in response to obtaining the data to be identified; wherein the large model combination includes at least one large model; the large model corresponds to the data modality of the data to be identified; based on the large model in the large model combination, identify the data modality corresponding to the large model in the data to be identified to obtain the user intention; wherein the user intention represents the vehicle control task for the vehicle;

[0163] The execution module is used to execute the vehicle control task represented by the user's intention.

[0164] Optionally, the processing module is used to:

[0165] Repeat the following steps until the preset conditions are met;

[0166] Input the data to be identified into the i-th large model in the large model combination; based on the i-th large model in the large model combination, identify the data modality corresponding to the i-th large model in the data to be identified, and output the first identification information of the i-th large model;

[0167] Add the first identification information of the i-th large model to the data to be identified, obtain updated data to be identified, and determine that the value of i is added by 1; wherein i is a positive integer greater than or equal to 1;

[0168] Among them, the recognition information output by the last large model in the large model combination is the user intention.

[0169] Optionally, the processing module is used to:

[0170] Based on the data modality corresponding to the large model in the large model combination, data of the data modality is obtained from the data to be identified;

[0171] Using a large model in the large model combination, identifying data of a data modality corresponding to the large model, and obtaining second identification information output by the large model in the large model combination;

[0172] The user intention is determined according to the second identification information output by each large model in the large model combination.

[0173] Optionally, the processing module is used to:

[0174] Determine a preliminary intention according to the second identification information output by the large model; the preliminary intention includes at least one function to be controlled and / or at least one control information, the function to be controlled is an existing function in the vehicle, and the control information is used to control the existing function in the vehicle;

[0175] Determine the user's intention by integrating the preliminary intentions corresponding to each major model.

[0176] Optionally, the processing module is used to:

[0177] Summarize the functions to be controlled and the control information included in the preliminary intentions corresponding to each major model;

[0178] If there are multiple control information corresponding to the function to be controlled, and there is a conflict between the multiple control information, a preset decision algorithm is used to determine the control information corresponding to the function to be controlled;

[0179] The vehicle control task in the user's intention is generated according to the vehicle's functions to be controlled and the control information corresponding to each function to be controlled.

[0180] Optionally, the processing module is used to:

[0181] Identify the data to be identified, and obtain the service category corresponding to the data to be identified; wherein the service category represents the category of existing functions in the vehicle;

[0182] According to the business category corresponding to the data to be identified, determine the large model combination corresponding to the data to be identified.

[0183] Optionally, the processing module is used to:

[0184] According to the business category corresponding to the data to be identified, determine the calling interface corresponding to the business category; wherein the calling interface corresponds to the large model combination;

[0185] Based on the calling interface corresponding to the business category, the large model combination corresponding to the calling interface is determined to obtain the large model combination corresponding to the data to be identified.

[0186] Optionally, the processing module is used to:

[0187] Determine the data source of the data to be identified; and determine the feature extraction model corresponding to the data source;

[0188] Based on the feature extraction model corresponding to the data source, feature processing is performed on the data to be identified to obtain data features of the data to be identified;

[0189] The data features are input into the preset classification model for classification processing to obtain the business category corresponding to the data to be identified.

[0190] Optionally, the processing module is used to:

[0191] According to the data source and the preset mapping table, a feature extraction model corresponding to the data source is determined; wherein the preset mapping table indicates the corresponding relationship between the data source and the feature extraction model.

[0192] Optionally, the processing module is used to:

[0193] In response to the trigger instruction, acquiring interaction information input by the user;

[0194] Obtaining the trigger command response time, vehicle operation data and vehicle external data;

[0195] Generate data to be identified based on interactive information, operating data and external data.

[0196] Optionally, the processing module is used to:

[0197] The data to be identified and / or the user's intention are used as the data to be verified, and the data to be verified is input into a preset security verification program for verification to obtain a verification result;

[0198] If the verification result indicates that the data to be verified is abnormal, an alarm message is generated and sent; the alarm message indicates that the data to be verified is abnormal.

[0199] Optionally, the processing module is used to:

[0200] Determine the modality of the data to be verified; and call the security verification model corresponding to the modality of the data to be verified to identify the data to be verified, and determine the verification result of the data to be verified;

[0201] Among them, the safety verification model is a binary classification model corresponding to the modality; the classification result of the safety verification model is safe or abnormal.

[0202] The vehicle control based on user intention recognition provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0203] Fig.11 This is a schematic diagram of the structure of the electronic device provided in this application. Fig.11As shown, the electronic device 600 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the electronic device 600 further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0204] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0205] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0206] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0207] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0208] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0209] The present application also provides a vehicle, wherein the vehicle further comprises: Figure 6 The controller shown can implement the above method when executed.

[0210] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0211] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0212] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0213] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0214] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0215] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0216] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0217] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0218] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0219] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art on the basis of the present invention is within the protection scope of the present invention.

Claims

1. A vehicle control method based on user intention recognition, characterized in that: The method comprises: In response to obtaining the data to be identified, determining a large model combination corresponding to the data to be identified; wherein the large model combination includes at least one large model; the large model corresponds to the data modality of the data to be identified; Based on the large model in the large model combination, the data of the data modality corresponding to the large model in the data to be identified is identified to obtain the user intention; wherein the user intention represents the vehicle control task for the vehicle; Execute the vehicle control task represented by the user intention.

2. The method according to claim 1, characterized in that: Based on the large model in the large model combination, data of the data modality corresponding to the large model in the data to be identified are identified to obtain the user intention, including: Repeat the following steps until the preset conditions are met; Input the data to be identified into the i-th large model in the large model combination; based on the i-th large model in the large model combination, identify the data of the data modality corresponding to the i-th large model in the data to be identified, and output the first identification information of the i-th large model; Add the first identification information of the i-th large model to the data to be identified to obtain updated data to be identified, and determine that the value of i is added by 1; wherein i is a positive integer greater than or equal to 1; Among them, the recognition information output by the last large model in the large model combination is the user intention.

3. The method according to claim 1, characterized in that Based on the large model in the large model combination, data of the data modality corresponding to the large model in the data to be identified are identified to obtain the user intention, including: Based on the data modality corresponding to the large model in the large model combination, acquiring data of the data modality from the data to be identified; Using the large model in the large model combination, identifying the data of the data modality corresponding to the large model, and obtaining second identification information output by the large model in the large model combination; The user intention is determined according to the second identification information output by each major model in the major model combination.

4. The method according to claim 3, characterized in that Determining the user intention according to the second identification information output by each major model in the major model combination includes: Determine a preliminary intention according to the second identification information output by the large model; the preliminary intention includes at least one function to be controlled and / or at least one control information, the function to be controlled is an existing function in the vehicle, and the control information is used to control the existing function of the vehicle; The preliminary intentions corresponding to the major models are integrated to determine the user intention.

5. The method according to claim 4, characterized in that The preliminary intentions corresponding to the various models are integrated to determine the user intention, including: Summarizing the functions to be controlled and the control information included in the preliminary intentions corresponding to each major model; If there are multiple control information corresponding to the function to be controlled, and there is a conflict between the multiple control information, using a preset decision algorithm to determine the control information corresponding to the function to be controlled; The vehicle control task in the user's intention is generated according to the to-be-controlled functions of the vehicle and the control information corresponding to each of the to-be-controlled functions.

6. The method according to claim 1, characterized in that Determining a large model combination corresponding to the data to be identified includes: Identify the data to be identified to obtain a service category corresponding to the data to be identified; wherein the service category represents a category of existing functions in the vehicle; According to the business category corresponding to the data to be identified, a large model combination corresponding to the data to be identified is determined.

7. The method according to claim 6, characterized in that According to the business category corresponding to the data to be identified, determining a large model combination corresponding to the data to be identified, including: According to the business category corresponding to the data to be identified, determining a calling interface corresponding to the business category; wherein the calling interface corresponds to a large model combination; Based on the calling interface corresponding to the business category, a large model combination corresponding to the calling interface is determined to obtain a large model combination corresponding to the data to be identified.

8. The method according to claim 6, characterized in that Identifying the data to be identified and obtaining the business category corresponding to the data to be identified includes: Determine the data source of the data to be identified; and determine a feature extraction model corresponding to the data source; Based on the feature extraction model corresponding to the data source, feature processing is performed on the data to be identified to obtain data features of the data to be identified; The data features are input into a preset classification model for classification processing to obtain the business category corresponding to the data to be identified.

9. The method according to claim 8, characterized in that Determining a feature extraction model corresponding to the data source includes: According to the data source and the preset mapping table, a feature extraction model corresponding to the data source is determined; wherein the preset mapping table indicates the corresponding relationship between the data source and the feature extraction model.

10. The method according to any one of claims 1 to 9, characterized in that The data to be identified is obtained, including: In response to the trigger instruction, acquiring interaction information input by the user; Acquiring the trigger command response time, the vehicle operation data and the vehicle external data; The data to be identified is generated according to the interaction information, the operation data and the external data.

11. The method according to any one of claims 1 to 9, characterized in that The method further comprises: The data to be identified and / or the user intention are used as data to be verified, and the data to be verified is input into a preset security verification program for verification to obtain a verification result; If the verification result indicates that the data to be verified is abnormal, an alarm message is generated and sent; the alarm message indicates that the data to be verified is abnormal.

12. The method according to claim 11, characterized in that The data to be verified is input into a preset security verification program for verification, and a verification result is obtained, including: Determine the modality of the data to be verified; and call the security verification model corresponding to the modality of the data to be verified to identify the data to be verified, and determine the verification result of the data to be verified; Among them, the safety verification model is a binary classification model corresponding to the modality; the classification result of the safety verification model is safe or abnormal.

13. A vehicle control device based on user intention recognition, characterized in that: The device comprises: A processing module, configured to, in response to obtaining data to be identified, determine a large model combination corresponding to the data to be identified; wherein the large model combination includes at least one large model; the large model corresponds to the data modality of the data to be identified; based on the large model in the large model combination, identify the data of the data modality corresponding to the large model in the data to be identified to obtain a user intention; wherein the user intention represents a vehicle control task for the vehicle; An execution module is used to execute the vehicle control task represented by the user intention.

14. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 12.

15. A vehicle, characterized in that: The vehicle comprises the electronic device as claimed in claim 14.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images