Vehicle information interaction method and system and vehicle

Abstract

The invention discloses a vehicle information interaction method and system and a vehicle. The method comprises the steps that dialogue information of a vehicle in a target scene is detected, and dialogue content of the dialogue information is used for representing the target scene; multi-modal information associated with the dialogue information is obtained, and the multi-modal information is used for representing information in multiple modals meeting the dialogue intention of the dialogue content; an information processing model is called to conduct reasoning on the multi-modal information, a target task execution strategy of the vehicle is obtained, the information processing model is obtained by training an agent model through multi-modal information samples in different scenes, and the target task execution strategy is used for representing a rule of task operation to be executed by the vehicle; and controlling the vehicle to execute the corresponding task operation according to the target task execution strategy. The technical problem that the information interaction efficiency of vehicles is low is solved or partially solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent voice services, and in particular to a vehicle information interaction method, system and vehicle. Background Art

[0002] At present, with the development of technology, intelligent voice interaction systems (such as in-vehicle intelligent voice systems) play an increasingly important role in daily life and work. Intelligent voice interaction systems are usually based on speech recognition, natural language processing and machine learning technologies. Users can control smart homes, check weather and news, play music, send short messages, etc. through voice.

[0003] As a bridge connecting users (for example, drivers) and vehicles, the main functions of the in-vehicle intelligent voice system rely on basic voice recognition technology and direct command execution mechanism. The original intention of setting up the above method is to simplify the driver's operating procedures and improve driving safety and convenience. The current in-vehicle intelligent voice system requires the use of preset, specific voice commands to control the vehicle, and is unable to perform fuzzy recognition of conversations or infer users' potential needs. The above dependencies limit the flexibility and responsiveness of the in-vehicle intelligent voice system, requiring users to send voice commands accurately during use to avoid misoperation, which increases the complexity of human-computer interaction. Therefore, there is still a technical problem of low efficiency of vehicle information interaction.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] Embodiments of the present invention provide a vehicle information interaction method, system and vehicle to at least solve or partially solve the technical problem of low vehicle information interaction efficiency.

[0006] According to one aspect of an embodiment of the present invention, a vehicle information interaction method is provided, the method comprising: detecting dialogue information in a target scene in which a vehicle is located, wherein the dialogue content of the dialogue information is used to characterize the target scene; acquiring multimodal information associated with the dialogue information, wherein the multimodal information is used to represent information in multiple modes that satisfies the dialogue intent of the dialogue content; calling an information processing model to infer the multimodal information to obtain a target task execution strategy for the vehicle, wherein the information processing model is obtained by training an agent model using multimodal information samples in different scenes, and the target task execution strategy is used to represent the rules of the vehicle's task operation to be performed; and controlling the vehicle to perform the corresponding task operation according to the target task execution strategy.

[0007] Optionally, obtaining multimodal information associated with the dialogue information includes: performing scene recognition on the dialogue information to obtain the scene type of the target scene; determining an information perception strategy associated with the scene type, wherein the information perception strategy is used to represent a rule for detecting multimodal information under the scene type; and controlling the vehicle to detect multimodal information under the scene type according to the information perception strategy.

[0008] Optionally, determining the information awareness policy associated with the scene type includes: searching for the information awareness policy in an information awareness policy set by using the scene type, wherein the information awareness policy set includes information awareness policies respectively associated with different scene types.

[0009] Optionally, the information perception strategy set includes a speech perception strategy and / or an image perception strategy, wherein the speech perception strategy is used to represent the rules for detecting speech information for a vehicle under a scene type, and the image perception strategy is used to represent the rules for detecting image information for a vehicle under a scene type.

[0010] Optionally, the information perception strategy set includes at least one of the following: a state perception strategy associated with a first scenario type, wherein the first scenario type is used to represent a service scenario type associated with a vehicle state of a vehicle, and the state perception strategy is used to represent rules for detecting state information of the vehicle state under the first scenario type; an environment perception strategy associated with a second scenario type, wherein the second scenario type is used to represent a service scenario type associated with a surrounding geographical environment of the vehicle, and the environment perception strategy is used to represent rules for detecting environmental parameters of the surrounding geographical environment under the second scenario type; a network perception strategy associated with a third scenario type, wherein the third scenario type is used to represent a service scenario type associated with a network resource of the vehicle, and the network perception strategy is used to represent rules for obtaining resource information of network resources through the network under the third scenario type.

[0011] Optionally, according to the information perception strategy, the vehicle is controlled to detect multimodal information under the scene type, including: according to the information perception strategy, determining at least one detection device in the vehicle; triggering a control instruction to the detection device to control the detection device to detect multimodal information under the scene type.

[0012] Optionally, calling an information processing model to infer multimodal information to obtain a target task execution strategy for the vehicle, including: calling an information processing model to infer multimodal information and historical behavior data of the vehicle to obtain a target task execution strategy, wherein the historical behavior data is used to represent the behavior generated by the vehicle within a historical period.

[0013] Optionally, the information processing model is called to infer the multimodal information and the historical behavior data of the vehicle to obtain a target task execution strategy, including: calling the information processing model to infer the historical behavior data to obtain an initial task execution strategy; using the information processing model to adjust the initial task execution strategy according to the multimodal information to obtain the target task execution strategy.

[0014] Optionally, the information processing model is used to adjust the initial task execution strategy according to the multimodal information to obtain the target task execution strategy, including: using the information processing model to adjust the initial task execution strategy according to the environmental parameters of the vehicle's surrounding geographical environment in the multimodal information to obtain the target task execution strategy.

[0015] Optionally, calling the information processing model to infer the multimodal information to obtain the target task execution strategy of the vehicle includes: calling the information processing model to infer the resource information of the vehicle's network resources in the multimodal information to obtain the target task execution strategy.

[0016] Optionally, the information processing model is called to infer the multimodal information to obtain the target task execution strategy of the vehicle, including: calling the information processing model to identify the state information of the vehicle in the multimodal information to obtain the state identification result of the vehicle; determining the target task execution strategy corresponding to the state identification result, wherein the target task execution strategy is used to represent the rules for the vehicle to perform control task operations on the equipment associated with the state information.

[0017] Optionally, the method also includes: generating inquiry information that matches the conversation intention, wherein the inquiry information is used to inquire whether it is allowed to control the vehicle to perform the task operation; obtaining reply information corresponding to the inquiry information; in response to the reply information indicating that it is allowed to control the vehicle to perform the task operation, starting from the following steps: obtaining multimodal information associated with the conversation information.

[0018] Optionally, query information matching the dialogue intent is generated, including: using an information processing model to predict the behavior data of the predicted vehicle to obtain target behavior data matching the dialogue intent, wherein the target behavior data is used to enable the vehicle to perform a task operation; and using the information processing model to generate query information from the target behavior data.

[0019] According to another aspect of an embodiment of the present invention, there is also provided an information interaction method for a smart device, the method comprising: detecting dialogue information in a target scenario in which the smart device is located, wherein the dialogue content of the dialogue information is used to characterize the target scenario; acquiring multimodal information associated with the dialogue information, wherein the multimodal information is used to represent information in multiple modes that satisfies the dialogue intent of the dialogue content; calling an information processing model to infer the multimodal information to obtain a target task execution strategy for the smart device, wherein the information processing model is obtained by training an agent model using multimodal information samples in different scenarios, and the target task execution strategy is used to represent the rules for the task operation to be performed by the smart device; and controlling the smart device to perform the corresponding task operation according to the target task execution strategy.

[0020] According to another aspect of an embodiment of the present invention, there is also provided a vehicle information interaction system, comprising: a detection device, for detecting dialogue information in a target scenario in which a vehicle is located, wherein the dialogue content of the dialogue information is used to characterize the target scenario; a processor, for acquiring multimodal information associated with the dialogue information, wherein the multimodal information is used to represent information in multiple modes that satisfies the dialogue intent of the dialogue content; calling an information processing model to infer the multimodal information to obtain a target task execution strategy for the vehicle, wherein the information processing model is obtained by training an agent model using multimodal information samples in different scenarios, and the target task execution strategy is used to represent the rules for the vehicle's task operations to be performed; and a controller, for controlling the vehicle to perform corresponding task operations according to the target task execution strategy.

[0021] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present invention is executed when the program is running.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0024] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0025] According to another aspect of the embodiments of the present invention, a computer program is further provided. When the computer program is executed by a processor, the methods in the embodiments of the present invention are implemented.

[0026] According to another aspect of an embodiment of the present invention, a vehicle is provided, including a memory and a processor, wherein the memory stores an executable program, and the processor is used to run the program, wherein the method in each embodiment of the present invention is executed when the program is run.

[0027] In an embodiment of the present invention, if the dialogue information initiated by the user is monitored, the dialogue information in the target scene can be detected, and the multimodal information associated with the dialogue intention in the dialogue information can be obtained. The multimodal information can be input into the information processing model, and the information processing model can be used to infer the multimodal information to determine the target task execution strategy. According to the target task execution strategy, the vehicle is controlled to perform the corresponding task operation. In this embodiment, the target scene in which the current vehicle is located can be actively identified by combining the dialogue content with the multimodal information, so that the user's dialogue intention can be understood more accurately. The above-mentioned scene recognition capability enables the in-vehicle intelligent voice system to provide customized services according to the target environment, which improves the intelligence and personalization of the service. The information processing model can reason in different target scenarios, understand the deep meaning behind the multimodal information, and make intelligent decisions, so that it can not only respond to instructions, but also predict and initiate dialogues, and actively provide information or suggestions to users, which greatly enhances the intelligence and initiative of the in-vehicle intelligent voice system. To sum up, by realizing the free dialogue function, scene recognition service provision, multimodal perception method, and the autonomous thinking and logical reasoning ability of the information processing model, the purpose of providing users with a more natural, efficient and personalized interactive experience is achieved, thereby achieving the technical effect of improving the information interaction efficiency of the vehicle, and solving or partially solving the technical problem of low information interaction efficiency of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 is a flow chart of a vehicle information interaction method according to an embodiment of the present invention;

[0030] Figure 2 is a flow chart of an information interaction method of a smart device according to an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of an intelligent voice service system based on multi-modal perception and autonomous thinking according to an embodiment of the present invention;

[0032] Figure 4 is a system block diagram of a vehicle information interaction system according to an embodiment of the present invention;

[0033] Figure 5 is a structural block diagram of a vehicle information interaction device according to an embodiment of the present invention;

[0034] Figure 6 is a structural block diagram of an information interaction device for an intelligent device according to an embodiment of the present invention;

[0035] Figure 7 is a structural block diagram of an autonomous driving vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] According to an embodiment of the present invention, an embodiment of a vehicle information interaction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The embodiment of the present application provides a navigation control method for a vehicle. The method can be used to provide an augmented reality navigation function for a target vehicle in a preset application scenario to guide the target vehicle to travel. The above-mentioned preset application scenarios may include the following scenarios in the vehicle field: commuting automatic driving scenarios, artificial intelligence (AI) driving scenarios for family cars, automatic parking assistance (APA) scenarios (such as memory parking for self-owned parking spaces in garages, smart parking for designated parking spaces in parking lots, etc.), and intelligent navigation assistance (Navigation Guided Pilot, NGP) scenarios in urban areas or high-speed areas. In addition, the above-mentioned preset application scenarios may also include but are not limited to: automatic driving scenarios in which an augmented reality navigation function is required for smart driving trucks or unmanned trucks in the field of logistics and transportation, automatic driving scenarios in which an augmented reality navigation function is required for automatic driving of automatic driving agricultural vehicles in the field of agricultural machinery, automatic driving scenarios in which an augmented reality navigation function is required for drones, and automatic driving scenarios in which an augmented reality navigation function is required for intelligent robots (such as cleaning robots, service robots, delivery robots, etc.).

[0040] When the above-mentioned preset application scenarios are scenarios in fields other than the vehicle field, those skilled in the art should be able to understand that the vehicle in the above-mentioned vehicle information interaction method can be replaced with other objects (such as agricultural machinery, drones, robots, etc.), and accordingly, the use of augmented reality navigation function to guide the above-mentioned vehicle driving is replaced with navigation for other objects, and guiding other objects to move, fly or drive. On this basis, in the embodiments of the present application, the specific implementation method of the above-mentioned vehicle information interaction method is exemplified by taking the vehicle field as an example.

[0041] Figure 1 is a flow chart of a vehicle information interaction method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0042] Step S102, detecting the dialogue information in the target scene where the vehicle is located.

[0043] In the technical solution provided in the above step S102 of the present invention, the dialogue information can be the voice instructions or expressions of intentions transmitted between the user and the on-board intelligent system. It can be a direct command (for example, "navigate home") or an indirect expression (for example, "I'm so hot"). For the former, the on-board intelligent voice system can directly control the vehicle to perform the corresponding task operation. For the latter, the on-board intelligent voice system needs to understand the implicit needs behind the dialogue information (for example, adjust the air-conditioning temperature). The dialogue content of the dialogue information can be used to characterize the target scene. The target scene can be the specific environment and condition in which the vehicle is currently located. It can be a minimum viable product (MVP) scene such as commuting, talking, parking, traveling, abnormal weather, etc. Each scene has its specific background and user needs. The above scenes can be identified to provide users with services that are more in line with actual needs.

[0044] It should be noted that the above-mentioned dialogue information and target scenes are only for illustration and are not specifically limited here. As long as the target scenes and corresponding dialogue information are set according to the functions of the vehicle, they are within the protection scope of the embodiments of the present invention.

[0045] In this embodiment, if the dialogue information sent by the user is obtained, the dialogue information can be detected.

[0046] Optionally, the conversation information inside the vehicle can be monitored and captured, which may come from the driver, passengers, or even warning information from the vehicle itself. Speech recognition technology can be used to transcribe the conversation information, and the real intention and needs of the conversation content can be understood through Natural Language Processing (NLP) algorithms.

[0047] Optionally, through the recognition of the target scene and in-depth understanding of the dialogue information, it is possible to proactively predict the user's possible dialogue intentions before the user makes a request, and provide corresponding services or suggestions, greatly improving the naturalness and intelligence of human-computer interaction.

[0048] Step S104: acquiring multimodal information associated with the dialogue information.

[0049] In the technical solution provided in the above step S104 of the present invention, multimodal information can be used to represent information in multiple modes that satisfies the dialogue intention of the dialogue content. Multimodal information can come from information of different perceptual modes, including but not limited to voice, image, text, video, touch, position, environmental parameters, etc. Multimodal information generally refers to voice recognition results, ambient sound captured by a microphone, images or videos captured by a camera, vehicle status information provided by vehicle sensors, location information provided by a global positioning system (GPS), and external information such as weather and traffic conditions obtained through networking. The dialogue intention can be the potential demand or purpose expressed by the user through the dialogue information. For example, when the user says "I'm so hot", the dialogue intention can be "need to reduce the temperature in the car in some way". Through comprehensive analysis of the dialogue information and combined with multimodal information, the user's dialogue intention can be identified and understood.

[0050] It should be noted that the above-mentioned multimodal information is only for illustration and is not specifically limited here. As long as the information can help the in-vehicle intelligent voice system understand the target scene in which the vehicle is currently located more accurately and in detail, it is within the protection scope of the embodiments of the present invention.

[0051] In this embodiment, after detecting the dialogue information in the target scene where the vehicle is located, multimodal information associated with the dialogue information may be acquired.

[0052] Optionally, based on the detected dialogue information, multimodal information associated with the dialogue information is started to be obtained. The above method means that the in-vehicle intelligent voice system (intelligent voice service system) not only relies on dialogue information (voice instructions), but can also more deeply understand the user's needs and the current environment of the target scene by comprehensively analyzing multimodal information from multiple information sources.

[0053] For example, when a user says "I'm so hot", the in-vehicle intelligent voice system can not only understand the literal meaning of the voice command, but also needs to obtain the following multimodal information: in-vehicle temperature sensor data to confirm the actual temperature in the car; GPS location data to understand whether the vehicle is in a hot area or under direct sunlight; window and sunshade status information to determine whether it is necessary to adjust the window opening or sunshade status to block the sunlight; air conditioning system status to understand whether the air conditioning is turned on, as well as the current working mode and temperature setting; user historical preference data to understand whether the user prefers to use natural ventilation rather than air conditioning; weather forecast information to determine whether the weather changes in the future and whether it is necessary to adjust the vehicle settings in advance. It should be noted that the dialogue information expressed by the above users and the corresponding multimodal information obtained are only for example and are not specifically limited here.

[0054] Optionally, the acquired multimodal information associated with the current target scene's dialogue information is integrated for in-depth analysis and reasoning to identify the full picture of the user's dialogue intention. The purpose of acquiring multimodal information is to build a comprehensive environment and user demand model, through which the user's real needs can be more accurately judged, such as whether the air conditioner needs to be turned on, the temperature adjusted to a suitable level, whether the car windows need to be closed, etc., so as to make a more intelligent and personalized response.

[0055] In the embodiment of the present invention, through the above method, the in-vehicle intelligent voice system in the vehicle realizes a multi-dimensional understanding of the user's intention and environmental conditions, which not only improves the scene adaptability and user experience of the in-vehicle intelligent voice system, but also provides a solid information foundation for subsequent reasoning planning and task execution. It can ensure that the intelligent voice system can still accurately capture and respond to the user's real needs in complex and changing scenes, reflecting a high degree of intelligence and autonomy.

[0056] Step S106, calling the information processing model to infer the multimodal information to obtain the vehicle's target task execution strategy.

[0057] In the technical solution provided in the above step S106 of the present invention, the information processing model can be obtained by training the agent model using multimodal information samples in different scenarios. The information processing model refers to an algorithm framework used in the vehicle-mounted intelligent voice system for analyzing, understanding and reasoning multimodal information. In the vehicle-mounted intelligent voice system, it can be an artificial intelligence agent (AI Agent) model, which is based on deep learning (DL), natural language processing, machine learning (ML) and other technologies, and can understand user intentions, analyze environmental conditions, and perform logical reasoning and decision-making based on historical data and preset rules. Through training and learning, the information processing model can extract features from a large number of multimodal information samples, establish a model of user behavior and environmental changes, and thus realize intelligent reasoning and decision-making. The target task execution strategy can be used to represent the rules for the vehicle to perform the task operation, which can also be called a decision or user suggestion. The target task execution strategy can be the inference result obtained by the in-vehicle intelligent voice system based on the information processing model to infer multimodal information, and the action rules or suggestions that meet user needs and current scenarios. The target task execution strategy can include adjusting vehicle settings (for example, air conditioning temperature, seat position), performing specific tasks (for example, navigating to a car wash), providing information feedback (for example, weather forecast, traffic conditions), etc.

[0058] It should be noted that the above-mentioned target task execution strategy is only for illustration and is not specifically limited here. As long as the target task execution strategy can be formulated by considering multiple aspects such as user preferences, environmental conditions, task safety and feasibility to ensure that the action rules or suggestions for providing appropriate responses and services to users are within the protection scope of the embodiments of the present invention.

[0059] In this embodiment, after obtaining the multimodal information associated with the dialogue information, the information processing model can be called to reason about the multimodal information to obtain the target task execution decision of the vehicle.

[0060] Optionally, a pre-trained information processing model (e.g., AI Agent model) is called to perform in-depth analysis and reasoning on the acquired multimodal information. The information processing model can comprehensively consider user dialogue information, vehicle status, environmental parameters, historical behavior data and other information to identify user potential needs and predict possible scenario changes.

[0061] For example, if the in-vehicle intelligent voice system detects that the user says "I'm so hot", and the vehicle is currently located in a high-temperature area, the windows are open, the sunshades are not closed, and other multimodal information, the AI ​​Agent model will perform reasoning analysis to understand that the user's real need is to lower the temperature in the car, and take into account the current status of the equipment in the car and external environmental factors, and generate target task execution strategies such as "turn on the air conditioner and set it to the user's preferred temperature, close the windows and sunshades".

[0062] In an embodiment of the present invention, the above method realizes intelligent reasoning of user needs and deep understanding of environmental status, and can dynamically generate the execution strategy that best suits the current scenario and user preferences. This not only improves the user's personalized service experience, but also enhances the system's autonomous decision-making ability, so that the in-vehicle intelligent voice system can more flexibly handle complex driving environments and changes in user needs. For example, when the in-vehicle intelligent voice system recognizes that the user may be about to travel, but the vehicle's battery is low, it can actively suggest the user to go to the nearest charging station and plan a suitable driving route to ensure a smooth journey. Through the above-mentioned intelligent decision-making and task execution capabilities, the intelligence level of the in-vehicle intelligent voice system is improved, the complexity of user operations is reduced, and a more natural, efficient and safe interaction method is provided for users. In addition, through continuous learning and optimization, the AI ​​Agent model can gradually improve its reasoning accuracy, provide more accurate and personalized services, help to establish user trust and dependence on the system, and improve user satisfaction.

[0063] Step S108, controlling the vehicle to execute the corresponding task operation according to the target task execution strategy.

[0064] In the technical solution provided in the above step S108 of the present invention, the task operation can be a specific action or adjustment that the vehicle needs to perform based on the target task execution strategy formulated by the intelligent voice service of the in-vehicle intelligent voice system. The above task operation can be the actual measures taken by the in-vehicle intelligent voice system to meet user needs or improve the current scenario after responding to user needs or predicting user intentions. For example, the types of task operations can include vehicle-mounted device control, vehicle system adjustment, vehicle external service call, and user information interaction.

[0065] For example, the task operations of the vehicle equipment control type may include adjusting the air conditioning temperature and wind speed, opening and closing of windows, seat heating or ventilation, opening and closing of sunshades, volume and playback content of the audio system, and operating the navigation system to set the route or destination. The task operations of the vehicle system adjustment type may include adjusting the vehicle driving mode (for example, sports, economy mode), controlling the vehicle to automatically park or adjust the parking position, adjusting the interior and exterior lighting of the vehicle (for example, turning on the headlights, ambient lights), and adjusting the vehicle safety settings (for example, starting or disabling the assisted driving function). The task operations of the vehicle external service call type may include finding the nearest gas station or charging station and navigating to it, booking a car wash service or maintenance service, and querying and reporting real-time information such as weather and traffic conditions. The task operations of the user information interaction type may include responding to user inquiries or providing relevant information, actively initiating conversations to discuss topics of interest to users, and providing users with upcoming events (for example, meetings, appointments).

[0066] It should be noted that the above-mentioned different types of task operations and the specific contents of the executed task operations are only for illustration and are not specifically limited here, and may correspond to the functions possessed by the vehicle.

[0067] In this embodiment, after obtaining the target task execution strategy, a specific vehicle control instruction can be generated, and the above vehicle control instruction can be sent to the corresponding vehicle control unit through the vehicle network or wireless communication technology, thereby realizing the automated operation of the vehicle.

[0068] For example, when the in-vehicle intelligent voice system infers that the user needs to cool down and generates a strategy of "turn on the air conditioner and set the temperature to 22 degrees, close the windows, and close the sunshade", it will send instructions to the air conditioning system, window control unit, and sunshade control unit to perform the cooling operation. In the above process, the in-vehicle intelligent voice system can also monitor the progress of the task operation and dynamically adjust the strategy according to the actual situation to ensure the realization of the final goal.

[0069] Optionally, the target task execution strategy is converted into an executable control instruction, that is, the abstract description of the target task execution strategy can be converted into a specific control instruction (for example, a vehicle control code or signal). Through the vehicle network or wireless communication technology, a connection is established with each control unit of the vehicle to send control instructions. This may involve technical challenges such as multi-protocol compatibility and secure data transmission. During the execution of the task operation, the vehicle status and environmental changes can be continuously monitored to ensure that the progress of the task operation is in line with expectations. If obstacles or changes are encountered, the strategy can be adjusted quickly to ensure the smooth execution of the task. Once the task operation is completed, the execution result can be confirmed and the operation status can be reported to the user, for example, through voice broadcast "The air conditioner is turned on, the temperature is set to 22 degrees, the windows and sunshades are closed, and the cooling operation is completed."

[0070] In the embodiment of the present invention. The in-vehicle intelligent voice system can convert decisions into actual operations, realize intelligent control of the vehicle and instant response to user needs. The above process not only improves the practicality and user experience of the in-vehicle intelligent voice system, but also reflects the value of the in-vehicle intelligent voice system in the vehicle environment, that is, it can provide users with safe, convenient and personalized services, reduce the user's operating burden, and improve driving safety and comfort. Through the closed-loop control mechanism, the in-vehicle intelligent voice system can ensure the accuracy and effectiveness of task operations, and further enhance the autonomous service capabilities of the in-vehicle intelligent voice system.

[0071] In the above steps S102 to S108 of the present invention, if the dialogue information initiated by the user is monitored, the dialogue information under the target scene can be detected, and the multimodal information associated with the dialogue intention in the dialogue information can be obtained. The multimodal information can be input into the information processing model, and the information processing model can be used to infer the multimodal information to determine the target task execution strategy. According to the target task execution strategy, the vehicle is controlled to perform the corresponding task operation. In this embodiment, the target scene in which the current vehicle is located can be actively identified by combining the dialogue content with the multimodal information, so that the user's dialogue intention can be understood more accurately. The above scene recognition capability enables the in-vehicle intelligent voice system to provide customized services according to the target environment, which improves the intelligence and personalization of the service. The information processing model can perform reasoning in different target scenarios, understand the deep meaning behind the multimodal information, and make intelligent decisions, so that it can not only respond to instructions, but also predict and initiate dialogues, and actively provide information or suggestions to users, which greatly enhances the intelligence and initiative of the in-vehicle intelligent voice system. To sum up, by realizing the free dialogue function, scene recognition service provision, multimodal perception method, and the autonomous thinking and logical reasoning ability of the information processing model, the purpose of providing users with a more natural, efficient and personalized interactive experience is achieved, thereby achieving the technical effect of improving the information interaction efficiency of the vehicle, and solving or partially solving the technical problem of low information interaction efficiency of the vehicle.

[0072] The following further describes the process of how to obtain multimodal information associated with the dialogue information in this embodiment.

[0073] As an optional implementation, step S104, obtaining multimodal information associated with the dialogue information, includes: performing scene recognition on the dialogue information to obtain the scene type of the target scene; determining an information perception strategy associated with the scene type, wherein the information perception strategy is used to represent the rules for detecting multimodal information under the scene type; and controlling the vehicle to detect multimodal information under the scene type according to the information perception strategy.

[0074] In this embodiment, in the process of acquiring multimodal information associated with the dialogue information, scene recognition can be performed on the dialogue information to obtain the scene type of the target scene. An information perception strategy associated with the scene type can be determined. According to the information perception strategy, the vehicle can be controlled to detect multimodal information under the scene type. Among them, the information perception strategy can be used to represent the rules for detecting multimodal information under the scene type, which can also be called a perception method.

[0075] Optionally, scene recognition is a process of identifying the type of scene the user is currently in or describing based on user conversation information through natural language processing (NLP) technology. Scene types may include but are not limited to: driving, parking, commuting, traveling, charging, talking, abnormal weather, etc. The in-vehicle intelligent voice system needs to understand the context of the conversation content, identify keywords or semantics, and thus determine the specific scene the user is in. For example, when the user says "I'm so hot", the in-vehicle intelligent voice system may identify the driving scene and further determine that the user may need to lower the temperature in the car. Information perception strategy refers to the rules or methods formulated for detecting and collecting multimodal information based on the results of scene recognition in a specific scene. The formulation of the above-mentioned information perception strategy can take into account multiple factors such as scene characteristics, user needs, vehicle status and environmental conditions. For example, in a driving scenario, the in-vehicle intelligent voice system may need to pay attention to information such as the temperature inside and outside the vehicle, the air conditioning status, and the position of the sunshade; in a parking scenario, the in-vehicle intelligent voice system may pay more attention to the vehicle location and the safety of the surrounding environment.

[0076] Optionally, according to the determined information perception strategy, the in-vehicle intelligent voice system will control the sensors, cameras, microphones and other devices on the vehicle, as well as the vehicle system itself (for example, air conditioning, window control unit) to detect and collect multimodal information related to the current scene type. This process may include: calling on-board sensors to detect environmental parameters inside and outside the vehicle, such as temperature, humidity, light intensity, etc. Starting the in-vehicle camera to capture the status of passengers in the car or changes in the environment outside the car. Listen to the microphone input to obtain the user's subsequent conversation information. Control the vehicle system, such as starting the air conditioner, checking the battery level, checking the seat status, etc., to obtain information about the internal status of the vehicle.

[0077] Optionally, through scene recognition technology, the scene context in the user's conversation information can be understood. Subsequently, according to the scene type, an information perception strategy is intelligently formulated, which can guide the vehicle to selectively detect and collect multimodal information in a specific scene, avoiding the blindness and redundancy of information collection and improving the efficiency and accuracy of information collection.

[0078] Optionally, the step of controlling the vehicle to detect multimodal information is the process of converting the information perception strategy into actual operation, which requires the vehicle intelligent voice system to have good integration and control capabilities with vehicle equipment. Through the above process, the vehicle intelligent voice system can obtain comprehensive information on user needs and environmental status, providing a solid data foundation for subsequent reasoning, judgment and decision-making.

[0079] In an embodiment of the present invention, the above method significantly enhances the scene adaptability and intelligent response capability of the in-vehicle intelligent voice system, so that the in-vehicle intelligent voice system can more accurately understand and respond to user needs and provide more personalized and intelligent services. For example, when a user expresses "I'm so hot", the in-vehicle intelligent voice system can not only understand that the user may need to cool down, but also automatically detect information such as the temperature in the car and the status of the air conditioner through perception strategies, thereby intelligently adjusting the air conditioning settings to improve the user's comfort. In parking scenarios, the in-vehicle intelligent voice system can detect the safety conditions around the vehicle, provide corresponding safety tips or service suggestions, and enhance the user's sense of security. In the above manner, the intelligent voice service system can achieve more efficient, smarter, and more humane interactions and services.

[0080] The following further describes the process of how to determine the information perception strategy associated with the scene type in this embodiment.

[0081] As an optional implementation, determining the information perception strategy associated with the scene type includes: searching for the information perception strategy in the information perception strategy set by using the scene type, wherein the information perception strategy set includes information perception strategies associated with different scene types respectively. The information perception strategy set includes a speech perception strategy and / or an image perception strategy, wherein the speech perception strategy is used to represent a rule for detecting speech information for a vehicle under a scene type, and the image perception strategy is used to represent a rule for detecting image information for a vehicle under a scene type.

[0082] In this embodiment, in the process of determining the information perception strategy associated with the scene type, the corresponding information perception strategy can be found from the information perception strategy set using the scene type. The information perception strategy set includes information perception strategies respectively associated with different scene types. The information perception strategy set may include voice perception strategies and / or image perception strategies. The voice perception strategy can be used to represent the rules for detecting voice information for a vehicle under a scene type. The image perception strategy can be used to represent the rules for detecting image information for a vehicle under a scene type.

[0083] Optionally, the information perception strategy set is a database containing multiple scenario types and their corresponding perception strategies. The information perception strategy here refers to what types of multimodal information the in-vehicle intelligent voice system should collect in a specific scenario and how to collect this information. The scenario types that may be included in the strategy set include driving, parking, commuting, traveling, charging, calling, abnormal weather, etc. Each scenario type has its own specific information perception strategy.

[0084] For example, in a driving scenario, the information perception strategy may include monitoring the temperature inside and outside the car, detecting the status of passengers in the car, obtaining road conditions and traffic information, etc. In an abnormal weather scenario, the information perception strategy may focus more on obtaining weather forecasts, checking the vehicle's waterproof and anti-skid settings, adjusting the vehicle speed and driving mode, etc.

[0085] Optionally, in the process of building the information perception strategy set, the characteristics of various scenarios can be fully considered, as well as the specific information types and methods that the in-vehicle intelligent voice system needs to collect in order to make intelligent decisions in these scenarios. The construction of the strategy set can be based on expert knowledge, historical data, and machine learning models to ensure the comprehensiveness and accuracy of the strategy.

[0086] Optionally, after the in-vehicle intelligent voice system determines the type of scene the user is currently in through scene recognition technology, it will search for an information perception strategy that matches the scene type in the information perception strategy set. The above process can be implemented through database query or index matching technology to quickly locate the correct strategy. During the search process, the in-vehicle intelligent voice system will consider multiple dimensions of the scene type, such as time, location, weather, user behavior, etc., to ensure that the perception strategy found can fully reflect the characteristics of the current scene, thereby improving the pertinence and efficiency of information collection.

[0087] For example, if the in-vehicle intelligent voice system recognizes that the user is in a driving scenario, it will search for all information perception strategies associated with the driving scenario from the strategy set, including monitoring the temperature inside and outside the car, detecting road conditions, listening to user conversations, etc. The above information strategy collection will guide the system on how to collect and process information in the driving scenario.

[0088] Optionally, once information perception strategies that match the current scene type are found, the in-vehicle intelligent voice system will execute these strategies to control in-vehicle sensors, cameras, microphones and other devices, as well as the vehicle system itself, to collect multimodal information related to the current scene. This process may include starting specific sensors, adjusting the detection parameters of the device, and setting the priority of information collection. In order to ensure the real-time and accuracy of information, the in-vehicle intelligent voice system can perform multi-threading or asynchronous processing to quickly collect the required information without affecting the user's interactive experience. In addition, the in-vehicle intelligent voice system may also dynamically adjust the perception strategy according to environmental changes to adapt to the changing scene requirements.

[0089] In an embodiment of the present invention, by constructing an information perception strategy set and searching and executing corresponding perception strategies according to the scene type, the in-vehicle intelligent voice system can realize intelligent and efficient collection of multimodal information, ensuring that the collected information is highly relevant to user needs and the current scene. This process not only improves the pertinence of information collection and reduces unnecessary information processing burden, but also enhances the scene adaptability and intelligent response capabilities of the in-vehicle intelligent voice system. For example, in a driving scene, the in-vehicle intelligent voice system can quickly start strategies such as temperature monitoring inside and outside the car, road condition detection, and user dialogue monitoring to fully understand the driving environment and user needs, so as to make more accurate intelligent decisions, such as adjusting air conditioning settings, providing navigation suggestions, and actively initiating dialogues. In abnormal weather scenarios, the in-vehicle intelligent voice system can automatically detect weather changes, monitor the vehicle's waterproof and anti-skid settings, provide safe driving suggestions, and enhance the safety protection function of the in-vehicle intelligent voice system. In the above manner, the intelligent voice service can provide more personalized and intelligent services, significantly improve the user experience, and also demonstrate the flexibility and intelligence level of intelligent voice services in multiple scenarios.

[0090] Optionally, in order to more accurately control and utilize the vehicle-mounted equipment for multimodal information collection, the information perception strategy set can be subdivided into voice perception strategy and image perception strategy.

[0091] Optionally, the speech perception strategy is a rule and method for voice information collection, which guides the system on how to capture, parse and understand the user's voice commands or conversation content in different scenarios. The speech perception strategy may include the following points: microphone activation and positioning, which determines which microphones need to be activated according to the scene type, and how the microphones are positioned to optimize the sound collection quality. For example, in a noisy environment, it may be necessary to activate all microphones and use noise suppression algorithms, while in a quiet environment, only the microphone at the driver's position needs to be activated; speech recognition parameter adjustment, which adjusts the parameters of the speech recognition engine to adapt to the speech characteristics in different scenarios. For example, in a call scenario, it may be necessary to improve the recognition speed and accuracy, while in a travel scenario, it may be necessary to increase the filtering ability of background music and noise; speech content analysis, which defines how to parse voice information, identify keywords, and understand user intent. For example, in a driving scenario, the in-vehicle intelligent voice system may pay special attention to words related to vehicle control, such as "air conditioning", "windows", etc., while in a parking scenario, it may pay more attention to the user's description and feedback of the surrounding environment.

[0092] Optionally, image perception strategy refers to how the system uses the vehicle-mounted camera to collect image information in a specific scenario, and how to process and analyze these images. Image perception strategy may include: camera activation and adjustment, determining which cameras need to be activated according to the scene type, and adjusting the angle and resolution of the camera to ensure that the captured information is the most relevant. For example, in a driving scenario, it may be necessary to focus on monitoring the road ahead, while in a parking scenario, it may be more concerned with the safety conditions around the vehicle. Image recognition algorithm selection, in different scenarios, select the most appropriate image recognition algorithm to improve the accuracy and speed of recognition. For example, in an abnormal weather scenario, it may be necessary to use a special algorithm to identify road conditions in rainy and snowy weather, while in a travel scenario, it may focus more on identifying landmark buildings or scenery. Image content analysis, defining how to analyze image information, identify scene features, and understand environmental changes. For example, in a driving scenario, the in-vehicle intelligent voice system can analyze the congestion level of the road ahead, while in a parking scenario, it may evaluate the availability of surrounding parking spaces.

[0093] Optionally, once the voice perception strategy and image perception strategy that match the current scene type are determined, the system will execute these strategies to collect and process the corresponding voice and image information. This may include: controlling microphones and cameras. The in-vehicle intelligent voice system can start specific microphones and cameras according to the strategy and adjust their parameters to ensure the accuracy and real-time nature of information collection. Parsing and processing information. The in-vehicle intelligent voice system can use voice recognition and image recognition technology to parse and process the collected information and extract key features such as user needs, environmental changes, etc. Information fusion and decision making. The in-vehicle intelligent voice system can fuse the processed voice and image information and combine it with other multimodal information to make decisions or generate user suggestions. For example, if the in-vehicle intelligent voice system detects congestion on the road ahead in a driving scenario and the user says "I want to get to my destination as soon as possible", it may suggest that the user adjust the route or provide traffic status updates.

[0094] In the embodiment of the present invention. By subdividing into voice perception strategy and image perception strategy, the in-vehicle intelligent voice system can control the information collection process more accurately to ensure that the collected information is highly relevant to the current scene and user needs. This not only improves the efficiency and accuracy of information collection, but also enhances the scene adaptability of the in-vehicle intelligent voice system, and can provide more personalized and intelligent services for different scenes. For example, in the driving scene, through the voice perception strategy, the in-vehicle intelligent voice system can quickly respond to the user's instructions on vehicle control, such as adjusting the air conditioner, navigation settings, etc.; through the image perception strategy, the in-vehicle intelligent voice system can actively monitor road conditions, predict potential driving risks, and provide safe driving advice. In the parking scene, the voice perception strategy can help the in-vehicle intelligent voice system understand the user's intention to find a parking space, and the image perception strategy can identify the surrounding environment and evaluate the availability and safety of the parking space. Through the above method, the in-vehicle intelligent voice system can provide users with a more comprehensive, safe and convenient interactive and service experience, demonstrating the high flexibility and intelligence level of the in-vehicle intelligent voice system in multiple scenarios.

[0095] The information-aware policy set in this embodiment is further explained below.

[0096] As an optional implementation, the information perception strategy set includes at least one of the following: a state perception strategy associated with a first scenario type, wherein the first scenario type is used to represent a service scenario type associated with a vehicle state of a vehicle, and the state perception strategy is used to represent rules for detecting state information of the vehicle state under the first scenario type; an environment perception strategy associated with a second scenario type, wherein the second scenario type is used to represent a service scenario type associated with a surrounding geographical environment of the vehicle, and the environment perception strategy is used to represent rules for detecting environmental parameters of the surrounding geographical environment under the second scenario type; a network perception strategy associated with a third scenario type, wherein the third scenario type is used to represent a service scenario type associated with a network resource of the vehicle, and the network perception strategy is used to represent rules for obtaining resource information of network resources through the network under the third scenario type.

[0097] In this embodiment, the scene type may include a first scene type, a second scene type, and a third scene type. Among them, the first scene type can be used to represent a service scene type associated with the vehicle state of the vehicle. The second scene type can be used to represent a service scene type associated with the surrounding geographical environment of the vehicle. The third scene type can be used to represent a service scene type associated with the network resources of the vehicle. The information perception strategy set may include a state perception strategy associated with the first scene type, an environment perception strategy associated with the second scene type, and a network perception strategy associated with the third scene type. The state perception strategy can be used to represent the rules for detecting the state information of the vehicle state under the first scene type. The state information may include the vehicle control state, the system state, the device power, and the seat state. The environment perception strategy can be used to represent the rules for detecting the environmental parameters of the surrounding geographical environment under the second scene type. The network perception strategy can be used to represent the rules for the resource information of the network resources obtained through the network under the third scene type. Resource information can also be called networking information, such as weather forecast, traffic conditions, etc.

[0098] Optionally, the information perception strategy set is subdivided into state perception strategy, environment perception strategy and network perception strategy, each strategy targets different types of scenarios and service requirements, guiding the in-vehicle intelligent voice system on how to most effectively collect and utilize specific multimodal information.

[0099] Optionally, the state perception strategy focuses on the internal state of the vehicle, including vehicle control state (e.g., engine, air conditioning, windows, etc.), system state (e.g., battery power, fuel level, etc.), and device state (e.g., seats, audio system, etc.). In service scenarios associated with vehicle state (e.g., driving, charging, device control, etc.), the state perception strategy will guide the in-vehicle intelligent voice system on how to detect this state information in order to make intelligent decisions.

[0100] For example, in a driving scenario, the state perception strategy may include monitoring engine temperature, fuel level, air conditioning status, window opening, etc. to ensure driving safety and comfort. In a charging scenario, the strategy may focus on monitoring battery power, charging status, availability of charging facilities, etc. to optimize the charging plan.

[0101] Optionally, the environmental perception strategy focuses on the geographical environment around the vehicle, including road conditions, weather, obstacles, landmarks, etc. In service scenarios associated with the surrounding geographical environment (e.g., navigation, travel, abnormal weather, etc.), the environmental perception strategy will guide the in-vehicle intelligent voice system on how to collect and analyze these environmental parameters to provide more accurate services.

[0102] For example, in a navigation scenario, the environment perception strategy may include detecting road congestion, obstacles ahead, traffic lights, etc. to provide real-time route suggestions. In a travel scenario, the strategy may focus more on identifying surrounding scenic spots, catering, entertainment and other service facilities to enrich the user's travel experience.

[0103] Optionally, the network awareness strategy focuses on information resources obtained through the Internet, such as weather forecasts, traffic conditions, news information, music playlists, etc. In service scenarios associated with network resources (such as information query, entertainment services, emergency response, etc.), the network awareness strategy will guide the in-vehicle intelligent voice system on how to effectively use network resources to meet user needs.

[0104] For example, in an abnormal weather scenario, the network perception strategy may include querying weather forecasts, road slipperiness and other information to remind users to pay attention to safety. In the entertainment service scenario, the strategy may focus on searching for users' favorite music, audiobooks and other content to provide personalized entertainment experience.

[0105] Optionally, once the current scene type is determined, the intelligent voice service system will search for a state perception strategy, environment perception strategy, or network perception strategy that matches the scene type from the information perception strategy set. Subsequently, the intelligent voice service system will call the corresponding vehicle-mounted devices and network resources according to the above strategies to collect and process the required multimodal information.

[0106] For example, when the intelligent voice service system recognizes that the user is in a driving scenario, it will look for the state perception strategy and environment perception strategy associated with the first scenario type (driving scenario), activate the on-board sensors to monitor engine status, temperature inside and outside the vehicle, road conditions and other information, and query traffic conditions through network perception strategies to make intelligent driving decisions.

[0107] In an embodiment of the present invention, by subdividing the information perception strategy set into state perception strategy, environment perception strategy and network perception strategy, the system can collect and process information more specifically, which not only ensures the efficiency of information collection, but also enhances the intelligence and personalization of services, so that the intelligent voice service system can not only respond to the basic needs of users, such as vehicle control and navigation services, but also provide a richer, safer and more convenient driving experience according to environmental changes and network resources.

[0108] The following further describes how to control the vehicle to detect multimodal information under the scene type according to the information perception strategy in this embodiment.

[0109] As an optional implementation, according to the information perception strategy, the vehicle is controlled to detect multimodal information under the scene type, including: according to the information perception strategy, determining at least one detection device in the vehicle; triggering a control instruction to the detection device to control the detection device to detect multimodal information under the scene type.

[0110] In this embodiment, in the process of controlling the vehicle to detect multimodal information under a scene type according to the information perception strategy, at least one detection device in the vehicle can be determined according to the information perception strategy, so that corresponding control instructions can be triggered to the detection device to control the detection device to detect the corresponding multimodal information under the scene type.

[0111] Optionally, the intelligent voice service system dynamically controls the vehicle-mounted detection equipment through information perception strategies to adapt to the multimodal information collection needs in specific scenarios.

[0112] Optionally, based on the rules defined in the information perception strategy, the intelligent voice service system can determine which detection devices in the vehicle should be activated to collect the information required for the current scenario type. The above method requires the intelligent voice service system to have a comprehensive understanding of the vehicle-mounted devices, including device type, function, location, and relevance to specific scenarios.

[0113] For example, in a driving scenario, the intelligent voice service system can activate detection equipment such as microphones (for voice recognition), cameras (for monitoring road conditions), and temperature sensors (for monitoring the temperature inside the car). In a parking scenario, the intelligent voice service system may focus more on activating cameras (for monitoring the safety of the surrounding environment) and obstacle detection sensors (for automatic parking).

[0114] Optionally, the process of determining the detection device is based on a comprehensive assessment of the scenario requirements and device capabilities. The intelligent voice service system must not only ensure that the selected device can collect the required information, but also avoid waste of resources and ensure the rational use of the device.

[0115] Optionally, after the intelligent voice service system determines which detection devices need to be activated, it can then send control instructions to the corresponding detection devices, instructing them to collect information according to the requirements of the information perception strategy. The above process involves precise control of the vehicle-mounted devices, including device startup, parameter adjustment, data transmission, etc.

[0116] For example, if the information perception strategy indicates that the temperature inside and outside the car needs to be monitored in the driving scenario, the intelligent voice service system will send a control instruction to the temperature sensor, requiring it to start collecting temperature data at a certain frequency. If the strategy includes image recognition, the intelligent voice service system will send a control instruction to the camera, instructing it to adjust the focus and resolution to obtain clear image information. The sending of control instructions is the key to ensuring the accuracy of the information collection process. The instructions must include detailed information such as the specific parameters for device startup, the frequency of information collection, and the format of data transmission to ensure the timeliness and completeness of the information.

[0117] In an embodiment of the present invention, through the above method, the intelligent voice service system can flexibly control the vehicle-mounted detection equipment to achieve efficient and accurate information collection in multiple scenarios. The above process not only improves the pertinence of information collection and reduces unnecessary resource consumption, but also enhances the scene adaptability and intelligence level of the intelligent voice service system, and can provide users with a more personalized and intelligent service experience. For example, in a driving scenario, by accurately controlling sensors and cameras, the intelligent voice service system can monitor road conditions and vehicle status in real time, and provide safe driving suggestions, such as adjusting vehicle speed, avoiding obstacles, etc. In a parking scenario, the intelligent voice service system can help users quickly find a safe parking space and realize intelligent parking by activating cameras and obstacle detection sensors. In addition, this process also reflects the intelligence of the intelligent voice service system in device control, and can dynamically adjust the working mode of the device according to different scene requirements, thereby improving the overall efficiency and user experience of the intelligent voice service system. Through intelligent control and multimodal information collection, the system can more deeply understand user needs and environmental changes, and provide users with more comprehensive and accurate services.

[0118] The following further describes the process of how to call the information processing model to infer multimodal information in this embodiment.

[0119] As an optional implementation, an information processing model is called to infer multimodal information to obtain a target task execution strategy for the vehicle, including: calling an information processing model to infer historical behavior data to obtain a target task execution strategy, wherein the historical behavior data is used to represent the behavior of the vehicle within a historical period; using the information processing model to adjust the initial task execution strategy according to the environmental parameters of the vehicle's surrounding geographic environment in the multimodal information to obtain the target task execution strategy.

[0120] In this embodiment, in the process of inferring the multimodal information by calling the information processing model to obtain the execution strategy of the target task of the vehicle, the information processing model can be called to infer the multimodal information and the historical behavior data of the vehicle to obtain the target task execution strategy. In the process of adjusting the initial task execution strategy according to the multimodal information by using the information processing model to obtain the target task execution strategy, the information processing model can be used to adjust the initial task execution strategy according to the environmental parameters of the surrounding geographical environment of the vehicle in the multimodal information to obtain the target task execution strategy. Among them, the historical behavior data can be used to represent the behavior generated by the vehicle in the historical period, and can also be called user historical data.

[0121] Optionally, the intelligent voice service system calls the information processing model to perform deep reasoning on the collected multimodal information and historical behavior data of the vehicle (or user) to generate a target task execution strategy that best suits the current scenario and user needs.

[0122] Optionally, the information processing model is the core of the intelligent voice service system. It is based on technologies such as NLP, ML and DL, and can understand and reason about multimodal information. The intelligent voice service system will call this model and input the collected multimodal information and historical behavior data to perform intelligent reasoning in complex scenarios. The information processing model may include multiple sub-models, such as speech recognition model, image recognition model, knowledge graph model, behavior prediction model, etc., each of which is responsible for processing a specific type of information. For example, the speech recognition model is responsible for converting the user's voice commands into text form, the image recognition model is responsible for analyzing the image content captured by the camera, and the knowledge graph model is used to process and understand information in network resources.

[0123] Optionally, when calling the information processing model, the intelligent voice service system fuses the collected multimodal information with the historical behavior data of the vehicle (or user) as the model input. The historical behavior data may include information such as the user's preferences, habits, common routes and frequented places of the vehicle, as well as various states and operation records of the vehicle in the historical period. The fusion process requires the intelligent voice service system to pre-process the information, including format unification, feature extraction, noise filtering, etc., to ensure that the information can be effectively used by the information processing model. At the same time, the intelligent voice service system can also dynamically update and maintain the historical behavior data to reflect the user's latest needs and environmental changes. For example, if the user repeatedly expresses sensitivity to temperature in the driving scenario, the intelligent voice service system should fuse this historical behavior data with the current temperature perception information to generate a more personalized temperature adjustment strategy.

[0124] Optionally, the information processing model will perform deep reasoning based on the fused multimodal information and historical behavior data to generate a suitable target task execution strategy for the current scenario. This strategy may include vehicle control parameters (e.g., air conditioning temperature, seat position, etc.), service suggestions (e.g., recommended routes, destination suggestions, etc.), and potential proactive services (e.g., restaurant reservations, refueling reminders, etc.).

[0125] Optionally, the reasoning process can involve multiple levels of models and algorithms, from simple rule matching to complex deep learning models to process various features extracted from multimodal information, as well as user preferences and behavior predictions based on historical behavior data. For example, an intelligent voice service system may use a deep neural network model to analyze the user's current voice commands and image information, while using a behavior prediction model to determine the user's possible subsequent behavior to generate the most appropriate execution strategy.

[0126] In an embodiment of the present invention, by calling an information processing model to infer multimodal information and historical behavior data, the intelligent voice service system can generate a highly personalized and intelligent target task execution strategy, significantly improving the accuracy of the service and user experience. The above process not only reflects the intelligence of the system in information processing and decision-making, but also emphasizes the important role of historical behavior data in understanding user needs and predicting user behavior. For example, in a driving scenario, the intelligent voice service system can generate driving route suggestions based on current weather conditions, traffic information, and the user's historical route preferences. In a parking scenario, the intelligent voice service system can automatically search and recommend the most suitable parking space based on the vehicle's historical parking preferences and current environmental information. In a travel scenario, the intelligent voice service system can analyze the user's historical travel locations and activity preferences, and actively provide personalized travel suggestions or reservation services. The above method realizes the flexibility and adaptability of the intelligent voice service system in multiple scenarios, and can provide all-round, all-time personalized services through deep reasoning and intelligent decision-making to meet the needs of users in different scenarios.

[0127] Optionally, by calling the information processing model, the user's historical data and real-time data are used for inference and judgment to learn the user's preferences and habits, thereby improving the accuracy of determining user needs.

[0128] Optionally, the intelligent voice service system can collect and integrate the user's historical behavior data and real-time multimodal information in the current scenario. Historical behavior data can include the user's historical operation records in the vehicle, preference settings, common routes, frequented places, etc., which reflects the user's habits and preferences. Real-time multimodal information covers the voice, image, vehicle status, environmental parameters and network information in the current scenario, providing the specific details needed to understand the current situation.

[0129] Optionally, once this information is collected, the intelligent voice service system will call the information processing model to perform deep reasoning on it. The information processing model may include machine learning (ML) and deep learning (DL) technologies, which can extract patterns of user preferences and habits from historical behavior data, while analyzing real-time multimodal information to understand the user's current needs and context.

[0130] Optionally, based on the inference results, the information processing model will generate a personalized target task execution strategy, which will guide how the vehicle responds to the user's needs, including vehicle control, service recommendations, and proactive service strategies. Since the strategy is generated based on a deep understanding of user preferences and the current situation, it can more accurately meet user needs and provide a more considerate service experience.

[0131] For example, in a driving scenario, the intelligent voice service system may detect that the user often adjusts the air conditioning temperature at a certain time point, so at the same time point next time, the intelligent voice service system will automatically adjust the air conditioning to suit the user's temperature preference. In a parking scenario, if the user's historical data shows that the user tends to park in the shade, the intelligent voice service system will combine the current weather information and the surrounding environment to find the most suitable shaded parking space.

[0132] Optionally, by repeatedly calling the information processing model for reasoning and executing strategies, the intelligent voice service system can continuously learn and remember the user's preferences and habits, thereby improving its accuracy in determining user needs over time. This learning and memory ability enables the intelligent voice service system to adapt to changes in user needs and provide more personalized and intelligent services. For example, if a user often chooses the same location to park in a specific parking lot, the intelligent voice service system will learn this habit, and in similar situations in the future, the system can actively recommend this location as a parking space even if the user does not explicitly indicate it.

[0133] In an embodiment of the present invention, by integrating user historical data and real-time data and calling the information processing model for deep reasoning, the intelligent voice service system can significantly improve its ability and accuracy in understanding user needs. This method based on data learning and contextual reasoning not only improves user satisfaction and experience, but also demonstrates the potential of intelligent systems in personalized services and scene adaptation. In summary, the intelligent voice service system achieves more accurate and intelligent demand understanding and service provision by learning user preferences and habits and combining multimodal information in the current context. Optionally, the intelligent voice service system calls the information processing model in a staged manner, and can generate an initial task execution strategy based on user historical behavior data, and adjust this strategy according to real-time multimodal information to obtain the final target task execution strategy.

[0134] Optionally, the intelligent voice service system will call an information processing model to analyze the user's vehicle historical behavior data. This model may use machine learning algorithms, such as cluster analysis, association rule learning, etc., to identify patterns in user habits and preferences. Based on these patterns, the model will generate an initial task execution strategy that reflects the user's behavioral tendencies in similar situations.

[0135] For example, if historical data shows that the user often goes to a nearby coffee shop every Friday afternoon, the system will generate an initial strategy to recommend a route to the coffee shop on Friday afternoons. If the user often adjusts the seat heating in winter, the intelligent voice service system may generate an initial strategy to automatically turn on the seat heating function in cold winter weather.

[0136] Optionally, after generating the initial strategy, the intelligent voice service system will further adjust it using real-time multimodal information to adapt it to the current specific situation. Multimodal information may include the user's voice command, the current vehicle status (e.g., fuel level, battery power), environmental parameters (e.g., weather, road conditions), and network information (e.g., real-time traffic conditions, business hours at the destination).

[0137] For example, if the initial strategy recommends that the user go to a coffee shop, but the real-time weather information shows that there is a rainstorm warning, the intelligent voice service system will adjust the strategy and recommend a closer coffee shop or suggest that the user go to a safe place to shelter from the rain first. If the initial strategy is to turn on the seat heating based on the user's habit, but the real-time vehicle status information shows that the battery power is low, the system will adjust the strategy and suggest that the user use other heating methods to avoid excessive battery consumption.

[0138] Optionally, by combining the analysis results of historical behavior data and real-time multimodal information, the information processing model will generate a more accurate target task execution strategy. The above target task execution strategy not only takes into account the user's historical habits, but also fully considers the changes in the current situation, so as to better meet the needs of users and provide more intelligent services.

[0139] For example, in a driving scenario, if the user's historical data shows that he or she prefers quiet music, but the real-time vehicle status information shows that the external noise is high, the system will adjust the volume strategy to ensure that the sound of the music playback can cover the external noise without being too loud.

[0140] In an embodiment of the present invention, the above process embodies the multi-stage decision-making mechanism of the intelligent voice service system, which can integrate historical data and real-time situations to generate a target task execution strategy that is close to user preferences and adapts to the current environment. This method not only improves the accuracy and personalization of the strategy, but also enhances the flexibility and adaptability of the system, and can better cope with changing environments and user needs. Through deep reasoning of information processing models, intelligent voice services can continuously learn and optimize. Over time, it will be able to more and more accurately predict user intentions and needs, and provide more intimate and intelligent services. This process also shows how to use machine learning and deep learning technologies to process complex multimodal data to achieve intelligent situational analysis and personalized service recommendations.

[0141] In summary, the intelligent voice service system can generate highly personalized and scenario-adaptive target task execution strategies by calling the information processing model in stages, combining historical behavior data and real-time multimodal information, which significantly improves the user experience and vehicle intelligence level.

[0142] Optionally, the information processing model is used to dynamically adjust the initial task execution strategy according to the environmental parameters of the geographic environment surrounding the vehicle to generate a target task execution strategy that is more in line with the current situation.

[0143] Optionally, environmental parameters of the geographic environment around the vehicle are collected through a multimodal sensing device (detection device). These parameters may include weather conditions (e.g., temperature, humidity, wind speed, probability of rainfall), road conditions (e.g., traffic congestion, slippery road surface), geographic information (e.g., nearby service facilities, attractions, gas stations, etc.), and possible environmental noise levels. Through devices and technologies such as on-board cameras, microphones, GPS positioning, and weather application programming interfaces (APIs), the intelligent voice service system can obtain and analyze these environmental parameters in real time.

[0144] For example, the intelligent voice service system may obtain information about service facilities near the current vehicle location through GPS positioning and map API, determine the road traffic conditions through cameras and image recognition technology, evaluate the environmental noise level using microphones and voice recognition technology, and obtain real-time weather forecast data through weather API.

[0145] Optionally, the intelligent voice service system will use information processing models, such as deep neural networks, convolutional neural networks, or long short-term memory networks, to conduct in-depth analysis of the collected environmental parameters. The information processing model will evaluate the impact of these parameters on the initial task execution strategy, such as the impact of weather conditions on vehicle air conditioning control and road conditions on navigation route selection.

[0146] For example, if the weather forecast shows that the current temperature is high, the intelligent voice service system will adjust the air-conditioning control strategy to ensure a comfortable temperature in the car; if the road conditions show that there is traffic congestion ahead, the intelligent voice service system will re-plan a route to avoid the congestion; if there are scenic spots frequently visited by users nearby, the system may suggest that the user take a short tour.

[0147] Optionally, based on the analysis results of the environmental parameters, the information processing model will adjust the initial task execution strategy to adapt to the current surrounding geographical environment. The above adjustment process may involve modifying vehicle control parameters (such as air conditioning temperature, window switches, seat adjustment, etc.), updating route selection, providing service suggestions (such as nearby restaurants, gas stations, etc.) or active reminders (such as weather changes, road safety, etc.).

[0148] For example, if the initial strategy is to close the car windows according to the user's habits, but the real-time environmental parameters show that the current air pollution index is high, the intelligent voice service system will adjust the strategy and recommend the user to turn on the in-car circulation mode to avoid pollutants entering the car. If the initial strategy is to recommend a gas station that the user often uses, but the real-time information shows that the gas station is under maintenance, the intelligent voice service system will re-search and recommend other available gas stations nearby.

[0149] Optionally, after analyzing environmental parameters and adjusting strategies, the information processing model will generate a final target task execution strategy. This strategy integrates the user's historical preferences, current task requirements, and real-time changes in the surrounding geographical environment, ensuring the personalization, intelligence, and scenario adaptability of the service.

[0150] For example, in a travel scenario, if the initial strategy is to recommend a scenic spot based on user habits, but real-time weather information shows that there will be a storm at the location, the system will adjust the strategy and recommend an indoor activity venue that is safer or more suitable for the current weather. In daily driving, if the initial strategy is to set the air conditioning temperature according to user habits, but real-time weather parameters show that the external temperature changes suddenly, the system will adjust the strategy to ensure that the temperature inside the car is comfortable and energy-saving.

[0151] In an embodiment of the present invention, by analyzing and adjusting the environmental parameters of the vehicle's surrounding geographical environment, the intelligent voice service system can significantly improve the scene adaptability and user experience of its services. This method not only takes into account the user's personal preferences, but also fully considers the real-time dynamics of the external environment, ensuring the flexibility and applicability of the strategy. In addition, the above process also demonstrates the intelligent association ability of the information processing model in processing geographical environment information and vehicle control strategies, which can effectively cope with complex and changing situations and provide a safe, comfortable and energy-saving driving experience. Through continuous learning and adaptation, the intelligent voice service system will be able to better understand user needs and environmental changes, and provide users with more intimate and intelligent services. In summary, the intelligent voice service system dynamically adjusts the initial task execution strategy by utilizing the environmental parameters of the surrounding geographical environment, generates a highly scene-adaptive and personalized final strategy, and significantly improves the intelligence level and user experience of intelligent vehicle services.

[0152] The following further describes the process of how to call the information processing model to perform reasoning on multimodal information in this embodiment.

[0153] As an optional implementation, calling an information processing model to infer multimodal information to obtain a target task execution strategy for the vehicle, including: calling an information processing model to infer resource information of the vehicle's network resources in the multimodal information to obtain a target task execution strategy; or calling an information processing model to identify the vehicle's state information in the multimodal information to obtain a vehicle state identification result; determining a target task execution strategy corresponding to the state identification result, wherein the target task execution strategy is used to represent the rules for the vehicle to execute control task operations on devices associated with the state information.

[0154] In this embodiment, in the process of calling the information processing model to infer the multimodal information and obtain the target task execution strategy of the vehicle, the information processing model can be called to infer the resource information of the network resources of the vehicle in the multimodal information to obtain the target task execution strategy. In the process of calling the information processing model to infer the multimodal information and obtain the target task execution strategy of the vehicle, the information processing model can be called to identify the state information of the vehicle in the multimodal information to obtain the state identification result of the vehicle. The target task execution strategy corresponding to the state identification result can be determined. Among them, the target execution strategy can be used to represent the rules for the vehicle to perform control task operations on the equipment associated with the state information.

[0155] Optionally, the intelligent voice service system calls the information processing model to reason about multimodal information including vehicle network resources to formulate a more intelligent and scenario-adaptive target task execution strategy.

[0156] Optionally, the intelligent voice service system can collect resource information of network resources related to the vehicle through various channels. Such resource information may include: online maps and navigation services, providing real-time traffic conditions, route planning, traffic violation query and other functions; weather forecast API, providing current and future weather conditions, including temperature, humidity, precipitation probability, etc.; service facility information, such as the business hours, location, user reviews, etc. of nearby restaurants, gas stations, and charging stations; user account information, such as the user's music, video service subscriptions, social network status, etc., for personalized content recommendation; real-time news and information, providing news, stock information, sports event results, etc. that users are interested in. The resource information of the above network resources will be integrated with multimodal information such as voice, image, vehicle status, environmental parameters, etc. to provide comprehensive reasoning input for the information processing model.

[0157] Optionally, an information processing model, such as a deep learning model, is called to reason about the integrated multimodal information. This process may involve the following steps: Analyze network resource information. The information processing model will analyze network resource information, including the availability of service facilities, the matching degree of user personalized needs, and the correlation between resource information and the current vehicle status and environmental parameters. Fusion of multimodal data. The information processing model will fuse data from different modalities, such as voice commands with online map information, vehicle status data with weather forecast API data, to obtain a more comprehensive scene understanding. Reasoning user needs. Through deep learning technology, the model can understand the user's intentions and needs, even if these needs are implicit or indirectly expressed. For example, if the user says "I'm hungry", the model will combine location information and the business hours of nearby restaurants to infer that the user's possible need is to find a restaurant.

[0158] Optionally, based on the inference results, the information processing model will generate a target task execution strategy for network resources. This strategy may include: proactive service recommendations based on network resource information, such as recommending the nearest charging station, providing traffic warnings, and recommending activities or places that users may be interested in. Personalized content provision, providing personalized music, radio or video content based on user account information and preferences. Route planning, combining real-time traffic and weather information to plan the safest, fastest or most energy-saving driving route.

[0159] For example, if a user expresses fatigue after a long drive, the system can use network resource information to recommend nearby rest stops and provide options for purchasing coffee or snacks. It can also recommend a relaxing music playlist based on the user's preferences to help the user relax and refresh themselves.

[0160] Optionally, a series of operations will be implemented to meet the user's needs based on the generated target task execution strategy. This may include adjusting vehicle control parameters (such as seat heating, air conditioning temperature), starting navigation guidance, actively recommending services or content, etc. Through the above process, the intelligent voice service system can make full use of the real-time information of network resources, combine the user's personalized needs and the current vehicle status, and formulate a more intelligent and personalized target task execution strategy, which significantly improves the scene adaptability and service quality of the intelligent voice service system.

[0161] In the embodiment of the present invention, the intelligent voice service system can provide services that are closer to the user's interests and needs based on the user's network resource information, such as music and video recommendations, or personalized service suggestions based on location and time. By analyzing the real-time status of network resources, the system can make more reasonable decisions, such as route planning and rapid response of emergency services, thereby improving driving safety and efficiency. The intelligent voice service system can actively perceive user needs and use network resources to provide fast and accurate services, reducing the user's operation complexity and improving the convenience and satisfaction of the service.

[0162] Optionally, the intelligent voice service system performs in-depth identification and analysis of vehicle status information in multimodal information by calling an information processing model to generate a personalized and intelligent target task execution strategy for vehicle equipment control.

[0163] Optionally, the intelligent voice service system will call the information processing model to identify the vehicle status information contained in the multimodal information. The vehicle status information may include but is not limited to the following: vehicle speed and driving status, engine temperature and load, vehicle temperature and humidity, fuel or battery power, working status of lights, wipers, air conditioners and other equipment, and vehicle safety system status, such as braking system and tire pressure.

[0164] Optionally, the information processing model uses deep learning techniques, such as convolutional neural networks and recurrent neural networks, to identify and classify this state information for subsequent reasoning and decision-making. For example, the information processing model may identify that the current temperature inside the car is too high, the engine temperature is also at a high level, or detects that the wiper speed does not match the current rainfall. The information processing model will conduct an in-depth analysis of the identified vehicle state information to evaluate whether the working status of the vehicle equipment is normal and whether these states need to be adjusted to adapt to the current environment or improve driving comfort. For example, if the information processing model identifies that the temperature inside the car is too high, it will further analyze user behavior data and current weather information to determine whether to turn on the air conditioner and adjust it to an appropriate temperature.

[0165] Optionally, based on the results of state recognition and analysis, the information processing model will generate equipment control task operation rules associated with the vehicle state information, that is, the target task execution strategy. This strategy will guide the vehicle on how to adjust the equipment to adapt to the current state and improve driving safety and driving comfort. For example, if the state recognition result indicates that the engine temperature is too high, the intelligent voice service system can generate a strategy to recommend reducing the vehicle speed or finding the nearest parking spot for cooling. If it is recognized that the temperature inside the car is too high, the system will generate a strategy to turn on the air conditioner and adjust it to a comfortable temperature. If the state information shows that it is currently raining, the system will generate a strategy to automatically turn on the wipers and adjust the speed to match the actual rainfall.

[0166] Optionally, the intelligent voice service system will control the vehicle equipment accordingly according to the determined target task execution strategy. This may include automatically adjusting the air conditioning temperature, turning on the wipers, adjusting the vehicle speed, and even actively reminding the driver to pay attention to safety. Through the above process, the intelligent voice service system can automatically adjust the equipment to adapt to environmental changes based on the real-time status information of the vehicle, improving driving safety and driving comfort, while also demonstrating the intelligence of the system and the initiative of the service.

[0167] In the embodiment of the present invention, through the above method, the intelligent voice service system can automatically adjust the equipment control strategy according to the vehicle status information, reduce the driver's intervention needs, and improve the convenience and safety of driving. By real-time identification and analysis of vehicle status information, the intelligent voice service system can respond quickly to changing driving environments, such as automatically turning on air conditioning and wipers, etc., to improve driving comfort. The intelligent voice service system can actively monitor the vehicle status and adjust the equipment in time to adapt to user needs and environmental changes, reducing the operating burden and improving the overall driving experience.

[0168] The following further describes how to provide personalized proactive services in this embodiment, that is, how to proactively initiate a dialogue with a user and control the vehicle based on user feedback.

[0169] As an optional implementation, the method also includes: generating inquiry information that matches the dialogue intention, wherein the inquiry information is used to inquire whether it is allowed to control the vehicle to perform the task operation; obtaining reply information corresponding to the inquiry information; in response to the reply information indicating that it is allowed to control the vehicle to perform the task operation, starting from the following steps: obtaining multimodal information associated with the dialogue information.

[0170] In this embodiment, a query message matching the dialogue intention may be generated. If the user sends a corresponding reply message based on the query message, the reply message may be analyzed to determine whether the reply message allows the vehicle to be controlled to perform the task operation. If so, multimodal information associated with the dialogue message may be obtained. The query message may be used to inquire whether the vehicle is allowed to be controlled to perform the task operation.

[0171] Optionally, the intelligent voice service system interacts with the user for confirmation by generating inquiry information that matches the conversation intent to ensure that the user has explicit authorization when performing vehicle control task operations, thereby enhancing the security of the system and user participation.

[0172] Optionally, when the intelligent voice service system recognizes that the user may have a certain need or intention (for example, the user says "I am so cold"), the intelligent voice service system will generate an inquiry information that matches this intention, such as "Do you need my help to increase the temperature in the car?" The generation of this inquiry information is based on natural language understanding and dialogue management technology, ensuring the accuracy and pertinence of the inquiry.

[0173] Optionally, the intelligent voice service system obtains the user's response information to the above inquiry information through voice recognition or touch screen input. The user can clearly answer "yes" or "no", or give a more detailed demand description, such as "please adjust the temperature to 22 degrees". The user's response information is an important basis for the intelligent voice service system to make decisions, which directly affects whether the subsequent vehicle control task operation is executed.

[0174] Optionally, the intelligent voice service system will analyze the user's response information to determine whether the user allows the vehicle to perform a specific task operation. This analysis process may be based on text analysis technologies, such as sentiment analysis, intent recognition, etc., to ensure accurate understanding of the user's intention. For example, if the user answers "yes", the intelligent voice service system will recognize that the user allows the temperature in the car to be increased; if the user answers "no", the intelligent voice service system will not perform the temperature adjustment operation.

[0175] Optionally, once it is confirmed that the user allows the vehicle to perform a specific task operation, the system will obtain multimodal information associated with the current dialogue information. This information may include: Voice command: the user's specific requirements for temperature adjustment. Vehicle status information: the current vehicle's air conditioning system status, interior temperature, etc. Environmental information: external temperature, humidity, etc. Historical data: the user's past temperature preference settings in similar environments. By obtaining the above multimodal information, the intelligent voice service system can understand user needs more comprehensively and formulate more accurate and personalized control strategies.

[0176] Optionally, the intelligent voice service system will perform vehicle control task operations that match the user's reply information based on the acquired multimodal information. For example, adjust the air conditioning temperature to the user's required 22 degrees, close the windows to keep the car warm, or turn on the seat heating in cold weather. The above process not only reflects the dialogue interaction between the intelligent voice service system and the user, but also demonstrates the rigor and safety of the system's task execution, ensuring that each step of the operation is carried out under the premise of user permission, enhancing the user experience and user-friendliness of the system.

[0177] In the embodiment of the present invention, by conducting a clear dialogue confirmation with the user, the intelligent voice service system ensures that the user's authorization is obtained before executing the vehicle control operation, avoiding the potential safety hazards caused by unauthorized automatic operation. The intelligent voice service system can operate according to the user's clear reply, fully respecting the user's wishes and needs, making the interaction process more humane and considerate. Through natural language processing and multimodal information fusion, the intelligent voice service system can understand the user's implicit needs, execute more accurate and personalized vehicle control strategies, and improve the intelligence level of the service. In summary, by generating inquiry information that matches the conversation intention and performing vehicle control task operations based on the user's reply information, the intelligent voice service system not only ensures the safety and legality of the operation, but also significantly improves the intelligence level of the service and user experience, fully demonstrating the ability and value of the intelligent voice service system in humanized interaction and personalized services.

[0178] The following further describes the process of how to generate inquiry information matching the conversation intention in this embodiment.

[0179] As an optional implementation, query information matching the dialogue intent is generated, including: using an information processing model to predict the vehicle's behavior data to obtain target behavior data matching the dialogue intent, wherein the target behavior data is used to enable the vehicle to perform a task to be performed; and using the information processing model to generate query information from the target behavior data.

[0180] In this embodiment, in the process of generating inquiry information matching the dialogue intention, the information processing model can be used to predict the behavior data of the vehicle, thereby obtaining the target behavior data matching the dialogue intention. The information processing model can be used to generate inquiry information from the target behavior data. The target behavior data can be used to enable the vehicle to perform the task operation.

[0181] Optionally, the intelligent voice service system predicts vehicle behavior data by utilizing an information processing model, combines the user's conversation intention, generates target behavior data that matches the user's needs, and converts it into inquiry information to obtain the user's explicit permission to perform specific vehicle control tasks.

[0182] Optionally, the intelligent voice service system can use information processing models (such as deep learning models) to predict and analyze the vehicle's behavior data. These behavior data may include but are not limited to: vehicle driving modes, such as economy mode and sports mode; vehicle regular usage habits, such as common routes, times, and destinations; vehicle equipment usage frequency, such as air conditioning, audio, and navigation; vehicle performance in different environments: such as driving speed in rainy days and starting mode in snowy days.

[0183] Optionally, by analyzing the above historical data, the information processing model can predict the behaviors or tasks that the vehicle may need to perform under specific dialogue intentions. For example, when the user expresses "I'm so cold", the information processing model may predict that the temperature inside the car needs to be increased.

[0184] Optionally, based on the conversation intent, the information processing model will filter out the target behavior data that is most relevant to the current situation from the prediction results. This data will serve as the basis for the subsequent query information generation to ensure the accuracy and pertinence of the query. For example, if the user explicitly says "I'm so cold", the information processing model may predict that the user wants to increase the temperature in the car and use this demand as the target behavior data.

[0185] Optionally, the information processing model will use natural language generation (NLG) technology to convert the target behavior data into user-friendly inquiry information. This process may involve the following steps: Constructing an inquiry statement: The model will generate a statement asking the user whether to allow the behavior to be performed based on the target behavior data. For example, if the target behavior data is to increase the temperature in the car, the inquiry statement may be "Do you want to increase the temperature in the car?" Personalized inquiry: The system can customize the inquiry statement based on the user's preferences and historical interaction data to make it more in line with the user's language habits and needs. For example, the intelligent voice service system may ask "Do you think the temperature needs to be raised a little bit, like the last time you drove in the snow?".

[0186] Optionally, the inquiry information will be presented through voice or screen display, and the user's response will be waited for. If the user agrees, the system will perform the task operation that matches the target behavior data; if the user disagrees or proposes a modification, the system will adjust the operation based on the user's feedback. Through the above process, the intelligent voice service system can not only predict the user's possible behavioral needs, but also obtain the user's explicit permission in the form of inquiry, ensuring the safe execution of the control task and the personalization of the user experience.

[0187] In the embodiment of the present invention, by analyzing the vehicle behavior data, the system can predict the possible needs of the user and improve the initiative of the service. Before performing any vehicle control operation, the system will obtain the user's explicit permission through inquiry information to ensure the safety and legality of the operation. The generation of inquiry information fully considers the user's language habits and needs, making the interaction process more humane and considerate, and improving user satisfaction. In summary, by using the information processing model to predict vehicle behavior data and generate inquiry information that matches the conversation intention, the intelligent voice service system not only improves the intelligence and initiative of the service, but also ensures the safety of operation and the personalization of user experience.

[0188] Figure 2 is a flow chart of an information interaction method of a smart device according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0189] Step S202: Detecting the dialogue information of the target scene where the smart device is located.

[0190] In the technical solution provided in the above step S202 of the present invention, the dialogue content of the dialogue information is used to characterize the target scene.

[0191] In this embodiment, the dialogue information received by the smart device in the target scene is detected so that the current scene can be understood and corresponding services can be provided. In the above steps, the dialogue content of the dialogue information is used to characterize the target scene, thereby providing an important basis for subsequent scene analysis, task execution and personalized services. The smart device can be a device in the smart home field scene, such as a smart light bulb, a smart socket, a smart door lock, etc., which is only used as an example and is not specifically limited.

[0192] Step S204: Acquire multimodal information associated with the dialogue information.

[0193] In the technical solution provided in the above step S204 of the present invention, the multimodal information can be used to represent information in multiple modes that satisfies the dialogue intention of the dialogue content.

[0194] In this embodiment, after detecting the dialogue information in the target scene where the smart device is located, multimodal information associated with the dialogue information may be acquired.

[0195] Optionally, based on the detected dialogue information, multimodal information associated with the dialogue information is started to be acquired. The above method means that the smart device not only relies on dialogue information (voice command), but can also more deeply understand the user's needs and the current environment of the target scene by comprehensively analyzing multimodal information from multiple information sources.

[0196] Step S206: calling the information processing model to infer the multimodal information to obtain the target task execution strategy of the smart device.

[0197] In the technical solution provided in the above step S206 of the present invention, the information processing model is obtained by training the agent model using multimodal information samples in different scenarios, and the target task execution strategy is used to represent the rules for the task operation to be executed by the smart device.

[0198] In this embodiment, after obtaining the multimodal information associated with the dialogue information, the information processing model can be called to infer the multimodal information to obtain the target task execution decision of the smart device.

[0199] Optionally, a pre-trained information processing model is called to perform in-depth analysis and reasoning on the acquired multimodal information. The information processing model can comprehensively consider user conversation information, the status of smart devices, environmental parameters, historical behavior data and other information to identify users' potential needs and predict possible scenario changes.

[0200] Step S208: Control the smart device to execute the corresponding task operation according to the target task execution strategy.

[0201] In the technical solution provided in the above step S208 of the present invention,

[0202] In this embodiment, after obtaining the target task execution strategy, a specific smart device control instruction can be generated, and the smart device control instruction can be sent to the corresponding smart device control unit through the network or wireless communication technology, thereby realizing the automated operation of the smart device.

[0203] In the above steps S202 to S208 of the present application, if the dialogue information initiated by the user is monitored, the dialogue information under the target scene can be detected, and the multimodal information associated with the dialogue intention in the dialogue information can be obtained. The multimodal information can be input into the information processing model, and the information processing model is used to infer the multimodal information to determine the target task execution strategy. According to the target task execution strategy, the smart device is controlled to perform the corresponding task operation. In this embodiment, the target scene in which the current smart device is located can be actively identified in combination with the dialogue content and multimodal information, so that the user's dialogue intention can be understood more accurately. The above scene recognition capability enables the smart device to provide customized services according to the target environment, which improves the intelligence and personalization of the service. The information processing model can reason in different target scenarios, understand the deep meaning behind the multimodal information, and make intelligent decisions, so that it can not only respond to instructions, but also predict and initiate dialogues, and actively provide information or suggestions to users, which greatly enhances the intelligence and initiative of smart devices. To sum up, by realizing the free dialogue function, scene recognition service provision, multimodal perception method, and the autonomous thinking and logical reasoning ability of the information processing model, the purpose of providing users with a more natural, efficient and personalized interactive experience is achieved, thereby achieving the technical effect of improving the information interaction efficiency of smart devices and solving the technical problem of low information interaction efficiency of smart devices.

[0204] The technical solution of the embodiment of the present invention is illustrated below in conjunction with preferred implementation modes.

[0205] At present, in-vehicle intelligent voice systems mainly rely on simple voice recognition and command execution, lacking the ability of multimodal perception and autonomous thinking. These systems can usually only operate according to the user's explicit instructions, and cannot actively understand the user's intentions or make intelligent adjustments according to the environment. In addition, the current in-vehicle intelligent voice systems still have problems such as a single interaction method and insufficient dialogue capabilities, which cannot meet users' needs for intelligent and personalized services. Therefore, there is still a technical problem of low efficiency of vehicle information interaction.

[0206] However, an embodiment of the present invention proposes an intelligent voice service system based on multi-modal perception and autonomous thinking, which can actively understand user intentions, make intelligent adjustments according to the environment, and provide full-scene, conversational, personalized proactive services. Specifically, this embodiment adopts multi-modal perception technology to achieve a comprehensive understanding of user intentions and the surrounding environment through multiple perception methods such as voice, image, vehicle, environment, system, and network information. The AI ​​Agent is introduced, which has the capabilities of autonomous thinking, logical reasoning, summary generation, learning and memory, and can disassemble complex tasks and call tools to complete the task loop. The free dialogue function is realized, from the user getting on the car to getting off the car, the user is actively chatting about topics of interest, identifying scenes to provide services, and realizing full-scene, conversational, and personalized proactive services. The technical effect of improving the information interaction efficiency of the vehicle is achieved, and the technical problem of low information interaction efficiency of the vehicle is solved or partially solved.

[0207] The method of the embodiment of the present invention is further illustrated below.

[0208] Figure 3 is a schematic diagram of an intelligent voice service system based on multi-modal perception and autonomous thinking according to an embodiment of the present invention. Figure 3 As shown, free dialogue 301 realizes free dialogue with users through natural language processing technology and dialogue management system (DMS). DMS can identify user intentions, actively initiate dialogues, and make intelligent adjustments based on user feedback. AI Agent model 302 builds AI Agent model based on natural language processing (NLP), machine learning (ML) and deep learning (DL) technologies to enable it to have autonomous thinking and logical reasoning capabilities. AI Agent can analyze user historical data, learn user preferences and habits, and intelligently adjust task execution strategies based on environmental information. Multimodal perception technology 303 uses microphones, cameras, sensors and other devices, combined with deep learning algorithms, to achieve real-time perception and understanding of voice, images, vehicle status, environmental information, etc.

[0209] Alternatively, if Figure 3As shown, if the user needs to issue a command to control the vehicle directly or indirectly by voice, the obtained command can be analyzed to determine the current MVP scenario, such as scenario one (I am so hot) or scenario two (my car is so dirty), and a decision can be made to determine whether the vehicle needs to be controlled accordingly, such as whether the air conditioner needs to be turned on, whether the window needs to be closed, etc. In addition, after obtaining the command, the current vehicle status can be obtained, such as the vehicle control status, system status, etc. For example, the air conditioning status, window status, seat status, etc. can be obtained through memory. Network information can also be obtained through perception, such as weather conditions (rainfall conditions in the past three days, car wash index), encyclopedia (for example, surrounding car wash shops), etc., and the surrounding environment can also be obtained, such as obstacles and other nearby conditions. Combined with the scenario under the current command and the multimodal information obtained above, it can be decided whether the corresponding vehicle control needs to be executed.

[0210] Optionally, through the above method, the intelligence and autonomy of the intelligent voice service system are improved, the user's operation difficulty and complexity are reduced, and a rich user interaction mode is provided to meet the user's demand for intelligent and personalized services.

[0211] In this embodiment, the system design architecture and technical solutions proposed above can be extended to the field of smart home to realize cross-domain intelligent voice services. By integrating smart home (such as smart bulbs, smart sockets, smart door locks, etc.), smart office, etc. With this system, users can control various devices in their home through voice commands and enjoy a more convenient and intelligent life experience. Add more intelligent application scenarios. In order to avoid hardware or technical obstacles, or in certain specific scenarios, the system needs to perform operations according to preset rules. Users can set a series of instruction rules in advance according to their needs and habits. When the system receives a specific trigger condition, it directly performs operations according to the preset rules without autonomous thinking.

[0212] In the embodiment of the present invention, a comprehensive and in-depth understanding of the user's intention and the surrounding environment is achieved by integrating multiple perception methods such as voice, image, vehicle status, environmental parameters (such as nearby gas stations, nearby tourist attractions, etc.), system status (such as device power, seat status) and network information (such as weather forecast, traffic conditions). Creativity of autonomous thinking: Give user suggestions through reasoning and judgment based on user historical data and real-time data. This multi-modal data input greatly improves the system's perception ability and the accuracy of user needs, allowing the system to respond to user instructions and predict user needs more intelligently. An advanced autonomous thinking mechanism is introduced, which makes reasoning and judgment based on user historical data and real-time data, so as to give personalized suggestions or predict possible user behaviors. Through autonomous thinking, the system can not only respond to the user's direct instructions, but also actively provide useful information or suggestions when necessary, enhancing the system's intelligence level and user experience. Thereby achieving the technical effect of improving the information interaction efficiency of the vehicle, solving or partially solving the technical problem of low information interaction efficiency of the vehicle.

[0213] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0214] According to another aspect of an embodiment of the present invention, corresponding to the embodiment of the above-mentioned vehicle information interaction method, this specification also provides a vehicle information interaction system, Figure 4 is a system block diagram of a vehicle information interaction system according to an embodiment of the present invention. Figure 4 As shown, the vehicle information interaction system 400 may include: a detection device 402 , a processor 404 and a controller 406 .

[0215] The detection device 402 is used to detect the dialogue information in the target scene where the vehicle is located.

[0216] Processor 404 is used to obtain multimodal information associated with the dialogue information; call the information processing model to reason about the multimodal information to obtain the vehicle's target task execution strategy.

[0217] The controller 406 is used to control the vehicle to perform the corresponding task operation according to the target task execution strategy.

[0218] In the information interaction system of the vehicle of this embodiment, if the dialogue information initiated by the user is monitored, the dialogue information under the target scene can be detected, and the multimodal information associated with the dialogue intention in the dialogue information can be obtained. The multimodal information can be input into the information processing model, and the information processing model can be used to infer the multimodal information to determine the target task execution strategy. According to the target task execution strategy, the vehicle is controlled to perform the corresponding task operation. In this embodiment, the target scene in which the current vehicle is located can be actively identified by combining the dialogue content with the multimodal information, so that the user's dialogue intention can be understood more accurately. The above-mentioned scene recognition capability enables the in-vehicle intelligent voice system to provide customized services according to the target environment, which improves the intelligence and personalization of the service. The information processing model can reason in different target scenarios, understand the deep meaning behind the multimodal information, and make intelligent decisions, so that it can not only respond to instructions, but also predict and initiate dialogues, and actively provide information or suggestions to users, which greatly enhances the intelligence and initiative of the in-vehicle intelligent voice system. To sum up, by realizing the free dialogue function, scene recognition service provision, multimodal perception method, and the autonomous thinking and logical reasoning ability of the information processing model, the purpose of providing users with a more natural, efficient and personalized interactive experience is achieved, thereby achieving the technical effect of improving the information interaction efficiency of the vehicle, and solving or partially solving the technical problem of low information interaction efficiency of the vehicle.

[0219] According to another aspect of the embodiments of the present invention, corresponding to the embodiments of the vehicle information interaction method described above, the present specification also provides a vehicle information interaction device, Figure 5 is a structural block diagram of a vehicle information interaction device according to an embodiment of the present invention. Figure 5 As shown, the vehicle information interaction device 500 may include: a first detection unit 502 , a first acquisition unit 504 , a first calling unit 506 and a first control unit 508 .

[0220] The first detection unit 502 is used to detect the dialogue information in the target scene where the vehicle is located.

[0221] The first acquisition unit 504 is configured to acquire multimodal information associated with the dialogue information.

[0222] The first calling unit 506 is used to call the information processing model to infer the multimodal information to obtain the target task execution strategy of the vehicle.

[0223] The first control unit 508 is used to control the vehicle to perform corresponding task operations according to the target task execution strategy.

[0224] In this embodiment, the first detection unit 502 detects the dialogue information in the target scene where the vehicle is located. The first acquisition unit 504 acquires multimodal information associated with the dialogue information. The first calling unit 506 calls the information processing model to reason about the multimodal information and obtains the target task execution strategy of the vehicle. The first control unit 508 controls the vehicle to execute the corresponding task operation according to the target task execution strategy. Thus, the technical effect of improving the information interaction efficiency of the vehicle is achieved, and the technical problem of low information interaction efficiency of the vehicle is solved or partially solved.

[0225] According to another aspect of the embodiments of the present invention, corresponding to the embodiments of the information interaction method of the smart device, the present specification also provides an information interaction device of the smart device, Figure 6 is a structural block diagram of an information interaction device of an intelligent device according to an embodiment of the present invention. Figure 6 As shown, the information interaction device 600 of the smart device may include: a second detection unit 602 , a second acquisition unit 604 , a second calling unit 606 and a second control unit 608 .

[0226] The second detection unit 602 is used to detect the dialogue information of the smart device in the target scene.

[0227] The second acquisition unit 604 is configured to acquire multimodal information associated with the dialogue information.

[0228] The second calling unit 606 is used to call the information processing model to infer the multimodal information and obtain the target task execution strategy of the smart device.

[0229] The second control unit 608 is used to control the smart device to execute the corresponding task operation according to the target task execution strategy.

[0230] In this embodiment, the second detection unit 602 detects the dialogue information in the target scene where the smart device is located. The second acquisition unit 604 acquires multimodal information associated with the dialogue information. The second calling unit 606 calls the information processing model to reason about the multimodal information, and obtains the target task execution strategy of the smart device. The second control unit 608 controls the smart device to perform the corresponding task operation according to the target task execution strategy. Thereby, the technical effect of improving the information interaction efficiency of the vehicle is achieved, and the technical problem of low information interaction efficiency of the vehicle is solved or partially solved.

[0231] An embodiment of the present application further provides a vehicle, comprising: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present invention is executed when the program is running.

[0232] Figure 7is a structural block diagram of an automatic driving vehicle according to an embodiment of the present invention. Figure 7 As shown, the components of the autonomous driving vehicle 700 include but are not limited to a memory 710 and a processor 720. The processor 720 and the memory 710 are connected via a bus 730, and a database 760 is used to store data.

[0233] The autonomous vehicle 700 may also include an access device 740 that enables the autonomous vehicle 700 to communicate via one or more networks 750. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0234] In one embodiment of the present disclosure, the above components of the autonomous driving vehicle 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The structure block diagram of the autonomous driving vehicle shown is for illustrative purposes only and is not intended to limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.

[0235] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0236] An embodiment of the present application further provides a computer program product, including a computer program, which implements the methods in various embodiments of the present invention when executed by a processor.

[0237] An embodiment of the present application further provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0238] The embodiments of the present application further provide a computer program, which implements the methods in the above-mentioned embodiments of the present invention when executed by a processor.

[0239] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0240] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0241] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0242] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0243] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0244] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A vehicle information interaction method, characterized in that: include: Detecting dialogue information in a target scene where the vehicle is located, wherein dialogue content of the dialogue information is used to characterize the target scene; Acquire multimodal information associated with the conversation information, wherein the multimodal information is used to represent information in multiple modes that satisfies the conversation intention of the conversation content; Calling an information processing model to infer the multimodal information to obtain a target task execution strategy for the vehicle, wherein the information processing model is obtained by training an agent model using multimodal information samples in different scenarios, and the target task execution strategy is used to represent a rule for the operation of the task to be performed by the vehicle; According to the target task execution strategy, the vehicle is controlled to execute the corresponding task operation.

2. The method according to claim 1, characterized in that Acquiring multimodal information associated with the conversation information, including: Performing scene recognition on the dialogue information to obtain a scene type of the target scene; Determining an information perception strategy associated with the scene type, wherein the information perception strategy is used to represent a rule for detecting the multimodal information under the scene type; According to the information perception strategy, the vehicle is controlled to detect the multimodal information under the scenario type.

3. The method according to claim 2, characterized in that Determining an information awareness strategy associated with the scenario type, including: In the information perception strategy set, the information perception strategy is searched using the scene type, wherein the information perception strategy set includes the information perception strategies respectively associated with different scene types; the information perception strategy set includes voice perception strategies and / or image perception strategies, wherein the voice perception strategy is used to represent the rules for detecting voice information for the vehicle under the scene type, and the image perception strategy is used to represent the rules for detecting image information for the vehicle under the scene type.

4. The method according to claim 3, characterized in that The information awareness strategy set includes at least one of the following: a state awareness policy associated with a first scenario type, wherein the first scenario type is used to represent a service scenario type associated with a vehicle state of the vehicle, and the state awareness policy is used to represent a rule for detecting state information of the vehicle state under the first scenario type; an environment perception strategy associated with a second scenario type, wherein the second scenario type is used to represent a service scenario type associated with a surrounding geographical environment of the vehicle, and the environment perception strategy is used to represent a rule for detecting an environmental parameter of the surrounding geographical environment under the second scenario type; A network awareness policy associated with a third scenario type, wherein the third scenario type is used to represent a service scenario type associated with the network resources of the vehicle, and the network awareness policy is used to represent rules for obtaining resource information of the network resources through the network under the third scenario type.

5. The method according to claim 2, characterized in that: According to the information perception strategy, controlling the vehicle to detect the multimodal information in the scenario type includes: Determining at least one detection device in the vehicle according to the information perception strategy; A control instruction is triggered to the detection device to control the detection device to detect the multimodal information under the scene type.

6. The method according to any one of claims 1 to 5, characterized in that The information processing model is called to infer the multimodal information to obtain the target task execution strategy of the vehicle, including: Calling the information processing model to infer the historical behavior data to obtain an initial task execution strategy, wherein the historical behavior data is used to represent the behavior generated by the vehicle in a historical period; The information processing model is used to adjust the initial task execution strategy according to the environmental parameters of the surrounding geographical environment of the vehicle in the multimodal information to obtain the target task execution strategy.

7. The method according to any one of claims 1 to 5, characterized in that The information processing model is called to infer the multimodal information to obtain the target task execution strategy of the vehicle, including: calling the information processing model to infer the resource information of the network resources of the vehicle in the multimodal information to obtain the target task execution strategy; or Call the information processing model to identify the state information of the vehicle in the multimodal information to obtain a state identification result of the vehicle; determine the target task execution strategy corresponding to the state identification result, wherein the target task execution strategy is used to represent the rules for the vehicle to perform control task operations on the equipment associated with the state information.

8. An information interaction method for a smart device, characterized in that: include: Detecting dialogue information of a target scene in which a smart device is located, wherein dialogue content of the dialogue information is used to characterize the target scene; Acquire multimodal information associated with the conversation information, wherein the multimodal information is used to represent information in multiple modes that satisfies the conversation intention of the conversation content; Calling an information processing model to infer the multimodal information to obtain a target task execution strategy of the smart device, wherein the information processing model is obtained by training an agent model using multimodal information samples in different scenarios, and the target task execution strategy is used to represent a rule for the task operation to be performed by the smart device; According to the target task execution strategy, the smart device is controlled to execute the corresponding task operation.

9. A vehicle information interaction system, characterized in that: include: A detection device, used to detect dialogue information in a target scene where the vehicle is located, wherein the dialogue content of the dialogue information is used to characterize the target scene; A processor is used to obtain multimodal information associated with the dialogue information, wherein the multimodal information is used to represent information in multiple modes that satisfies the dialogue intention of the dialogue content; call an information processing model to infer the multimodal information to obtain a target task execution strategy for the vehicle, wherein the information processing model is obtained by training an agent model using multimodal information samples in different scenarios, and the target task execution strategy is used to represent a rule for the vehicle to perform a task operation; The controller is used to control the vehicle to perform corresponding task operations according to the target task execution strategy.

10. A vehicle, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 8 when running.

Images