Non-sensitive interaction method, device and equipment

Abstract

The invention provides a non-sensitive interaction method, device and equipment, and is applied to the technical field of intelligent cabin control. The method comprises the steps of obtaining an interface state of a main display interface in response to an operation instruction of a user; and generating a virtual execution environment for executing the operation instruction according to the interface state and the operation instruction. And rendering an application interface of the target application indicated by the operation instruction in the virtual execution environment, and loading the target application to the virtual execution environment. And according to the operation instruction, simulating and executing the user operation in the virtual execution environment. After the operation instruction is executed, the execution result of the operation instruction is fed back to the user, and the problems that a traditional GUI agent must preempt a main display interface when executing a task, so that interaction experience is split, and potential safety hazards exist can be solved.

Description

Technical Field

[0001] This application relates to the field of intelligent cockpit control technology, and in particular to a non-sensory interaction method, device and equipment. Background Technology

[0002] With the development of automotive intelligence, voice interaction has become the core interaction method in smart cockpits. However, due to commercial barriers or technical architecture limitations, third-party applications (such as food delivery, video, and social media software) are often unable to open their underlying control interfaces to the vehicle's infotainment system. This results in voice assistants being unable to deeply control these applications through APIs, creating a dilemma where the applications can be opened but not used.

[0003] To address this issue, graphical user interface (GUI)-based intelligent agent technology has emerged. GUI intelligent agents are mainly divided into two categories: one type simulates operations by obtaining a control tree through accessibility services, and the other type predicts operation coordinates by recognizing screenshots using visual language models. However, both approaches share a common drawback: they both presuppose that the intelligent agent must perceive and operate within the currently visible, real-world interface.

[0004] This technological premise leads to the following: when an intelligent agent performs a task, it must seize the focus of the main screen. If the user is using navigation or watching a video, a sudden change in the screen not only causes a broken experience, but also distracts the driver and poses a safety hazard. At the same time, the system restricts background applications from accessing UI information, which prevents the intelligent agent from operating background applications or processing complex commands in parallel. In addition, the complex skins and dynamic effects of real interfaces can easily interfere with the accuracy of visual recognition, and screenshot processing poses a risk of privacy leakage. Summary of the Invention

[0005] The purpose of this application is to provide a seamless interaction method, device, and equipment to solve the problem that traditional GUI intelligent agents must seize the main display interface when performing tasks, resulting in a fragmented interactive experience and security risks.

[0006] In a first aspect, embodiments of this application provide a seamless interaction method applied to an intelligent cockpit system. The method includes: responding to a user's operation command by acquiring the interface state of the main display interface; generating a virtual execution environment for executing the operation command based on the interface state and the operation command; rendering the application interface of the target application indicated by the operation command in the virtual execution environment and loading the target application into the virtual execution environment; simulating the execution of the user operation in the virtual execution environment according to the operation command; and providing feedback to the user on the execution result of the operation command after execution.

[0007] The seamless interaction method provided in this application determines the virtual execution environment based on the interface state of the main display interface of the intelligent cockpit system and the user's operation instructions. The loading, rendering, and operation execution of the target application are transferred to the virtual execution environment, avoiding the intelligent agent from taking over the focus of the main screen when performing tasks. Even if the user is using high-priority applications such as navigation or watching videos, the main display interface will not be interrupted or covered, eliminating the interference of sudden screen changes on the driver's attention and significantly improving driving safety.

[0008] One possible implementation involves generating a virtual execution environment for executing operation instructions based on the interface state and the operation commands. This includes: activating the seamless execution mode of the intelligent cockpit system when the interface state indicates the main display interface is in a preset high-priority scenario, thus maintaining the interface state of the main display interface unchanged; determining the task type of the task to be executed based on the operation commands; and creating a virtual execution environment in the background of the intelligent cockpit system if the task type does not require user visual confirmation. Alternatively, creating a virtual execution environment in a preset floating area of ​​the main display interface if the task type requires user visual confirmation.

[0009] One possible implementation involves creating a virtual execution environment, including: allocating a first frame buffer for the virtual execution environment in a memory region. The first frame buffer is isolated from the second frame buffer corresponding to the main display interface. The output target of the virtual execution environment is set to the first frame buffer. The display content in the first frame buffer does not participate in the compositing of the main display interface.

[0010] One possible implementation involves rendering the application interface of the target application, instructed by the operation, within a virtual execution environment. This includes: optimizing the application interface to generate a simplified interface of the target application; the simplified interface containing the core controls of the target application; and then rendering the simplified interface within the virtual execution environment.

[0011] One possible implementation involves optimizing the application interface to generate a simplified interface for the target application. This includes filtering out background rendering instructions from the target application to remove background images and dynamic effects, and / or increasing the display contrast of core controls.

[0012] One possible implementation involves simulating user operations in a virtual execution environment based on operation instructions. This includes: acquiring image data from the first frame buffer corresponding to the virtual execution environment; determining the coordinates of each operable control in the virtual execution environment based on the image data; identifying the target control and its operation type from the operable controls based on the operation instructions; constructing a simulated event based on the target control's coordinates and operation type; and injecting the simulated event into the input channel of the virtual execution environment to cause the target application to execute the simulated event.

[0013] One possible implementation of the seamless interaction method provided in this application embodiment may further include: when it is determined that the operation instruction is a composite instruction, determining multiple sub-operation instructions based on the composite instruction; creating corresponding virtual execution environments for each of the multiple sub-operation instructions; and executing the corresponding sub-operation instructions in parallel within the multiple virtual execution environments.

[0014] One possible implementation involves, after executing an operation instruction, providing feedback to the user regarding the execution result, including: obtaining the execution result of the operation instruction; determining the corresponding feedback method based on the content type of the execution result; and providing feedback to the user according to the feedback method.

[0015] Secondly, embodiments of this application provide a contactless interaction device for use in an intelligent cockpit system. The device includes: an acquisition module, a generation module, a rendering module, an execution module, and a feedback module.

[0016] The acquisition module is used to obtain the interface state of the main display interface in response to the user's operation command.

[0017] The generation module is used to generate a virtual execution environment for executing operation instructions based on the interface status and operation instructions.

[0018] The rendering module is used to render the application interface of the target application in the virtual execution environment, which is instructed by the operation command, and to load the target application into the virtual execution environment.

[0019] The execution module is used to simulate and execute user operations in a virtual execution environment based on operation instructions.

[0020] The feedback module is used to provide feedback to the user on the execution result of the operation command after it has been executed.

[0021] One possible implementation involves the module determining the virtual execution environment for executing operation instructions based on the interface state and operation commands. Specifically, this involves: activating the seamless execution mode of the intelligent cockpit system when the interface state indicates the main display interface is in a preset high-priority scenario, thus maintaining the interface state of the main display interface unchanged; determining the task type of the task to be executed based on the operation commands; and creating a virtual execution environment in the background of the intelligent cockpit system if the task type does not require user visual confirmation. Alternatively, creating a virtual execution environment in a preset floating area of ​​the main display interface if the task type requires user visual confirmation.

[0022] One possible implementation involves the determining module, when creating the virtual execution environment, specifically: allocating a first frame buffer for the virtual execution environment in a memory region. This first frame buffer is isolated from the second frame buffer corresponding to the main display interface. The output target of the virtual execution environment is set to the first frame buffer. The display content in the first frame buffer does not participate in the compositing of the main display interface.

[0023] One possible implementation involves the rendering module, when rendering the application interface of the target application in a virtual execution environment, specifically optimizing the application interface to generate a simplified interface of the target application: the simplified interface contains the core controls of the target application. The simplified interface is then rendered in the virtual execution environment.

[0024] One possible implementation is that, when the production module optimizes the application interface and generates a simplified interface for the target application, it specifically filters out background rendering instructions for the target application to remove background images and dynamic effects, and / or increases the display contrast of core controls.

[0025] One possible implementation involves the execution module, when simulating user operations in a virtual execution environment based on operation instructions, specifically performing the following: acquiring image data from the first frame buffer corresponding to the virtual execution environment; determining the coordinates of each operable control in the virtual execution environment based on the image data; determining the target control and its operation type from the operable controls based on the operation instructions; constructing a simulated event based on the target control's coordinates and operation type; and injecting the simulated event into the input channel of the virtual execution environment to cause the target application to execute the simulated event.

[0026] In one possible implementation, the contactless interaction device provided in this application embodiment can also be used to: when it is determined that the operation instruction is a composite instruction, determine multiple sub-operation instructions based on the composite instruction; create corresponding virtual execution environments for each of the multiple sub-operation instructions; and execute the corresponding sub-operation instructions in parallel within the multiple virtual execution environments.

[0027] One possible implementation is that, when the feedback module is used to provide feedback to the user on the execution result of an operation instruction, it specifically performs the following steps: Obtaining the execution result of the operation instruction; determining the corresponding feedback method based on the content type of the execution result; and providing feedback to the user according to the feedback method.

[0028] Thirdly, embodiments of this application provide a contactless interaction device that has the function of implementing the contactless interaction method of the first aspect or any possible implementation of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0029] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the non-sensory interaction method described in the first aspect or any possible implementation thereof.

[0030] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, enable the computer to execute the non-intrusive interaction method described in the first aspect or any possible implementation thereof.

[0031] The technical effects of any of the design methods in aspects two through five can be found in aspect one or in different possible implementations of aspect one, and will not be repeated here. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 A system architecture diagram of an intelligent cockpit system provided in this application embodiment; Figure 2 A system structure diagram of a contactless interaction system provided in this application embodiment; Figure 3 A flowchart illustrating a non-intrusive interaction method provided in this application embodiment; Figure 4 A specific example diagram of a non-sensory interaction method provided in this application embodiment; Figure 5 A schematic diagram of a non-sensory interaction device provided in this application embodiment; Figure 6 Another system architecture diagram of a seamless interaction system provided in this application embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0036] Among related technologies, GUI-based automation technologies are mainly divided into two categories. One category is the control tree parsing scheme based on accessibility services. This scheme uses the accessibility service interface provided by the operating system to obtain the UI control tree of the front-end interface, locates the target control by traversing node attributes, and simulates operations such as clicking. The characteristic of this type of scheme is that it can understand and operate the page by analyzing the control hierarchy without modifying the application's internal code.

[0037] Another type is the screenshot recognition solution based on visual language models. With the development of large-scale model technology, directly using visual language models to understand screenshots has become an emerging trend. These models can identify icons, buttons, and their semantic functions from the entire screen image, predict operation coordinates, and simulate operations through underlying events. This solution does not rely on the underlying control tree structure and is suitable for applications with opaque interface structures or non-standard control development.

[0038] However, both of the above solutions assume that the agent must perceive and operate on the currently visible, active, real-world display interface. This technical premise leads to problems in practical applications, including a fragmented interactive experience, potential driving safety hazards, limited background execution and multi-tasking concurrency bottlenecks, poor visual recognition stability, and privacy risks.

[0039] Based on this, embodiments of this application provide a contactless interactive control method, apparatus, and device. The method includes: acquiring the interface state of the main display interface in response to a user's operation command; determining a virtual execution environment for executing the operation command based on the interface state and the operation command; rendering the application interface of a target application in the virtual execution environment and loading the target application indicated by the operation command into the virtual execution environment; simulating the user operation in the virtual execution environment according to the operation command; and providing feedback to the user on the execution result of the operation command after execution.

[0040] The seamless interaction method provided in this application determines the virtual execution environment based on the interface state of the main display interface of the intelligent cockpit system and the user's operation instructions. The loading, rendering, and operation execution of the target application are transferred to the virtual execution environment, avoiding the intelligent agent from taking over the focus of the main screen when performing tasks. Even if the user is using high-priority applications such as navigation or watching videos, the main display interface will not be interrupted or covered, eliminating the interference of sudden screen changes on the driver's attention and significantly improving driving safety.

[0041] The methods provided in the embodiments of this application will now be described in conjunction with the specific accompanying drawings.

[0042] On the one hand, embodiments of this application provide an intelligent cockpit system. For example... Figure 1 As shown, the intelligent cockpit system 100 may include: a cockpit domain controller 101, a display and interaction device 102, an audio processing device 103, a virtual display 104, a communication module 105, and a sensor group 106.

[0043] The cockpit domain controller 101 is communicatively connected to the display interaction device 102, audio processing device 103, virtual display 104, communication module 105, and sensor group 106, respectively, and is used to run the vehicle operating system and coordinate the operation of various devices in the intelligent cockpit system. The cockpit domain controller 101 may include a processor and a memory, wherein the memory stores a computer program. When the processor executes the computer program, it implements the contactless interaction method provided in this application embodiment, determines the virtual execution environment based on the user's operation instructions and the interface state of the main display interface, loads the target application into the virtual execution environment and renders its application interface, simulates the execution of user operations in the virtual execution environment, and provides feedback to the user on the execution results of the operation instructions.

[0044] The display interaction device 102 is communicatively connected to the cockpit domain controller 101. The display interaction device 102 can be the main display screen of the smart cockpit device. The main display screen is used to display the interface content of foreground applications (such as navigation, video, vehicle settings, etc.) and supports touch interaction, allowing users to input commands to the system via touch. A floating display area can be provided on the main display screen. This floating display area is located in an unobstructed area (such as a corner) of the main display screen and is used to display a floating window for tasks requiring visual confirmation from the user. This floating window can present a simplified interface of the target application or the operation process of the intelligent agent.

[0045] Audio processing device 103 is communicatively connected to cockpit domain controller 101. Audio processing device 103 may include a microphone array and vehicle speakers. The microphone array can be used to collect user voice commands, supporting sound source localization and noise suppression. The collected voice commands are transmitted to the cockpit domain controller for parsing. The vehicle speakers are used to output audio feedback, such as music playback, voice announcements, and prompts. The audio feedback is generated and controlled by the cockpit domain controller based on the execution results of the operation commands, enabling feedback on task execution results to the user while the main display screen remains unchanged.

[0046] The virtual display 104, communicatively connected to the cockpit domain controller 101 and the display interaction device 102, is used to create a virtual execution environment in the background of the intelligent cockpit system. This virtual execution environment includes a virtual display device and a corresponding video memory buffer. The virtual display device is independent of the main display screen, and its rendering output is directed to an independent video memory buffer. The display content in the video memory buffer is scheduled by the cockpit domain controller 101 and does not participate in the image composition of the main display screen. The virtual display 104 also supports the creation of multiple independent virtual execution environments to achieve concurrent multi-task processing.

[0047] The communication module 105 is connected to the cockpit domain controller 101 for data interaction with the external network. It supports OTA upgrades, cloud service calls, and other functions. It can also obtain services such as real-time traffic information, online music, and third-party application data. The obtained data is transmitted to the cockpit domain controller 101 for its use.

[0048] The sensor group 106, which is communicatively connected to the cockpit domain controller 101, includes an in-vehicle camera, a driver monitoring sensor, and a vehicle status sensor. It is used to perceive the in-vehicle environment and the driver's status, and the collected data is transmitted to the cockpit domain controller 101 to provide data support for scene perception and interactive decision-making. For example, the driver monitoring sensor can determine whether the driver is focused, assisting the cockpit domain controller 101 in determining whether to activate the non-contact execution mode.

[0049] It should be noted that the above Figure 1 The illustrated intelligent cockpit system 100 is merely an example illustrating the application scenario of the solution in this application, and is not intended to limit the application scenario of the solution in this application.

[0050] On the one hand, embodiments of this application provide a seamless interaction system 200, which can be deployed in... Figure 1 The intelligent cockpit system 100 shown is as follows. Figure 2 As shown, the seamless interaction system 200 may include a physical display layer 201, a virtual environment construction layer 202, and an intelligent agent execution layer 203.

[0051] The physical display layer 201 may include a display service module 2011 and a main display interface 2012. The display service module is responsible for managing the display output of the main physical screen and coordinating the layout and composition of various application windows. The main display interface is the screen area currently visible to the user and is used to present the interface content of high-priority applications (such as full-screen navigation, immersive games, etc.).

[0052] The virtual environment construction layer 202 includes a virtual display management module 2022 and a frame buffer management module 2023. When the system decides to adopt the virtual canvas mode (i.e., a task scenario that does not require user visual confirmation), the virtual display management module 2022 calls the system display service interface to create a virtual display in the memory area. This virtual display has a unique identifier (such as Virtual Display ID: 0) and does not correspond to any physical screen interface; its function is to provide an independent rendering target for the target application. The virtual display management module 2022 starts the target application (such as a third-party application like NetEase Cloud Music) and binds it to the virtual display, directing the application's interface rendering output to the virtual display.

[0053] During application loading, the Virtual Display Management Module 2022 is also responsible for optimizing the application interface. Addressing common third-party application issues such as splash screen ads, pop-up notifications, and complex album art backgrounds, the system forces the application to adopt a minimalist rendering mode in the virtual display through rendering configuration parameter injection or system-level filters: removing Gaussian blur backgrounds, blocking ads and pop-ups, retaining only core functional controls (such as the search box, playlist, and play button), and presenting these controls in high contrast. The optimized rendered image data is output to an independent frame buffer maintained by the frame buffer management module. This frame buffer is isolated from the frame buffer corresponding to the main display interface; its stored display content does not participate in the compositing of the main display interface, ensuring that all rendering processes in the virtual environment are completely invisible to the user.

[0054] The intelligent agent execution layer 203 includes an image acquisition module 2031, a visual recognition module 2032, an event construction module 2033, and an event injection module 2034. The image acquisition module directly reads image data from the frame buffer management module via the system API to obtain the interface image in the current virtual display. The visual recognition module calls a pre-trained visual language model to analyze the acquired image data and identify the position coordinates of each operable control in the image. Because it has been optimized, the interface image is a high-contrast simplified interface containing only the core controls. The visual recognition module can accurately locate the screen coordinates of target controls (such as the "search box" and "play button"), solving the problem of third-party applications not providing open control interfaces.

[0055] The event construction module 2033, based on the user's original operation command, determines the target control from the identified operable controls and the type of operation to be performed on that target control (such as click, long press, swipe, text input, etc.). Based on the target control's coordinates and the operation type, the event construction module 2033 constructs a corresponding simulated event, encapsulating information such as event type, event parameters, and timestamp. The event injection module 2034 injects the constructed simulated event into the corresponding input channel of the virtual display through the system input management service. The injected event is forwarded to the target application bound to the virtual display, causing the target application to respond to these events.

[0056] It should be noted that the above Figure 2 The illustrated seamless interaction system 200 is merely an example of the application scenario of the solution in this application, and is not intended to limit the application scenario of the solution in this application.

[0057] On the one hand, embodiments of this application provide a seamless interaction method, which can be deployed by... Figure 2 The seamless interactive system 200 shown is as follows Figure 1 The intelligent cockpit system 100 shown is executed. For example... Figure 3 As shown, the method may include the following steps.

[0058] S301, in response to the user's operation command, obtains the interface state of the main display interface.

[0059] The user's operation instructions can come from various interaction methods, such as voice instructions, touch instructions, finger instructions, and eye expression instructions, etc. This application does not limit them.

[0060] The main display interface can show the application type and running status of the currently running application on the smart cockpit system's screen. For example, whether it is a full-screen navigation application, an immersive game application, or a video communication application.

[0061] Specifically, the intelligent cockpit system monitors user input and generates corresponding operation commands based on that input. For example, when a user issues a natural language command such as "navigate to the company" or "play a Jay Chou song," the intelligent cockpit system collects the voice signal through a microphone array and converts it into a text-based operation command via a voice recognition engine. When a user taps an application icon or function button on the touchscreen, the intelligent cockpit system receives the touch event through a touch sensor and parses it into a corresponding operation command. In addition, the intelligent cockpit system can also receive user gesture commands or eye-tracking commands through gesture recognition sensors or eye-tracking sensors.

[0062] Meanwhile, the intelligent cockpit system obtains the current main display interface state through the activity management service. For example, the intelligent cockpit system queries the application component currently at the top of the display stack to obtain its application package name, activity class name, and window layout parameters. Window layout parameters include information such as whether the window is in full-screen mode, whether the status bar is hidden, and whether it is in immersive mode. For instance, if the current foreground application is navigation software in full-screen navigation mode, the intelligent cockpit system can determine that the interface state is a high-priority scenario of full-screen navigation. This high-priority scenario includes, but is not limited to, the running state of a full-screen navigation application, a full-screen game application, or a video communication application. If the current foreground application is a desktop launcher or a regular settings list page, the intelligent cockpit system can determine that the interface state is a non-high-priority scenario, enter the normal foreground execution mode, and directly jump to the target app interface for execution. The intelligent cockpit system can also combine vehicle status sensors (such as vehicle speed sensors) to assist in the judgment. If the vehicle speed is higher than a preset threshold and the foreground application is a navigation or video application, the high-priority attribute of the scenario is further increased.

[0063] S302, based on the interface status and operation instructions, generates a virtual execution environment for executing operation instructions.

[0064] One possible implementation involves activating the seamless execution mode of the intelligent cockpit system when the interface state indicates that the main display interface is in a preset high-priority scenario, thus maintaining the interface state of the main display interface unchanged. Based on the operation instructions, the task type to be executed is determined.

[0065] Specifically, the intelligent cockpit system first determines whether the current main display interface is in a preset high-priority scenario. If the interface status indicates that it is in a high-priority scenario, the system activates the seamless execution mode. The core purpose of this mode is to keep the interface status of the main display interface unchanged, avoiding interruption of the user's current interactive activities due to the execution of user commands, thereby ensuring driving safety and user experience.

[0066] After activating the seamless execution mode, the system further determines the task type to be executed based on the content of the operation command. Task types are divided into two main categories: one category includes tasks that do not require visual confirmation from the user, such as switching playlists, adjusting the air conditioner temperature, or adding the current song to favorites—tasks with clear results that do not require the user to look at the screen for confirmation; the other category includes tasks that require visual confirmation from the user, such as checking the progress of a food delivery order, browsing a restaurant list, or confirming a navigation route—tasks that require the user to obtain information visually and may involve making selections. Depending on the task type, the system uses different methods to create the virtual execution environment.

[0067] One possible implementation is to create a virtual execution environment in the background of the intelligent cockpit system when the task type is determined to be one that does not require visual confirmation from the user.

[0068] Another possible implementation is to create a virtual execution environment in a preset floating area of ​​the main display interface when the task type is determined to require visual confirmation from the user.

[0069] Specifically, if the task type does not require visual confirmation from the user, the system creates a virtual execution environment in the background, which is completely invisible to the user; if the task type requires visual confirmation from the user, the system creates a virtual execution environment in a preset floating area (such as a corner area) of the main display interface. This environment is presented in the form of a floating window, which does not affect the main content of the main display interface, but provides the user with the necessary visual information.

[0070] Furthermore, regarding the two possible implementation methods mentioned above, the steps for creating a virtual execution environment in either the background of the intelligent cockpit system or in a preset floating area of ​​the main display interface may include: allocating a first frame buffer for the virtual execution environment in the memory area. The first frame buffer is isolated from the second frame buffer corresponding to the main display interface. The output target of the virtual execution environment is set to the first frame buffer. The display content in the first frame buffer does not participate in the image composition of the main display interface.

[0071] Specifically, the system allocates a first frame buffer for the virtual execution environment in the memory area. This first frame buffer is isolated from the second frame buffer corresponding to the main display interface to ensure that the display content of the two does not interfere with each other. The system sets the output target of the virtual execution environment to the first frame buffer, and the display content in the first frame buffer does not participate in the screen composition of the main display interface. At this time, the rendering result of the virtual execution environment only exists in memory and will not be displayed on the main screen without authorization.

[0072] For example, taking the user's command to "play 'Rice Fragrance' in NetEase Cloud Music" as an example, the intelligent cockpit system receives and parses the command through S301, and then determines through S302 that the current main display interface is in a high-priority scenario (such as full-screen navigation) and the task type does not require visual confirmation from the user (playing music does not require the user to look at the screen), and can determine to create a virtual execution environment in the background.

[0073] The system calls the display service interface to create a virtual display in the memory area. This virtual display has a unique identifier (such as Virtual Display ID: 0) and does not correspond to any physical screen interface; its purpose is to provide an independent rendering target for the target application. The system launches the target application, NetEase Cloud Music, and binds it to this virtual display, directing the application's interface rendering output to the virtual display.

[0074] S303 renders the application interface of the target application in the virtual execution environment and loads the target application, which is instructed by the operation command, into the virtual execution environment.

[0075] One possible implementation involves optimizing the application interface to generate a simplified version of the target application. This simplified interface contains the core controls of the target application. The simplified interface is then rendered in a virtual execution environment.

[0076] The loading process of the target application may include application startup, instantiation, and binding with the virtual execution environment.

[0077] Specifically, the system starts the target application through the application management service and redirects its display output to the virtual display corresponding to the virtual execution environment created in step S302, so that all interface rendering operations of the target application are completed in the virtual execution environment, rather than the main display interface.

[0078] During application interface rendering, the system can optimize the application interface to improve the accuracy and efficiency of subsequent visual recognition. The core goal of optimization is to generate a simplified interface for the target application, which contains only the core controls required to perform the current operation, while filtering out non-core visual elements. Core controls refer to interface elements directly related to user operation commands, such as search boxes, play buttons, list items, and confirmation buttons; non-core visual elements include splash screen ads, pop-up prompts, complex background images, dynamic effects, Gaussian blur areas, and other elements that may interfere with subsequent visual recognition.

[0079] Further steps to optimize the application interface and generate a simplified interface for the target application may include: filtering out background rendering instructions for the target application to remove background images and dynamic effects; and increasing the display contrast of core controls.

[0080] Specifically, at the rendering instruction level, filtering is performed by intercepting and ignoring background rendering instructions from the target application at the system level, thereby removing background images and dynamic effects and preventing these redundant information from consuming processing resources or interfering with subsequent recognition. At the display effect level, enhancement is performed by increasing the display contrast of core controls through image processing algorithms, making the outlines and text of controls clearer and more distinguishable, facilitating the accurate capture of control positions and types by subsequent visual recognition models. These two optimization methods can be used individually or in combination, flexibly selected according to the needs of the actual application scenario.

[0081] Another alternative implementation is to optimize the application interface and generate a simplified interface for the target application, which can also be achieved by traversing the view tree of the target application.

[0082] Specifically, the system obtains the view tree structure of the target application in the virtual execution environment through accessibility services or view access interfaces. The view tree consists of all view nodes of the current interface, and each node corresponds to an interface element (such as a button, text box, image, etc.) and contains attribute information such as the element's type, position, size, and content description.

[0083] The system traverses all nodes in the view tree, filtering out core controls and non-core visual elements according to preset recognition rules. These recognition rules can be determined based on control type characteristics, resource ID characteristics, content description characteristics, layout hierarchy characteristics, and visibility characteristics. Based on the filtering results, the system prunes the view tree, removing or hiding nodes identified as non-core visual elements, retaining only the nodes corresponding to core controls. The interface corresponding to the pruned view tree is the simplified interface.

[0084] After optimization, the application interface rendered by the system in the virtual execution environment is a simplified interface containing only core controls and presented in high contrast. This simplified interface retains all the interactive elements necessary for performing user operations while removing redundant information that may interfere with recognition, providing a high-quality input image for subsequent simulation execution.

[0085] For example, continuing from step S302 where the user issues the command "Play 'Rice Fragrance' in NetEase Cloud Music". The system has already created a virtual display in the background through step S302, and now, in step S303, the NetEase Cloud Music application is loaded onto this virtual display. During the application loading process, the system optimizes common features in the NetEase Cloud Music application, such as splash screen ads, pop-up prompts, and complex album cover backgrounds: it removes the Gaussian blur background from the interface using a system-level filter, blocks the display of splash screen ads and pop-ups, retains only core controls such as the search box, playlist, and play button, and presents these controls in a high-contrast manner to ensure their clear outlines and easy identification. After optimization, the simplified interface image data rendered in the virtual display is output to an independent frame buffer for use in subsequent steps.

[0086] S304, based on the operation instructions, simulates the execution of user operations in the virtual execution environment.

[0087] One possible implementation involves simulating user operations in a virtual execution environment based on operation instructions. This includes: acquiring image data from the first frame buffer corresponding to the virtual execution environment; determining the coordinates of each operable control in the virtual execution environment based on the image data; identifying the target control and its operation type from the operable controls based on the operation instructions; constructing a simulated event based on the target control's coordinates and operation type; and injecting the simulated event into the input channel of the virtual execution environment to cause the target application to execute the simulated event.

[0088] Specifically, the intelligent cockpit system first retrieves image data from the corresponding first frame buffer within the virtual execution environment established in steps S302 and S303. This first frame buffer stores the current interface image rendered by the target application in the virtual execution environment. This image has been optimized to be a simplified interface containing only core controls and presented with high contrast. The system reads the bitmap data in this frame buffer through a system-level interface or memory access method as input for subsequent visual recognition.

[0089] After acquiring image data, the system calls the visual recognition module to process the image data and identify the positions of each operable control in the image. The visual recognition module can use a pre-trained visual language model, which can detect all interactive control elements from the interface image and output the type of each control (such as button, input box, list item, etc.) and its coordinate position in the image (usually bounding box coordinates). This recognition process does not depend on the application's control tree structure, so it is applicable to any third-party application, including game applications with opaque interface structures or highly customized HMI applications.

[0090] Then, based on the user's original operation command, the system identifies the target control from the identified operable controls and determines the type of operation to be performed on that target control. For example, if the operation command is "Play 'Rice Fragrance'", the system needs to identify the search box control to input the song name, the search button control to perform the search, and the play button control corresponding to "Rice Fragrance" in the list to perform playback. Operation types include, but are not limited to, clicking, long-pressing, swiping, and text input. The system uses semantic matching to associate the intent of the operation command with the type and function of the control, selects the control with the highest matching degree as the target control, and determines the corresponding operation type.

[0091] After determining the coordinates and operation type of the target control, the system constructs corresponding simulated events. Simulated events are encapsulations of system-level input events, containing information such as event type (e.g., touch event, key event), event parameters (e.g., touch coordinates, swipe trajectory, input text), and timestamps. The system constructs corresponding event objects based on the operation type: for click operations, a single-point touch event is constructed and the time interval between pressing and releasing is set; for text input, a series of key events are constructed or an input method text submission event is set.

[0092] Finally, the system injects the constructed simulated events into the input channel of the virtual execution environment. This input channel is bound to the virtual display and is the only entry point for the virtual display to receive external input. The injected events are sent to the virtual display through the system input management service, and the virtual display forwards them to the bound target application, causing the target application to respond to these events as if a real user were operating on the screen. The entire simulation execution process is completed in the background, and the user cannot perceive any trace of operation on the main display interface.

[0093] For example, continuing from steps S302 and S303 where the user requests to "play 'Rice Fragrance' on NetEase Cloud Music," the system retrieves the image data of the current NetEase Cloud Music interface from the frame buffer corresponding to the virtual display in step S304. The visual recognition module identifies operable controls such as the search box, search button, and playlist items, along with their coordinates. Based on the operation command "play 'Rice Fragrance,'" the system first determines the target control as the search box, with the operation type being click and text input. The system constructs a click event and injects it into the search box's coordinates, then constructs a text input event and injects it into the characters "Rice Fragrance." Next, it identifies the search button, constructs a click event to execute the search, and finally identifies the play button corresponding to "Rice Fragrance" in the search results list, constructs a click event to execute playback. Throughout this process, the user still sees the navigation interface on the main display screen, but the background has already completed the song search and playback.

[0094] S305 provides feedback to the user on the execution result of the operation command after it is executed.

[0095] One possible implementation involves obtaining the execution result of the operation command. Based on the content type of the execution result, the corresponding feedback method is determined. The execution result is then fed back to the user according to the feedback method.

[0096] Specifically, after completing the simulated operation of the target application, the intelligent cockpit system obtains the execution result corresponding to the operation command. The content type of the execution result can be divided into various forms according to the nature of the operation command and the task completion status, including audio type, notification type, and interface type.

[0097] For audio-based execution results, such as user commands like "play a song" or "play a podcast," the system acquires audio stream data upon successful execution. The system determines the audio feedback method based on the content type of the execution result, outputting the audio stream to the system mixer for playback through the car speakers. When the user hears the music or podcast, they perceive that the command has been successfully executed, while the navigation or video display on the main screen remains unchanged, achieving a seamless integration of auditory and visual perception.

[0098] For notification-type execution results, such as user commands like "set the air conditioner temperature to 24 degrees" or "add the current song to favorites," the system obtains the operation completion status information upon successful execution. The system determines the notification feedback method based on the content type of the execution result and displays a brief notification message in the notification area of ​​the main display interface (such as the status bar or notification center), such as "Air conditioner set to 24 degrees" or "Added to favorites." The notification message is presented in a non-intrusive manner, disappearing automatically after a brief display, without affecting the user's viewing and operation of the main display interface content.

[0099] For execution results of different interface types, such as user commands like "navigate to the company" or "check today's weather," the system retrieves a ready-to-use target application interface after successful execution. The system determines the interface switching feedback method based on the content type of the execution result, switching the target application interface that has been loaded and completed in the virtual execution environment to the main display interface. Taking a navigation command as an example, the system completes all operations such as destination input and route planning in the background virtual execution environment. After obtaining a navigation interface with a planned route, it seamlessly switches this interface to the main display interface, providing the user with a directly usable navigation screen, eliminating cumbersome intermediate steps.

[0100] During the execution of the three feedback methods described above, the original foreground application on the main display interface remains running and its displayed content is not covered or interrupted. For audio feedback and notification feedback, the original application interface is completely unaffected; for interface switching feedback, the original application interface is replaced by the target application interface, but this replacement is within the user's expectations and is completed smoothly, without any sudden interface hijacking or forced interruption.

[0101] Furthermore, for tasks that require visual confirmation from the user but have already been completed, the system can automatically close the floating window after feedback is received. For example, if a floating window is created for a task that requires visual confirmation from the user, and the user obtains necessary information through the floating window during task execution, the system can automatically close the floating window after a preset time (e.g., 5 seconds) after the task is completed, restoring the full display area of ​​the main display interface.

[0102] For example, continuing from steps S302 to S304, the scenario where the user issues the command "Play 'Rice Fragrance' on NetEase Cloud Music" is executed. After step S304, the system obtains the execution result in the form of an audio stream. Based on the content type, the system determines that audio feedback is the correct method and outputs the audio stream to the car's speakers, allowing the user to hear the song "Rice Fragrance." Throughout the playback process, the navigation screen on the main display remains unchanged, allowing the user to complete the music playback command without interrupting the navigation service, achieving a seamless interactive experience.

[0103] Furthermore, if the operation instruction is determined to be a compound instruction, multiple sub-operation instructions are determined based on the compound instruction. A corresponding virtual execution environment is created for each of the multiple sub-operation instructions, and the corresponding sub-operation instructions are executed in parallel within these multiple virtual execution environments.

[0104] Specifically, after receiving the user's operation command in step S301, the intelligent cockpit system first parses the command content using a natural language understanding model to determine whether the command is a composite command containing multiple independent tasks. A composite command can contain two or more independently executable sub-tasks in a single user command. For example, "play music and navigate to the company" includes both music playback and navigation planning tasks, while "turn on the air conditioning and search for nearby restaurants" includes both air conditioning control and restaurant search tasks.

[0105] When an operation instruction is determined to be a compound instruction, the system parses it into multiple independent sub-operation instructions. Each sub-operation instruction corresponds to an independently executable atomic task, with a clearly defined operation object and operation content. For example, "play music and navigate to the company" can be parsed into sub-operation instruction one "play a song in the music application" and sub-operation instruction two "plan a route to the company in the map application".

[0106] For the multiple sub-operation instructions obtained from parsing, the system executes the method described in steps S302 to S305 for each sub-operation instruction, including: creating a corresponding virtual execution environment for each sub-operation instruction, loading the target application indicated by each sub-operation instruction into the corresponding virtual execution environment, rendering the corresponding target application interface in each virtual execution environment, simulating the execution of the corresponding user operation in each virtual execution environment, and providing feedback to the user on the corresponding execution result after the execution is completed.

[0107] In this process, the virtual execution environments corresponding to multiple sub-operation instructions are independent of each other, and the simulation execution processes in each virtual execution environment are carried out in parallel. The system maintains multiple virtual displays and corresponding frame buffers simultaneously through a multi-task scheduling mechanism, allocating independent processing resources and execution threads to each sub-task. Operations such as image recognition, control positioning, and event injection in each virtual execution environment do not interfere with each other and can be performed concurrently.

[0108] Taking the complex instruction "play music and navigate to the company" as an example, after parsing the two sub-operation instructions, the system creates two independent virtual execution environments: Virtual execution environment A is used for the music playback task, loading music applications such as NetEase Cloud Music; Virtual execution environment B is used for the navigation planning task, loading navigation applications such as Gaode Maps. The two virtual execution environments run simultaneously, with the intelligent agent execution unit performing image recognition and simulation operations in parallel within each environment: recognizing the playback interface and clicking the play button in virtual execution environment A, and recognizing the search box, entering the destination "company," and clicking "start navigation" in virtual execution environment B. The two tasks are performed concurrently without waiting for each other, significantly improving the execution efficiency of complex instructions.

[0109] After the task is completed, the system provides feedback based on the result type of each subtask. For the music playback task, the feedback is audio output, and the user directly hears the music playing. For the navigation planning task, the feedback is a screen switch, changing the navigation interface with the planned route to the main display interface. The user experiences that the planned route is displayed simultaneously as the music plays, and the two tasks are completed quickly and concurrently.

[0110] The methods provided in the embodiments of this application will be described below with reference to specific examples.

[0111] Example 1 like Figure 4 As shown, taking the scenario where a user is viewing full-screen map navigation on the main display interface while asking "Where is my takeout?" as an example, the intelligent cockpit system receives and parses the user's voice command in step S201, recognizing that the command involves checking the takeout order progress, and the target application is a third-party takeout application such as Meituan Takeout. In step S202, the system determines that the current main display interface is in a high-priority scenario (full-screen navigation), and the task type requires visual confirmation from the user (checking the takeout progress requires the user to obtain information visually). Therefore, it determines to create a virtual execution environment in the preset floating area of ​​the main display interface.

[0112] The system utilizes a multi-window management mechanism to forcibly create a floating window in the non-critical information area of ​​the main display interface (such as the lower right corner). This floating window adopts a picture-in-picture mode, has the attribute of being displayed on top, and is small in size, so it will not obscure the core content of the main display interface (such as the main route area of ​​the navigation map). Although the Meituan Waimai app may not natively support in-car split-screen or small window display, the system forces the application to be loaded into the floating window through the window management service, so that its interface content is rendered and output to the display area corresponding to the floating window.

[0113] After the target application is created and loaded in the floating window, the system displays the order details page of the Meituan Waimai app in the floating window. Users can catch a glimpse of the content in the floating window, such as the rider's location map and estimated delivery time, while the map navigation and route guidance in the main task area remain completely unobstructed, without any page jumps or interruptions.

[0114] To make users perceive that the agent is actively interacting with a third-party application (rather than simply fetching background data), the system displays a virtual focus point for the agent on a floating window. This focus point is presented as a semi-transparent circular cursor, and its movement trajectory is synchronized with the click positions simulated by the agent in the background. For example, after the agent automatically recognizes the "Refresh" button or the "View Rider Location" link in the floating window, it controls the cursor to move to that position and perform a click operation. Users can intuitively see the cursor movement and click process in the floating window, enhancing the transparency and credibility of the interaction.

[0115] After the information is displayed, the system automatically destroys the floating window and restores the main display interface to full-screen mode. Automatic destruction is triggered by: a preset timeout (e.g., 5 seconds), user voice confirmation (e.g., saying "OK"), or the user closing the floating window via touch. After the floating window is destroyed, the main display interface returns to a full-screen navigation screen, allowing the user to continue focusing on driving.

[0116] The foregoing mainly describes the solutions provided in the embodiments of this application from the perspective of the working principle of the device. It is understood that, in order to achieve the above functions, the contactless interaction device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] This application embodiment can divide the contactless interaction device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module.

[0118] It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. When dividing functional modules according to their respective functions, Figure 5 A schematic diagram illustrating a possible composition of the contactless interaction device involved in the above and embodiment examples is shown. For example... Figure 5 As shown, the non-sensory interaction device 500 may include: an acquisition module 501, a generation module 502, a rendering module 503, an execution module 504, and a feedback module 505.

[0119] The acquisition module 501 is used to support the execution of the contactless interaction device 500. Figure 3 S301 in the schematic of the non-sensory interaction method.

[0120] Generation module 502 is used to support the execution of the contactless interaction device 500. Figure 3 S302 in the schematic of the non-sensory interaction method.

[0121] Rendering module 503 is used to support the execution of the non-contact interaction device 500. Figure 3 S303 in the schematic of the non-sensory interaction method.

[0122] Execution module 504 is used to support the execution of the contactless interaction device 500. Figure 3 S304 in the schematic of the non-sensory interaction method.

[0123] Feedback module 505 is used to support the execution of the contactless interaction device 500. Figure 3 S305 in the schematic of the non-sensory interaction method.

[0124] One possible implementation involves the module determining the virtual execution environment for executing operation instructions based on the interface state and operation commands. Specifically, this involves: activating the seamless execution mode of the intelligent cockpit system when the interface state indicates the main display interface is in a preset high-priority scenario, thus maintaining the interface state of the main display interface unchanged; determining the task type of the task to be executed based on the operation commands; and creating a virtual execution environment in the background of the intelligent cockpit system if the task type does not require user visual confirmation. Alternatively, creating a virtual execution environment in a preset floating area of ​​the main display interface if the task type requires user visual confirmation.

[0125] One possible implementation involves the determining module, when creating the virtual execution environment, specifically: allocating a first frame buffer for the virtual execution environment in a memory region. This first frame buffer is isolated from the second frame buffer corresponding to the main display interface. The output target of the virtual execution environment is set to the first frame buffer. The display content in the first frame buffer does not participate in the compositing of the main display interface.

[0126] One possible implementation involves the rendering module, when rendering the application interface of the target application in a virtual execution environment, specifically optimizing the application interface to generate a simplified interface of the target application: the simplified interface contains the core controls of the target application. The simplified interface is then rendered in the virtual execution environment.

[0127] One possible implementation is that, when the production module optimizes the application interface and generates a simplified interface for the target application, it specifically filters out background rendering instructions for the target application to remove background images and dynamic effects, and / or increases the display contrast of core controls.

[0128] One possible implementation involves the execution module, when simulating user operations in a virtual execution environment based on operation instructions, specifically performing the following: acquiring image data from the first frame buffer corresponding to the virtual execution environment; determining the coordinates of each operable control in the virtual execution environment based on the image data; determining the target control and its operation type from the operable controls based on the operation instructions; constructing a simulated event based on the target control's coordinates and operation type; and injecting the simulated event into the input channel of the virtual execution environment to cause the target application to execute the simulated event.

[0129] In one possible implementation, the contactless interaction device provided in this application embodiment can also be used to: when it is determined that the operation instruction is a composite instruction, determine multiple sub-operation instructions based on the composite instruction; create corresponding virtual execution environments for each of the multiple sub-operation instructions; and execute the corresponding sub-operation instructions in parallel within the multiple virtual execution environments.

[0130] One possible implementation is that, when the feedback module is used to provide feedback to the user on the execution result of an operation instruction, it specifically performs the following steps: Obtaining the execution result of the operation instruction; determining the corresponding feedback method based on the content type of the execution result; and providing feedback to the user according to the feedback method.

[0131] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0132] The contactless interaction device 500 provided in this application embodiment is used to perform the above-mentioned... Figure 3 The non-sensory interaction method shown can therefore achieve the same effect as the non-sensory interaction method described above.

[0133] This application also provides a contactless interaction device that can execute the contactless interaction method and related steps described in the above method embodiments.

[0134] This application also provides a computer-readable storage medium storing instructions thereon, which, when executed, perform the non-sensory interaction method and related steps in the above method embodiments.

[0135] This application also provides a computer program product that, when run on a computer, causes the computer to execute the non-sensory interaction method and related steps described in the above method embodiments.

[0136] In some embodiments, the methods shown in this application can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0137] This application also provides a contactless interaction system 200, such as... Figure 6 As shown, the contactless interaction system 200 includes at least one processor 601 and at least one interface circuit 602.

[0138] As an example, when the contactless interaction system 200 includes a processor and an interface circuit, the processor can be... Figure 6 The processor 601 shown in the solid box (or the processor 601 shown in the dashed box) can be an interface circuit. Figure 6 The interface circuit 602 is shown in the solid box (or the dashed box). When the contactless interaction system 200 includes two processors and two interface circuits, the two processors include... Figure 6 The processor 601 shown in the solid box and the processor 601 shown in the dashed box, these two interface circuits include Figure 6 Interface circuit 602 is shown in both solid and dashed boxes. No limitations are imposed on this.

[0139] Processor 601 and interface circuit 602 can be interconnected via a line. For example, interface circuit 602 can be used to receive signals. Alternatively, interface circuit 602 can be used to send signals to other devices (e.g., processor 601). For instance, interface circuit 602 can read computer instructions stored in memory and send those instructions to processor 601. Processor 601 executes the instructions and, in conjunction with input / output devices, implements the various steps in the above embodiments, such as implementing... Figure 3 and Figure 4 The illustrated method embodiments demonstrate the various steps performed. Of course, this contactless interaction system may also include other discrete components, and this application embodiment does not specifically limit this.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0144] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to it, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A seamless interaction method, characterized in that, Applied to intelligent cockpit systems, the method includes: In response to user commands, obtain the interface state of the main display screen; Based on the interface state and the operation instructions, a virtual execution environment is generated to execute the operation instructions; The application interface of the target application indicated by the operation instruction is rendered in the virtual execution environment, and the target application is loaded into the virtual execution environment; The user operation is simulated and executed in the virtual execution environment according to the operation instructions. After executing the operation instruction, the result of the operation instruction is fed back to the user.

2. The method according to claim 1, characterized in that, The step of generating a virtual execution environment for executing the operation instructions based on the interface state and the operation instructions includes: When the interface state indicates that the main display interface is in a preset high-priority scenario, the seamless execution mode of the intelligent cockpit system is activated to keep the interface state of the main display interface unchanged. Based on the operation instructions, determine the task type of the task to be executed; If the task type is determined to be one that does not require visual confirmation from the user, the virtual execution environment is created in the background of the intelligent cockpit system. Alternatively, if the task type is determined to require visual confirmation from the user, the virtual execution environment may be created in a preset floating area of ​​the main display interface.

3. The method according to claim 2, characterized in that, Creating the virtual execution environment includes: A first frame buffer is allocated in the memory region for the virtual execution environment; the first frame buffer is isolated from the second frame buffer corresponding to the main display interface. The output target of the virtual execution environment is set to the first frame buffer; the display content in the first frame buffer does not participate in the screen composition of the main display interface.

4. The method according to claim 1, characterized in that, The process of rendering the application interface of the target application indicated by the operation instructions in the virtual execution environment includes: The application interface is optimized to generate a simplified interface for the target application: the simplified interface includes the core controls of the target application; The simplified interface is rendered in the virtual execution environment.

5. The method according to claim 4, characterized in that, The optimization of the application interface to generate a simplified interface for the target application includes: Filter out background rendering instructions of the target application to remove the background image and dynamic effects of the target application; And / or, increase the display contrast of the core controls.

6. The method according to claim 1, characterized in that, The step of simulating the execution of the user operation in the virtual execution environment according to the operation instruction includes: Obtain the image data in the first frame buffer corresponding to the virtual execution environment; Based on the image data, determine the coordinates of each operable control in the virtual execution environment; Based on the operation instructions, determine the target control and the operation type of the target control from the operable controls; Based on the control coordinates of the target control and the operation type, a simulated event is constructed; The simulated event is injected into the input channel of the virtual execution environment so that the target application executes the simulated event.

7. The method according to claim 1, characterized in that, The method further includes: If the operation instruction is determined to be a composite instruction, multiple sub-operation instructions are determined based on the composite instruction; A corresponding virtual execution environment is created for each of the multiple sub-operation instructions, and the corresponding sub-operation instructions are executed in parallel in the multiple virtual execution environments.

8. The method according to claim 1, characterized in that, The step of providing feedback to the user on the execution result of the operation instruction after execution includes: Obtain the execution result of the operation instruction; The corresponding feedback method is determined based on the content type of the execution result; The execution result is fed back to the user according to the feedback method described above.

9. A contactless interactive device, characterized in that, The device, used in intelligent cockpit systems, includes: The acquisition module is used to obtain the interface state of the main display interface in response to the user's operation command; The generation module is used to generate a virtual execution environment for executing the operation instructions based on the interface state and the operation instructions; The rendering module is used to render the application interface of the target application indicated by the operation instructions in the virtual execution environment, and load the target application into the virtual execution environment; The execution module is used to simulate and execute the user operation in the virtual execution environment according to the operation instruction; The feedback module is used to provide feedback to the user on the execution result of the operation instruction after the operation instruction is executed.

10. A contactless interactive device, characterized in that, The contactless interaction device includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, and the processor executing the machine-executable instructions to implement the contactless interaction method according to any one of claims 1 to 8.

Images