A vehicle-mounted voice interaction method, device, equipment and medium

Abstract

A vehicle-mounted voice interaction method, device, equipment and medium are disclosed, the method comprising: receiving a voice instruction input by a user, and synchronously acquiring a current page screenshot of a vehicle-mounted screen; determining whether the current page supports the voice instruction based on the current page screenshot through a pre-trained graphical interface perception model; if so, generating a corresponding graphical interface operation based on the voice instruction, and executing the graphical interface operation in the current page. The present application cooperatively understands the voice instruction of the user and the real-time page state of the vehicle-mounted screen, so that the vehicle-mounted voice system can dynamically adapt to the current operation scenario of the user, avoid forced jumping to a default application or misexecution of an instruction due to skill solidification, and thus improve the accuracy of voice interaction and the consistency of user experience.

Description

Technical Field

[0001] This application relates to the field of voice interaction, specifically to an in-vehicle voice interaction method, device, equipment, and medium. Background Technology

[0002] With the rapid development of intelligent cockpit technology, in-vehicle voice assistants have become a core component of modern automotive human-machine interaction systems, and are widely used in high-frequency scenarios such as navigation, entertainment, and air conditioning control.

[0003] Currently, mainstream automakers and suppliers generally adopt a fixed interaction method of "ASR (Acoustic Speech) → NLU (Natural Language Understanding) → Preset Skill Execution". Under this architecture, voice commands are semantically parsed and mapped to a set of predefined functional skills, which are then executed by the system's default application. However, this approach has significant limitations: its execution logic highly relies on static configuration. While it can cover most basic function calls, it has a clear weakness in adapting to users' personalized usage habits, resulting in the execution results of voice commands often being out of sync with the user's true intentions.

[0004] For example, in actual use, users have manually installed and long-term used the First Music app as their primary audio player, and their personal playlists already include First Music songs. When a user issues a voice command to "play First Music song" while browsing the First Music reference interface on the car's central control screen, their reasonable expectation is to complete the playback operation directly within the currently displayed First Music app. However, due to skill binding, traditional voice systems will still forcibly invoke the preset default music app, not only causing application switching redundancy and interrupting the current operation flow, but also potentially causing the task to fail because the First Music song is unavailable in the default app.

[0005] Therefore, there is an urgent need for an in-vehicle voice interaction solution that can more accurately understand the user's true intentions and dynamically adapt the execution strategy according to the actual usage scenario. Summary of the Invention

[0006] This application provides an in-vehicle voice interaction method, device, equipment, and medium, which can solve the technical problem in the prior art where voice assistants cannot adapt to the user's current interface state and personalized usage habits due to their reliance on fixed application mapping, resulting in command execution results deviating from the user's true intentions.

[0007] In a first aspect, embodiments of this application provide an in-vehicle voice interaction method, the in-vehicle voice interaction method comprising: It receives voice commands input by the user and simultaneously captures a screenshot of the current page on the in-vehicle screen; Based on the current page screenshot, a pre-trained graphical interface perception model is used to determine whether the current page supports the voice command. If supported, a corresponding graphical interface operation is generated based on the voice command, and the graphical interface operation is executed on the current page.

[0008] In conjunction with the first aspect, in one implementation, the step of determining whether the current page supports the voice command based on the current page screenshot using a pre-trained graphical interface perception model includes: The screenshot of the current page is input into the graphical interface perception model to obtain the output list of operation instructions supported by the current page. The list of operation instructions contains one or more structured operation instructions, and each operation instruction is associated with an executable interface element identifier and a corresponding operation semantic. The voice commands are converted into command text, and the semantic information is extracted from them; The semantic information is semantically matched with the operation semantics of each operation instruction in the operation instruction list; If a matching operation command is found, it is determined that the current page supports the voice command; If no matching operation command is found, it is determined that the current page does not support the voice command.

[0009] In conjunction with the first aspect, in one implementation, the step of generating the corresponding graphical interface operation based on the voice command includes: The operation instruction that successfully matches the semantic information in the list of operation instructions is identified as the target operation instruction. Based on the semantic information, the target operation instruction, and the position information of the corresponding interface element in the current page screenshot, a graphical interface operation containing the operation type and operation parameters is generated. The operation type includes at least one of clicking, text input, or swiping, and the operation parameters include at least one of target coordinates, input string, or swiping trajectory.

[0010] In conjunction with the first aspect, in one implementation, after generating the corresponding graphical interface operation based on the voice command and before executing the graphical interface operation on the current page, the method further includes: The in-vehicle screen provides semantic and / or visual prompts for the interface elements that are about to be operated. If a confirmation instruction is received from the user, the graphical interface operation is executed; The confirmation instructions include: user's voice confirmation again, confirmation by touch in a specific area of ​​the screen, or no cancellation instruction from the user within a preset time.

[0011] In conjunction with the first aspect, in one implementation, before determining whether the current page supports the voice command based on the current page screenshot using a pre-trained graphical interface perception model, the method further includes: Collect multiple sets of training samples, each set of training samples including page screenshots, corresponding instruction text, and executable graphical interface operations; Based on the training samples, the graphical user interface perception model is trained through supervised learning to obtain a trained graphical user interface perception model.

[0012] In conjunction with the first aspect, in one implementation, the method further includes: If the current page does not support the voice command, the voice command will be parsed using a preset natural language understanding model, and a predefined default application will be called based on the parsing result to complete the response to the voice command.

[0013] Secondly, embodiments of this application provide an in-vehicle voice interaction device, the in-vehicle voice interaction device comprising: The acquisition module is used to receive voice commands input by the user and simultaneously acquire a screenshot of the current page of the vehicle screen; The judgment module is used to determine whether the current page supports the voice command based on the current page screenshot using a pre-trained graphical interface perception model. An operation module is used to generate a corresponding graphical interface operation based on the voice command, if supported, and to execute the graphical interface operation on the current page.

[0014] In conjunction with the second aspect, in one implementation, the determining module is further configured to: The screenshot of the current page is input into the graphical interface perception model to obtain the output list of operation instructions supported by the current page. The list of operation instructions contains one or more structured operation instructions, and each operation instruction is associated with an executable interface element identifier and a corresponding operation semantic. The voice commands are converted into command text, and the semantic information is extracted from them; The semantic information is semantically matched with the operation semantics of each operation instruction in the operation instruction list; If a matching operation command is found, it is determined that the current page supports the voice command; If no matching operation command is found, it is determined that the current page does not support the voice command.

[0015] Thirdly, embodiments of this application provide an in-vehicle voice interaction device, which includes a processor, a memory, and an in-vehicle voice interaction program stored in the memory and executable by the processor. When the in-vehicle voice interaction program is executed by the processor, it implements the steps of the in-vehicle voice interaction method as described in any of the above claims.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing an in-vehicle voice interaction program, wherein when the in-vehicle voice interaction program is executed by a processor, it implements the steps of the in-vehicle voice interaction method as described in any of the preceding claims.

[0017] The beneficial effects of the technical solutions provided in this application include: By receiving voice commands input by the user and simultaneously acquiring a screenshot of the current page on the in-vehicle screen, a pre-trained graphical interface perception model determines whether the current page supports the voice command based on the screenshot. If supported, a corresponding graphical interface operation is generated based on the voice command and executed on the current page. This achieves dynamic collaborative understanding of the user's voice commands and the current page state, avoiding the problem of traditional voice interaction being forced to jump to the default application or perform irrelevant operations due to skill binding. This significantly improves the accuracy, context adaptability, and user experience consistency of in-vehicle voice interaction. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an embodiment of the in-vehicle voice interaction method of this application; Figure 2 This is a schematic diagram of the functional modules of an embodiment of the in-vehicle voice interaction device of this application; Figure 3 This is a schematic diagram of the hardware structure of the in-vehicle voice interaction device involved in the embodiments of this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0020] First, some of the technical terms used in this application will be explained to help those skilled in the art understand this application.

[0021] GUI (Graphical User Interface): A graphical user interface is an interface through which users interact with electronic devices using graphical elements (such as windows, icons, buttons, menus, etc.).

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0023] In a first aspect, embodiments of this application provide an in-vehicle voice interaction method.

[0024] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the in-vehicle voice interaction method of this application. Figure 1 As shown, the in-vehicle voice interaction method includes: Step S101: Receive the voice command input by the user and simultaneously obtain a screenshot of the current page of the vehicle screen.

[0025] Specifically, the in-vehicle microphone array within the vehicle's voice interaction system can monitor in-vehicle voice signals in real time. Upon detecting a preset wake-up word or continuous voice activity, it begins collecting user-inputted voice commands. These commands can be natural language statements, such as "play the first song" or "search for People's Square." Within a preset time window after the voice command collection is complete, the in-vehicle operating system's screen capture function is invoked to obtain a screenshot of the currently displayed page on the in-vehicle screen. This screenshot is then stored in memory as visual input for subsequent graphical interface perception models.

[0026] Step S102: Based on the screenshot of the current page, determine whether the current page supports the voice command using a pre-trained graphical interface perception model.

[0027] In one embodiment, step S102 specifically includes: inputting the screenshot of the current page into the graphical interface perception model to obtain an output list of operation instructions supported by the current page, wherein the list of operation instructions contains one or more structured operation instructions, each operation instruction being associated with an executable interface element identifier and a corresponding operation semantic; converting the voice instruction into instruction text and extracting its semantic information; performing semantic matching between the semantic information and the operation semantics of each operation instruction in the list of operation instructions; if a successfully matched operation instruction exists, then it is determined that the current page supports the voice instruction; if no successfully matched operation instruction exists, then it is determined that the current page does not support the voice instruction.

[0028] For example, after obtaining a voice command, the voice command can first be converted into corresponding command text using a speech recognition function (e.g., converting the user's voice "search for People's Square" into text). Then, Natural Language Understanding (NLU) can be used to perform semantic parsing on the command text to extract structured semantic information. This semantic information includes the operation intent (e.g., performing a location search) and optional operation parameters (e.g., People's Square).

[0029] The graphical user interface (GUI) agent parses the screenshot of the current page, identifies all interactive areas on the interface and their functional semantics, and dynamically generates a list of operation instructions supported by the current page. The operation instruction list contains one or more structured operation instructions, each associated with an executable interface element identifier (such as the position of the search box, play button, volume slider, etc. in the image) and a corresponding operation semantic (such as intent labels such as search location, pause playback, adjust volume, etc.).

[0030] The semantic information of the voice command is semantically matched with the semantic information of each operation command in the operation command list. If a matching operation command exists, for example, if the semantic information of the voice command is "execute location search" and it matches the semantic information of the operation command "search location", then the current page is determined to support the voice command, and the matching operation command is recorded as the basis for generating subsequent graphical interface (GUI) actions. If no matching operation command exists, for example, if the semantic information of the user's voice command is "play the first song", but the current page is the vehicle settings menu, and its operation command list only includes "adjust the air conditioning temperature" and "open the windows", then the current page is determined to not support the voice command.

[0031] It is worth noting that the semantic matching process can be implemented by calculating semantic similarity based on pre-trained semantic vectors, matching through rule templates, or outputting matching results end-to-end through a graphical interface perception model.

[0032] Step S103: If supported, generate the corresponding graphical interface operation based on the voice command, and execute the graphical interface operation on the current page.

[0033] In one embodiment, generating a corresponding graphical interface operation based on the voice command includes: determining the operation command in the operation command list that successfully matches the semantic information as the target operation command; generating a graphical interface operation containing an operation type and operation parameters based on the semantic information, the target operation command, and the position information of the corresponding interface element in the current page screenshot; wherein the operation type includes at least one of clicking, text input, or swiping, and the operation parameters include at least one of target coordinates, input string, or swiping trajectory.

[0034] As an example, after determining that the current page supports voice commands, the operation command that successfully matches the semantic information of the semantic command is selected from the list of operation commands output by the graphical user interface perception model (GUI Agent) as the target operation command. For example, if the semantic information corresponding to the user's voice command is {intent: search location, parameter: People's Square}, and the operation command list contains the operation command "search location", then this operation command is determined as the target operation command.

[0035] Further, UI element detection is performed on the current page screenshot to locate interactive controls (such as input boxes, buttons, sliders, etc.) related to the target operation command, and their pixel coordinates in the current page screenshot are obtained as the position information of the interface elements.

[0036] Based on semantic information, target operation instructions, and the positional information of interface elements, the system generates specific graphical interface operations. These operations include operation types and operation parameters: the operation type includes at least one of clicking, text input, or swiping; the operation parameters include at least one of target coordinates (x, y), input string (such as "People's Square"), or swiping trajectory.

[0037] For example, when the semantic information contains the parameter "People's Square", the target operation instruction is "search location", and the coordinates of the search box and the search button in the page screenshot are (320, 150) and (580, 150) respectively, the operation sequence is generated as follows: click (320, 150) → enter the text "People's Square" → click (580, 150).

[0038] The generated graphical interface operations are passed to the simulated input module, which converts the graphical interface operations into standard input events and injects them into the current application process by calling the underlying interface of the operating system, thus realistically simulating user interaction behavior and realizing the response to user commands.

[0039] In one embodiment, before determining whether the current page supports voice commands based on a screenshot of the current page using a pre-trained graphical interface perception model, a step of training the graphical interface perception model is included. This training process specifically includes: Multiple training samples were collected. Each training sample included a screenshot of the in-vehicle screen, the corresponding instruction text, and the executable graphical interface operations performed on the interface corresponding to the screenshot. The screenshots were taken from real in-vehicle environments, covering typical application scenarios such as navigation, music, telephone, and vehicle settings. The instruction text was a natural language expression consistent with the semantics of the operation behavior. For example, the text corresponding to the user's command "Search People's Square" would be "Search People's Square". The executable graphical interface operations were recorded in the form of atomic-level sequences, such as: [Click (320,150), Enter text ("People's Square"), Click (580,150)]. Subsequently, based on the aforementioned training samples, a supervised learning approach can be adopted, using page screenshots and command text as model inputs and the high-level operational semantics corresponding to the executable operations (such as "search location" and "pause playback") as training targets to train the graphical interface perception model. After sufficient training, the resulting graphical interface perception model can output a dynamically generated list of operation commands based on any input page screenshots and command texts. This list contains structured operation commands that are executable in the current page context (each command is associated with interface element identifiers and operation semantics), used for subsequent matching with the semantic information of voice commands, and providing a basis for the generation of graphical interface operations, thereby achieving accurate and secure voice interaction control.

[0040] As a preferred implementation, after generating the corresponding graphical interface operation based on the voice command, and before executing the graphical interface operation on the current page, the method further includes: providing semantic and / or visual prompts for the interface element to be operated on the in-vehicle screen; if a user confirmation command is received, then executing the graphical interface operation; wherein, the confirmation command includes: user's re-voice confirmation, touch confirmation in a specific area of ​​the screen, or no cancellation command received from the user within a preset time. This mechanism can avoid accidental operation, especially improving interaction safety when the vehicle is in motion, and ensuring that the user retains ultimate control through prompts and confirmations, thereby enhancing the user experience and sense of trust.

[0041] In one embodiment, the method further includes: if the current page does not support the voice command, then parsing the voice command through a preset natural language understanding model, and calling a predefined default application based on the parsing result to complete the response to the voice command.

[0042] In one embodiment of the present invention, when the graphical interface perception model determines that the current page does not support user voice commands, the system automatically switches to a backup processing path. This path employs a traditional voice interaction mechanism: first, the voice command is sent to a preset Natural Language Understanding (NLU) model for parsing. The NLU model performs intent recognition and slot extraction on the command, for example, recognizing the intent "play music" and the song name "first song" from "play first song". Subsequently, based on the parsing results, the system calls a predefined default application bound to the intent. For example, the intent "play music" is associated with the first music app by default. Finally, the default application completes the response to the voice command, such as launching the application and playing the specified song. This fallback mechanism ensures that even if the GUI Agent cannot perform operations on the current page, user commands will not be ignored, guaranteeing the overall availability of the system.

[0043] This embodiment adopts a dual-mode architecture that prioritizes GUI and provides NLU fallback, enabling precise operation of the current page on the in-vehicle screen in supported scenarios, while reverting to a reliable traditional default solution when not supported, thus balancing intelligence and robustness.

[0044] The in-vehicle voice interaction method provided in this application has the following advantages: By jointly analyzing the current GUI page information and the semantics of voice commands, the system can prioritize responding to the user's operational intentions in real-world usage scenarios (such as executing commands in commonly used third-party applications), avoiding the reliance on the logic of default applications in traditional solutions, and making the command execution results more in line with user expectations. The GUI Agent module can perceive and understand the functional layout of the current page in real time, supporting refined operations directly in complex interfaces, breaking through the limitation of traditional voice systems that can only call fixed skills, and effectively covering more high-frequency and high-value vehicle usage scenarios. This solution adopts a dual-mode architecture of "GUI Action priority and traditional NLU skill backup". When the current page of the in-vehicle screen supports voice commands, it executes accurate operations on the spot; when it does not support them, it seamlessly reverts to the original default voice skill system. This design achieves intelligent upgrades while retaining the reliability of traditional solutions, avoiding command loss or execution failure. This method does not require large-scale reconstruction of existing in-vehicle voice systems, nor does it require customized interfaces from third-party applications. Capabilities can be upgraded simply by adding a GUI Agent module. It is compatible with existing system architectures, facilitates rapid deployment and continuous optimization, and can significantly reduce system integration and iteration costs.

[0045] Secondly, embodiments of this application also provide an in-vehicle voice interaction device.

[0046] In one embodiment, reference is made to Figure 2 , Figure 2 This is a functional module diagram of an embodiment of the in-vehicle voice interaction device of this application.Figure 2 As shown, the in-vehicle voice interaction device includes: The acquisition module is used to receive voice commands input by the user and simultaneously acquire a screenshot of the current page of the vehicle screen; The judgment module is used to determine whether the current page supports the voice command based on the current page screenshot using a pre-trained graphical interface perception model. An operation module is used to generate a corresponding graphical interface operation based on the voice command, if supported, and to execute the graphical interface operation on the current page.

[0047] Furthermore, in one embodiment, the determining module is further configured to: The screenshot of the current page is input into the graphical interface perception model to obtain the output list of operation instructions supported by the current page. The list of operation instructions contains one or more structured operation instructions, and each operation instruction is associated with an executable interface element identifier and a corresponding operation semantic. The voice commands are converted into command text, and the semantic information is extracted from them; The semantic information is semantically matched with the operation semantics of each operation instruction in the operation instruction list; If a matching operation command is found, it is determined that the current page supports the voice command; If no matching operation command is found, it is determined that the current page does not support the voice command.

[0048] Furthermore, in one embodiment, the operation module is also used for: The operation instruction that successfully matches the semantic information in the list of operation instructions is identified as the target operation instruction. Based on the semantic information, the target operation instruction, and the position information of the corresponding interface element in the current page screenshot, a graphical interface operation containing the operation type and operation parameters is generated. The operation type includes at least one of clicking, text input, or swiping, and the operation parameters include at least one of target coordinates, input string, or swiping trajectory.

[0049] Furthermore, in one embodiment, the operation module is also used for: The in-vehicle screen provides semantic and / or visual prompts for the interface elements that are about to be operated. If a confirmation instruction is received from the user, the graphical interface operation is executed; The confirmation instructions include: user's voice confirmation again, confirmation by touch in a specific area of ​​the screen, or no cancellation instruction from the user within a preset time.

[0050] Furthermore, in one embodiment, the device is also used for: Collect multiple sets of training samples. Each set of training samples includes a page screenshot, the instruction text corresponding to the page screenshot, and the executable graphical interface operation performed on the interface corresponding to the page screenshot. Based on the training samples, the page screenshots and the instruction text are used as model inputs, and the high-level operation semantics corresponding to the executable graphical interface operations are used as training targets to train the graphical interface perception model. The high-level operation semantics are used to construct a list of operation instructions supported by the current page, and the graphical interface perception model is configured to output the list of operation instructions based on the page screenshot during inference.

[0051] Furthermore, in one embodiment, the device is also used for: If the current page does not support the voice command, the voice command will be parsed using a preset natural language understanding model, and a predefined default application will be called based on the parsing result to complete the response to the voice command.

[0052] The functions of each module in the above-mentioned vehicle voice interaction device correspond to the steps in the above-mentioned vehicle voice interaction method embodiment, and their functions and implementation processes will not be described in detail here.

[0053] Thirdly, embodiments of this application provide an in-vehicle voice interaction device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0054] Reference Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of the in-vehicle voice interaction device involved in the embodiments of this application. In the embodiments of this application, the in-vehicle voice interaction device may include a processor, a memory, a communication interface, and a communication bus.

[0055] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0056] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the in-vehicle voice interaction device, as well as interfaces used for interconnecting the in-vehicle voice interaction device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0057] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0058] The processor can be a general-purpose processor, which can call the in-vehicle voice interaction program stored in the memory and execute the in-vehicle voice interaction method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the in-vehicle voice interaction program is called can be referred to in the various embodiments of the in-vehicle voice interaction method of this application, and will not be repeated here.

[0059] Those skilled in the art will understand that Figure 3 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0060] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0061] The present application provides a computer-readable storage medium storing an in-vehicle voice interaction program, wherein when the in-vehicle voice interaction program is executed by a processor, it implements the steps of the in-vehicle voice interaction method described above.

[0062] The method implemented when the in-vehicle voice interaction program is executed can be referred to in various embodiments of the in-vehicle voice interaction method of this application, and will not be repeated here.

[0063] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0064] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0065] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0066] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0067] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0069] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A vehicle-mounted voice interaction method, characterized in that, The in-vehicle voice interaction method includes: It receives voice commands input by the user and simultaneously captures a screenshot of the current page on the in-vehicle screen; Based on the current page screenshot, a pre-trained graphical interface perception model is used to determine whether the current page supports the voice command. If supported, a corresponding graphical interface operation is generated based on the voice command, and the graphical interface operation is executed on the current page.

2. The in-vehicle voice interaction method as described in claim 1, characterized in that, The step of using a pre-trained graphical user interface perception model to determine whether the current page supports the voice command based on the current page screenshot includes: The screenshot of the current page is input into the graphical interface perception model to obtain the output list of operation instructions supported by the current page. The list of operation instructions contains one or more structured operation instructions, and each operation instruction is associated with an executable interface element identifier and a corresponding operation semantic. The voice commands are converted into command text, and the semantic information is extracted from them; The semantic information is semantically matched with the operation semantics of each operation instruction in the operation instruction list; If a matching operation command is found, it is determined that the current page supports the voice command; If no matching operation command is found, it is determined that the current page does not support the voice command.

3. The in-vehicle voice interaction method as described in claim 2, characterized in that, The operation of generating the corresponding graphical interface based on the voice command includes: The operation instruction that successfully matches the semantic information in the list of operation instructions is identified as the target operation instruction. Based on the semantic information, the target operation instruction, and the position information of the corresponding interface element in the current page screenshot, a graphical interface operation containing the operation type and operation parameters is generated. The operation type includes at least one of clicking, text input, or swiping, and the operation parameters include at least one of target coordinates, input string, or swiping trajectory.

4. The in-vehicle voice interaction method as described in claim 1, characterized in that, After generating the corresponding graphical interface operation based on the voice command, and before executing the graphical interface operation on the current page, the method further includes: The in-vehicle screen provides semantic and / or visual prompts for the interface elements that are about to be operated. If a confirmation instruction is received from the user, the graphical interface operation is executed; The confirmation instructions include: user's voice confirmation again, confirmation by touch in a specific area of ​​the screen, or no cancellation instruction from the user within a preset time.

5. The in-vehicle voice interaction method as described in claim 1, characterized in that, Before determining whether the current page supports the voice command based on the current page screenshot using a pre-trained graphical interface perception model, the process also includes: Collect multiple sets of training samples. Each set of training samples includes a page screenshot, the instruction text corresponding to the page screenshot, and the executable graphical interface operation performed on the interface corresponding to the page screenshot. Based on the training samples, the page screenshots and the instruction text are used as model inputs, and the high-level operation semantics corresponding to the executable graphical interface operations are used as training targets to train the graphical interface perception model. The high-level operation semantics are used to construct a list of operation instructions supported by the current page, and the graphical interface perception model is configured to output the list of operation instructions based on the page screenshot during inference.

6. The in-vehicle voice interaction method as described in claim 1, characterized in that, The method also includes: If the current page does not support the voice command, the voice command will be parsed using a preset natural language understanding model, and a predefined default application will be called based on the parsing result to complete the response to the voice command.

7. A vehicle-mounted voice interaction device, characterized in that, The in-vehicle voice interaction device includes: The acquisition module is used to receive voice commands input by the user and simultaneously acquire a screenshot of the current page of the vehicle screen; The judgment module is used to determine whether the current page supports the voice command based on the current page screenshot using a pre-trained graphical interface perception model. An operation module is used to generate a corresponding graphical interface operation based on the voice command, if supported, and to execute the graphical interface operation on the current page.

8. The in-vehicle voice interaction device as described in claim 7, characterized in that, The judgment module is also used for: The screenshot of the current page is input into the graphical interface perception model to obtain the output list of operation instructions supported by the current page. The list of operation instructions contains one or more structured operation instructions, and each operation instruction is associated with an executable interface element identifier and a corresponding operation semantic. The semantic information of the voice command is semantically matched with each operation command in the operation command list; If a matching operation command is found, it is determined that the current page supports the voice command; If no matching operation command is found, it is determined that the current page does not support the voice command.

9. A vehicle-mounted voice interaction device, characterized in that, The in-vehicle voice interaction device includes a processor, a memory, and an in-vehicle voice interaction program stored in the memory and executable by the processor, wherein when the in-vehicle voice interaction program is executed by the processor, it implements the steps of the in-vehicle voice interaction method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an in-vehicle voice interaction program, wherein when the in-vehicle voice interaction program is executed by a processor, it implements the steps of the in-vehicle voice interaction method as described in any one of claims 1 to 6.

Images