Vehicle voice interaction method, device and equipment, vehicle and storage medium

Abstract

The invention relates to a vehicle voice interaction method, apparatus and device, a vehicle and a storage medium. The method comprises the steps of obtaining a user voice text and xml information of a current display page of a vehicle; according to the user voice text, the xml information and a preset cue word template, constructing cue words; the cue word is input into the visual language large model, and a control instruction for vehicle machine operation is obtained; and executing the control instruction. According to the method, the xml information in the cue word covers all the controls and the attribute description of the controls in the page, all the controls in the page can be operated, the operation is not limited to the operation on the character part in the page, and the scene of voice interaction is expanded. Besides, the cue word indicates the task instruction, the xml information of the page and the available control instruction, so that the visual language large model can more easily understand the intention of the user, and the interaction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicles, and in particular to a method, device, equipment, vehicle and storage medium for vehicle voice interaction. Background Art

[0002] With the rapid development of artificial intelligence and voice recognition technology, voice interaction technology has become an important part of automotive intelligence. This technology allows drivers to control various functions of the car, such as music playback, navigation, air conditioning adjustment, etc., through voice commands, without manual operation, reducing user distraction during driving and improving driving safety.

[0003] At present, if third-party applications want to support the "see and say" function, they generally provide relevant functional interfaces to achieve this scenario function. The implementation cost is high and the access cycle is long. To address this problem, existing technologies use image and text recognition technology to extract the text and its corresponding coordinates from page images, and then achieve the "see and say" effect by simulating click events.

[0004] However, the above method can only realize clicking a certain position in the displayed page, and cannot operate the area where no text is directly displayed in the page, and cannot understand the user's intention. For example, the user can play some videos on the page through voice control, but cannot search for the required video. Therefore, how to design a visible and audible solution that can understand the user's intention is an urgent problem to be solved. Summary of the invention

[0005] One of the purposes of the present invention is to provide a method for vehicle voice interaction to solve the problem in the prior art that operations cannot be performed on areas on a page where no text is displayed and the user's intention cannot be understood; a second purpose is to provide a vehicle voice interaction device; a third purpose is to provide an electronic device; a fourth purpose is to provide a vehicle; and a fifth purpose is to provide a storage cutoff.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for vehicle voice interaction, the method comprising:

[0008] Get the user's voice text and the XML information of the current display page of the vehicle computer;

[0009] Constructing prompt words according to the user voice text, the XML information and a preset prompt word template;

[0010] Input the prompt words into the visual language model to obtain control instructions for vehicle computer operation;

[0011] Execute the control instruction.

[0012] According to the above technical means, the XML information in the prompt word covers all the controls and control attribute descriptions in the page. Therefore, operations can be performed on all the controls in the page, rather than being limited to the text part of the page. In addition, the tasks and available instructions are indicated in the prompt word template, which makes it easier for the visual language model to understand the user's intention and improve the accuracy of the interaction.

[0013] Furthermore, the method further comprises:

[0014] After executing the control instruction, new XML information of a new display page of the vehicle computer is obtained;

[0015] Constructing an interactive prompt word according to the user voice text, the control instruction, the new XML information and a preset interactive prompt word template, wherein the interactive prompt word includes a control instruction of a historical operation;

[0016] Inputting the interactive prompt words into the visual language macromodel to obtain new control instructions for vehicle computer operation;

[0017] The new control instruction is executed until the new control instruction includes an identification of completing the user requirement.

[0018] According to the above technical means, based on the user's initial voice input, the big model and the car computer need to go through multiple interactions to reach the final goal, and all historical operations are input into the big model during the interaction process. Multiple rounds of interaction are automatically carried out, and users no longer need to issue commands through voice on each page, reducing the number of user inputs and improving user experience.

[0019] Furthermore, if the control instruction does not contain an identifier of an interactive control, the method further includes:

[0020] Obtaining a screenshot of the current display page of the vehicle computer;

[0021] Inputting the screenshot and the prompt word into the visual language macromodel to obtain new control instructions and interaction coordinates for vehicle computer operation;

[0022] The new control instruction is executed on the control of the interactive coordinates.

[0023] According to the above technical means, in some special cases, it is impossible to obtain the XML information of a certain control or page module. At this time, the large model cannot recognize the corresponding position and the operation fails. By sending the screenshot and prompt words to the large model and trying again, the defect of performing voice interaction only through XML is improved, and the accuracy of voice interaction operations is improved.

[0024] Further, constructing prompt words according to the user voice text, the XML information and a preset prompt word template includes:

[0025] The user voice text is written in the first position of the prompt word template, and the XML information is written in the second position of the prompt word template to obtain the prompt word; wherein the XML information includes the identifier of each control in the current display page of the vehicle computer and the attribute description of each control; the prompt word template includes at least one of the following control instructions indicating output: click operation, long press operation, input operation, sliding operation and completion operation.

[0026] According to the above technical means, the prompt word includes task instructions, XML information and usable instructions, so that the output instruction format is within a preset range, ensuring that the vehicle computer can execute the instruction.

[0027] Furthermore, the prompt word template also includes the package names of all application programs of the vehicle computer.

[0028] According to the above technical means, the initial prompt word includes the package names of all application programs, and responds to the user's instruction to open a certain application.

[0029] Further, obtain the XML information of the current display page of the vehicle computer, including:

[0030] Get the control set of the current display page of the vehicle computer;

[0031] For each control, use the control's text and description attributes as the content of the XML element, and use the control's interaction attributes as the tag of the XML element;

[0032] Assign index numbers to XML elements as identifiers of controls;

[0033] The XML information is obtained according to the content of the XML element, the tag of the XML element and the index number.

[0034] According to the above technical means, when constructing XML information, each XML element is indexed and numbered, which facilitates the execution of control instructions for a certain index number and improves execution efficiency and accuracy.

[0035] Furthermore, the method further comprises:

[0036] Determine whether the content of the xml element includes a preset blacklist word;

[0037] If the content of the XML element includes the blacklist vocabulary, the control corresponding to the XML element is closed and the XML information is regenerated.

[0038] According to the above technical means, by filtering blacklist words, it is possible to avoid advertisements in the page from blocking the page.

[0039] In a second aspect, the present invention further provides a vehicle voice interaction device, the device comprising:

[0040] The acquisition module is used to obtain the user's voice text and the XML information of the current display page of the vehicle computer;

[0041] A prompt word module, used for constructing prompt words according to the user voice text, the XML information and a preset prompt word template;

[0042] A large model module, used for inputting the prompt words into the visual language large model to obtain control instructions for vehicle computer operations;

[0043] The action simulation module is used to execute the control instruction.

[0044] further,

[0045] The acquisition module is also used to acquire new XML information of a new display page of the vehicle computer after executing the control instruction;

[0046] The prompt word module is also used to construct an interactive prompt word according to the user voice text, the control instruction, the new XML information and a preset interactive prompt word template, wherein the interactive prompt word includes the control instruction of the historical operation;

[0047] The large model module is also used to input the interactive prompt words into the visual language large model to obtain new control instructions for vehicle computer operation;

[0048] The action simulation module is also used to execute the new control instruction until the new control instruction includes an identification of completing the user's needs.

[0049] further,

[0050] The acquisition module is also used to obtain a screenshot of the current display page of the vehicle computer;

[0051] The large model module is also used to input the screenshot and the prompt word into the visual language large model to obtain new control instructions and interaction coordinates for vehicle computer operation;

[0052] The action simulation module is also used to execute the new control instruction on the control of the interactive coordinates.

[0053] Furthermore, the prompt word module is specifically used for:

[0054] The user voice text is written in the first position of the prompt word template, and the XML information is written in the second position of the prompt word template to obtain the prompt word; wherein the XML information includes the identifier of each control in the current display page of the vehicle computer and the attribute description of each control; the prompt word template includes at least one of the following control instructions indicating output: click operation, long press operation, input operation, sliding operation and completion operation.

[0055] Furthermore, the prompt word template also includes the package names of all application programs of the vehicle computer.

[0056] Furthermore, the acquisition module is specifically used for:

[0057] Get the control set of the current display page of the vehicle computer;

[0058] For each control, use the control's text and description attributes as the content of the XML element, and use the control's interaction attributes as the tag of the XML element;

[0059] Assign index numbers to XML elements as identifiers of controls;

[0060] The XML information is obtained according to the content of the XML element, the tag of the XML element and the index number.

[0061] Furthermore, the acquisition module is also used for:

[0062] Determine whether the content of the xml element includes a preset blacklist word;

[0063] If the content of the XML element includes the blacklist vocabulary, the control corresponding to the XML element is closed and the XML information is regenerated.

[0064] In a third aspect, the present invention further provides an electronic device, comprising: a memory, a processor;

[0065] The memory stores computer-executable instructions;

[0066] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method as described in any one of the first aspects.

[0067] In a fourth aspect, the present invention further provides a vehicle, comprising a vehicle computer, wherein the vehicle computer is used to execute the method as described in any one of the first aspects.

[0068] In a fifth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0069] In a sixth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any one of the first aspects above.

[0070] Beneficial effects of the present invention:

[0071] (1) The voice interaction method provided by the present invention does not require a third-party APP to provide a relevant functional interface, and voice interaction is achieved by simulating clicks in the car computer. In addition, the present invention covers all controls and control attribute descriptions in the page in the prompt words. Therefore, the control instructions generated according to the large model can perform corresponding operations on all controls in the page, rather than being limited to operating the text part of the page, which expands the scenario of voice interaction.

[0072] (2) Based on the user's initial voice input, the present invention can have multiple interactions between the large model and the vehicle computer, and input all historical operations into the large model during the interaction process. The large model and the vehicle computer automatically perform multiple rounds of interaction, and the user no longer needs to issue commands through voice on each page, which reduces the number of user inputs and improves the user experience.

[0073] (3) The present invention indicates task instructions, operable controls and available instructions in the prompt word template, which can make it easier for the visual language model to understand the user's intention, output instructions that meet the needs, and improve the accuracy of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0075] Figure 1 A schematic diagram of an application scenario provided by the present invention;

[0076] Figure 2 A flow chart of a first embodiment of a method for vehicle voice interaction provided by the present invention;

[0077] Figure 3 A flow chart of a second embodiment of the vehicle voice interaction method provided by the present invention;

[0078] Figure 4 A flowchart of a third embodiment of the vehicle voice interaction method provided by the present invention;

[0079] Figure 5 A schematic diagram of the architecture of the vehicle voice interaction system provided by the present invention;

[0080] Figure 6 A flowchart of the system execution provided by the present invention;

[0081] Figure 7 A schematic diagram of the prompt words provided by the present invention;

[0082] Figure 8 A schematic diagram of the correspondence between the current page and the XML information provided by the present invention;

[0083] Fig. 9 A schematic diagram of the structure of an embodiment of a vehicle voice interaction device provided by the present invention;

[0084] Fig.10 This is a schematic structural diagram of an electronic device provided by the present invention.

[0085] The above drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0086] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0087] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0088] See it and say it: refers to a voice interaction technology that allows users to control various vehicle functions such as navigation, music or air conditioning by speaking commands without using physical buttons or touch screens, thereby enhancing the convenience and safety of operation.

[0089] Currently, if a third-party APP wants to support the "see and speak" function, it generally provides relevant functional interfaces for the voice system to respond to relevant terms in order to achieve this function. Every time a new app is added, the application and the car system need to cooperate with each other, and the corresponding interface or SDK must be integrated to achieve the "see and speak" function. Therefore, the implementation cost is high and the access cycle is long.

[0090] To solve the above problems, the existing technology uses a screenshot of the page and combines it with the user's voice to extract the text in the picture and its corresponding coordinates, and then simulates the click event to achieve the effect of seeing and speaking. However, in the above method, only the position of the text in the picture can be operated, and other areas without text cannot be operated. And in this process, the user's semantics must be accurate, otherwise there may be a situation where the user's intention cannot be understood. In addition, the simulation operation through image recognition can only simulate simple click actions, and cannot accurately understand the anthropomorphic actions such as input and long press.

[0091] In view of the above problems, XML information is used instead of pictures. The XML information includes all controls in the current page and the attribute description of the controls. In order to improve the user's understanding of semantics, a large model is used to output the final control instructions. In order to increase the accuracy of the large model's understanding and output instructions, prompt words are constructed through prompt word templates and input into the large model. Prompt words include the current task description, available instructions, controls on the current page, attribute descriptions of controls, etc. In this way, the large model can output control instructions that can be executed by the car computer to meet the user's needs, achieving the effect of seeing is saying. This solution can perform operations on controls without text on the page, and the accuracy of the output instructions of the large model can be increased through prompt words.

[0092] Figure 1 A schematic diagram of an application scenario provided by the present invention, in Figure 1 In the scenario shown, the user inputs voice commands on the vehicle side, and the vehicle side uploads the aggregated data to the cloud server; the cloud server processes the data, inputs the processed information into the large model to obtain control commands, and sends the control commands to the vehicle side for execution.

[0093] In another application scenario, the big model is integrated into the vehicle. The user inputs voice commands on the vehicle side. The vehicle side inputs the summarized data information into the big model on the vehicle side, and the input control commands are executed on the vehicle side.

[0094] In another application scenario, the user inputs voice commands on the vehicle side, the vehicle side processes the aggregated data information, and sends the processed information to the cloud server; the cloud server inputs the processed information into the large model to obtain control instructions, and sends the control to the vehicle side for execution.

[0095] The vehicle voice interaction method provided by the present invention can be applied to any of the above scenarios.

[0096] The vehicle voice interaction method provided by the present invention is not only applicable to vehicles, but also to electronic devices such as mobile phones, tablets, and smart glasses.

[0097] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0098] Figure 2 A flow chart of a first embodiment of the vehicle voice interaction method provided by the present invention is shown in FIG. Figure 2 As shown, the following steps are included:

[0099] S101, obtaining the user's voice text and the XML information of the page currently displayed by the vehicle computer.

[0100] In this step, an app software based on the barrier-free mode is developed on the vehicle side. When the app software is running, it indicates that the user needs to use the see-and-speak function. When the software is started, the user inputs voice commands through voice interaction on the vehicle side, and the voice recognition model built into the APP software converts the voice into natural language text representation, that is, the user voice text is obtained.

[0101] When receiving the voice command, the XML information of the current display page of the vehicle is obtained. The XML information includes all the controls of the current display page of the vehicle and the attribute description of the controls. The format in the XML information is preset. For example, an editable control, the description attribute is search, and the interactive attribute is editable. Converting it to XML format is: a control with editable capabilities, the text of the control is search, and the control can be clicked and edited.

[0102] Optionally, in the XML information, each control is indexed to facilitate quick finding of the control to be operated later. For example, a control (3) with editable capability, the text of control (3) is search, and control (3) can be clicked and edited.

[0103] Optionally, in the XML information, index the interactive attributes of each control to facilitate the subsequent direct use of the index number to represent the operation instruction. For example, a control with editable capabilities, the text of the control is search, and the control can be clicked (4) and edited (5).

[0104] S102: construct prompt words according to the user voice text, XML information and a preset prompt word template.

[0105] In this step, in order to better enable the large model to understand user instructions and output instructions correctly, it is necessary to construct prompt words through prompt word templates. The prompt word template prompts at least one of the following control instructions that the user can output: click operation, long press operation, input operation, sliding operation and completion operation.

[0106] For example, the prompt word template is as follows:

[0107] You are an intelligent terminal assistant that helps users complete tasks by interacting with mobile applications. Given a task, determine the command to use next based on the XML code of the current application screen. If you have completed the given task, make sure to use the "finish" command. Make sure to remember to use the following commands: click, long-click, input, scroll, finish. The current user's command is "user voice text" and the XML control of the current program screen is: "xml information".

[0108] Write the user's voice text in the first position of the prompt word template, and write the XML information in the second position of the prompt word template to get the prompt word. Take the user's voice text as an example to open the xxx01 video, and the constructed prompt word is:

[0109] You are an intelligent terminal assistant that helps users complete tasks by interacting with mobile applications. Given a task, determine the command to use next based on the XML code of the current application screen. If you have completed the given task, make sure to use the "finish" command. Make sure to remember to use the following commands: click, long-click, input, scroll, finish. The current user's command is "Open xxx01 video", and the XML control of the current program screen is: "[ <button id＝"video"clickable＝"true"index＝"11"text＝"xxx01视频">,<button id="video2" clickable="true" index="12" text="xxx02 Video">].

[0110] In the above prompt, button id represents the control name, clickable represents whether it is clickable, index represents the index number, and text represents the text description of the component.

[0111] It should be noted that the process of constructing the prompt can be carried out on the vehicle side, or the xml information and voice text can be sent to the cloud server for the cloud server to construct.

[0112] S103. Input the prompt into the vision-language large model to obtain control instructions for operating the in-vehicle computer.

[0113] After obtaining the prompt, input the prompt into the vision-language large model. The vision-language large model can be pre-trained and stored locally in the vehicle (can be used offline), or it can be a common large model software on the market, or it can be constructed by the manufacturer itself through cloud training. The large model will output control instructions according to the user requirements, available instructions, and the current page xml information in the prompt. The control instructions are used to operate the current page of the in-vehicle computer. The control instructions include the control identifier to be operated and the specific instructions.

[0114] It should be noted that the control instructions can be multiple consecutive instructions, such as clicking on the search box, editing the content to xxx01 Video, and clicking on the search control. The control instructions can also be a single instruction.

[0115] S104. Execute the control instructions.

[0116] After receiving the control instructions, the in-vehicle computer executes the specific instructions on the control identified by the control identifier to meet the user requirements.

[0117] In a specific way, in the Android application, the accessibility service can be used to help simulate user operations such as clicking and inputting. Using this function, create an accessibility service. When receiving the control identifier to be operated and the actions to be performed, such as clicking, swiping, editing, etc., then obtain the specific control instance according to the control identifier, and finally call the accessibility service Action interface to simulate touch operations.

[0118] This embodiment provides a method for vehicle voice interaction, which includes obtaining user voice text and XML information of the current display page of the vehicle computer; constructing prompt words according to the user voice text, XML information and a preset prompt word template; inputting the prompt words into the visual language big model to obtain control instructions for the vehicle computer operation; and executing the control instructions. In this method, the XML information in the prompt words covers all the controls and control attribute descriptions in the page, and can perform operations on all controls in the page, rather than being limited to the text part in the operation page, thereby expanding the scenario of voice interaction. In addition, the prompt words indicate the task instructions, the XML information of the page, and the control instructions available, which can make it easier for the visual language big model to understand the user's intentions and improve the accuracy of the interaction.

[0119] In actual use by users, the first voice command issued is usually to open a certain software. For example, the user says to open a certain music software or video software. At this time, the initial prompt word template needs to include the package names of all application programs in the car computer to respond to the user's command to open a certain software.

[0120] To further increase applicability, the prompt word template includes not only the package name of each application, but also the functional description of each application. In this way, when users need to perform certain functions, they no longer need to ask to open a certain software, reducing the number of interactions. For example, if a user asks to play a certain song, based on the functional description and package name of each application in the prompt word template, the large model can accurately identify which software needs to be opened and issue a control instruction to open a certain package name program.

[0121] In the above example, the user said to play a song, but the control command only opened the music-related software, and it is necessary to further obtain the XML information of the currently displayed page before subsequent interaction. In some other scenarios, there may be multiple interactions to meet the user's final needs. The following is an example to introduce multiple interactive processes.

[0122] Figure 3 A flow chart of a second embodiment of the vehicle voice interaction method provided by the present invention is shown in FIG. Figure 3 As shown, the following steps are included:

[0123] S201: After executing a control instruction, determine whether the control instruction includes an identification of completing a user requirement.

[0124] If the control instruction includes an identification of completing the user's needs, the interaction with the user's voice instruction is terminated.

[0125] If the control instruction does not include an identification of completing the user's requirement, step S202 is executed.

[0126] S202: Obtain new XML information of a new display page of the vehicle computer.

[0127] The previous control instruction is a step instruction in the whole interactive process. A new page is opened based on the previous display page. Therefore, it is necessary to obtain new XML information of the new display page.

[0128] For example, the user's voice command is to play a song, the previous command opens the music software, and then obtains the XML information of the display page of the music software.

[0129] For another example, the user's instruction is "Send a hello message to a friend on WeChat", the first instruction output by the large model is to open the WeChat software; according to the controls in the current display page of the WeChat interface, the second instruction output by the large model is to open the address book control; according to the controls in the address book page displayed in the WeChat interface, the third instruction output by the large model is to open a friend's display page; according to the controls in the display page of a friend on WeChat, the fourth instruction output by the large model is to click the send message control to enter the chat box; according to the controls in the currently displayed chat box, the fifth instruction output by the large model is to enter a hello message; according to the controls in the chat box after entering your hello message, the sixth instruction output by the large model is to click the send control, and the sixth instruction includes an identifier of the user's need.

[0130] S203: constructing interactive prompt words according to the user voice text, the control instruction, the new XML information and the preset interactive prompt word template, wherein the interactive prompt words include the control instructions of the historical operations.

[0131] The exemplary embodiment of step S202 shows that multiple interaction processes are required to meet the needs of the end user. During multiple interaction processes, the visual language large model may forget the previous operation and purpose. Therefore, it is necessary to set interaction prompts. The interaction prompts are different from the previous prompts. The interaction prompts include control instructions for historical operations to enable the large model to understand what stage of the operation steps it is currently in.

[0132] The interactive prompt word template also includes a prompt model that outputs an identification of completing the user's needs after it considers that the instruction has been completed.

[0133] An exemplary interactive prompt word template is:

[0134] You are an intelligent terminal assistant that helps users complete tasks by interacting with mobile applications. Given a task, the previous UI operation, and the XML content of the current UI, your responsibility is to determine whether the task is completed by the previous operation, and if not, decide which UI element under the XML of the current UI can complete the instruction. If you have completed the given task, make sure to use the "finish" command.

[0135] The user's command is "user voice text".

[0136] The previous instruction operations are: control instruction 1, control instruction 2;

[0137] The XML control of the current program screen is: "XML information".

[0138] Based on the above description, which control's action should be executed next? Please return the control number to be executed and the action to be performed, such as click, long-click, input, scroll, finish.

[0139] The above is the interactive prompt word template. Fill in the user voice text, control instructions, and new XML information in the corresponding positions to get the interactive prompt words.

[0140] S204: Input the interactive prompt words into the visual language model to obtain new control instructions for vehicle computer operation.

[0141] The obtained interactive prompt words are input into the dialog box of the large model which is the same as the last prompt words, and new control instructions output by the large model are obtained.

[0142] S205: Execute the new control instruction until the new control instruction includes an identification of completing the user's requirement.

[0143] In the above example, the user requirement is identified by the word "finish". When it is detected that the control instruction includes finish, it means that the voice interaction is ended.

[0144] This embodiment provides a method for voice interaction. After executing a control instruction, if the control instruction does not include an identification of completing the user's needs, further interaction is required. According to the XML information of the new display page after executing the control instruction and the control instruction of the historical operation, a prompt word is generated and input into the large model again, and the interaction continues until the control instruction includes an identification of completing the user's needs. In this way, the large model and the application can interact multiple times according to the user's voice instruction, which increases the interaction capability and expands the scene of seeing and speaking. Now, compared with the existing technology, it is no longer limited to entering a voice instruction on each display page to enter the next step, that is, the user no longer needs to enter a voice once on each page, which improves the user experience.

[0145] In the actual application of this solution, some software has anti-scraping settings, and cannot obtain the XML information of some applications or modules. At this time, the large model may not be able to match the control identifier that needs to be executed on the page, and return the identifier of the interactive control that is not recognized. How the large model returns needs to be limited in the prompt word template or the interactive prompt word template. For example, the prompt word template includes: when the control identifier that needs to be executed cannot be recognized, return -2. In this case, screenshots are needed for auxiliary processing, which is introduced below with an embodiment.

[0146] Figure 4 The flowchart of the vehicle voice interaction method embodiment 3 provided by the present invention is as follows: Figure 4 As shown, the following steps are included:

[0147] S301. Obtain a screenshot of the current display page of the vehicle computer.

[0148] In this step, there is no interactive control mark in the control command, and a screenshot of the current display page of the vehicle computer is obtained at this time.

[0149] S302: Input the screenshot and prompt words into the visual language model to obtain new control instructions and interaction coordinates for vehicle computer operation.

[0150] The screenshot and the prompt word are input into the visual language model together. At this time, the prompt word needs to include the promotion model to process according to the screenshot. The model recognizes the screenshot and generates new control instructions according to the task instructions and output instructions in the prompt word. The model recognizes the text in the screenshot and the coordinates of the text in the screenshot. The model determines which text content is used to perform the corresponding operation and then generates a control instruction.

[0151] For example, in the video software, a screenshot of the video software is obtained. There is a corresponding text description under each video in the screenshot, and the word "search" is also in the search box. The user's instruction is to play Journey to the West. According to the text in the screenshot, it is determined whether the current page includes the Journey to the West video. If the Journey to the West text is not included, a first control instruction is generated, indicating the coordinates of clicking "search", editing the three words Journey to the West, and then clicking Enter (execute the search). A screenshot of the new page after the search is obtained. According to the screenshot after the search and the prompt word, the large model can continue to output the second control instruction, indicating to click the coordinates where a certain Journey to the West video is located.

[0152] S303: Execute new control instructions on the control of the interactive coordinates.

[0153] The input through the screenshot method does not mark the control identifier. Therefore, the control instructions output by the large model do not have the identifier of each control, but the control instructions and the interaction coordinates in the screenshot. The vehicle computer executes new control instructions for the control of the interaction coordinates based on the interaction coordinates.

[0154] By inputting the screenshots and prompt words into the visual language model together in the above manner, when the XML information of some applications or modules cannot be obtained, multiple interactions can be performed by supplementing the screenshots to meet user needs and improve user experience.

[0155] In the above embodiment, a screenshot is obtained and sent to the visual language large model only when there is no identification of an interactive control in the control instruction; a screenshot may also be sent to the large model each time a prompt word is sent, and the large model is indicated in the prompt word. If the interaction cannot be completed according to the XML information, a control instruction is attempted to be generated based on the text and coordinates in the screenshot.

[0156] In actual application, there may be ad pop-ups on the page, which will block part of the page content. The blocked content may be controls that require interactive operations, so the ad pop-ups need to be processed.

[0157] There are several ways to handle ad pop-ups:

[0158] In one implementation, the large model is prompted in the prompt word to first determine whether there is blacklist information in the XML information. The blacklist information includes the two words "advertisement" in the prompt word. When the two words "advertisement" appear, a control instruction to click the close control of the advertisement is generated. The generated control instruction is sent to the vehicle computer. The vehicle computer clicks the close control in the advertisement through the barrier-free model. If the control instruction does not include an identifier that completes the user's needs, it continues to obtain new XML information of the page. In this state, the advertisement has been closed. The subsequent interaction process is then continued according to the prompt word and the XML information.

[0159] In another implementation, the large model is prompted in the prompt word to first determine whether there is blacklist information in the text of the screenshot. The blacklist information includes the two words "advertisement" in the prompt word. When the two words "advertisement" appear, a control instruction for the interactive coordinates of clicking the close button of the advertisement is generated. The generated control instruction is sent to the vehicle computer, and the vehicle computer closes the advertisement by clicking the interactive coordinate position through the barrier-free model. If the control instruction does not include an identification of completing the user's needs, it will continue to obtain new xml information or screenshot information of the page. In this state, the advertisement has been closed. The subsequent interaction process will be continued according to the prompt word and xml information (or screenshot information). Identifying advertisements by identifying screenshot text can avoid the situation where the advertisement xml information cannot be obtained.

[0160] In another implementation, when the vehicle obtains the XML information, it directly determines whether the content of the XML element includes a preset blacklist word, such as an advertisement. If the content of the XML element includes a blacklist word, the vehicle closes the control corresponding to the XML element, for example, by setting the property of the control to "close" or hide the control. After closing the control corresponding to the blacklist, the XML information is regenerated.

[0161] Through the above method, the impact of advertising pop-ups on page interaction can be avoided and the accuracy of voice interaction can be improved.

[0162] Figure 5 The schematic diagram of the architecture of the vehicle voice interaction system provided by the present invention is as follows: Figure 5 As shown in the figure, the system architecture mainly includes terminal software, cloud platform, and visual language model.

[0163] The end side includes: socket client, barrier-free service, and command untils function. The above functions are integrated into the barrier-free mode APP, which mainly has four modules: voice command input, application page XML capture, screenshot, and action simulation.

[0164] 1. Voice command input module:

[0165] The user speaks the operation instructions for the current app interface through the microphone, such as: the user says "open xxx video" in the video application interface.

[0166] 2. Application page XML generation module:

[0167] After receiving the user's voice command, use the Android accessibility service to extract the screen layout information (NodeInfoMap), that is, the set of UI elements of all clickable or input operation components (NodeInfo: {className:android.widget.TextView; text:search; id:search; clickable:true}) on the current page. Map these UI elements to XML elements, where the text and description attributes of the UI are used as the content of the XML elements, and the interactive attributes of the UI are converted to corresponding XML tags.

[0168] For example, clickable UI elements are mapped to< / button> <button>Tags, editable UI elements are mapped to <input> Labels, scrollable UI elements are mapped to <scroll>Tags, etc.

[0169] Assign a unique index to each XML element, for example:< / scroll> < / button> <button id＝"search"clickable＝"true"index＝"28"text＝"搜索">.

[0170] Finally, all components are saved in a NodeInfoMap with index as key and component node as value.

[0171] 3. Screenshot module: The accessibility app will take a screenshot of the current application and send it to the cloud.

[0172] 4. Action simulation module: In Android applications, the barrier-free service can help simulate the user's click, input and other operations. Use this function to create an barrier-free service. When receiving the component index and the action to be performed such as click, scroll, input, etc. returned by the cloud server, get the specific component instance according to the index from the above NodeInfoMap, and finally call the barrier-free service Action interface to simulate the touch operation.

[0173] The cloud platform includes: speech ASR recognition model, responsible for speech recognition. Prompt prompt engineering, responsible for prompt word construction. Socket service, communicating with socket clients. Historical task memory, responsible for recording historical user tasks and historical operations. Application task list summary, responsible for maintaining all applications on the terminal side. Vector retrieval, responsible for finding the application that the user needs to open in the applications stored in the vector library.

[0174] The visual language model is used to perform matching tasks and generate task execution action scripts. The visual language model can perform image recognition and reasoning. Using the powerful understanding ability of the multimodal model, the user's operation intention is identified, and instructions are generated and sent to the terminal APP to implement complex operations such as clicking, inputting, and sliding.

[0175] Figure 6 The system execution flow chart provided by the present invention is as follows: Figure 6 As shown, the user inputs the voice-described command through voice. The interactive application, i.e., the application designed based on barrier-free service on the vehicle side, obtains the information in the current graphical user interface (GUI) and converts it into XML information and uploads it to the intelligent module (i.e., intelligent agent) of the cloud server. The interactive application is also responsible for executing control commands through barrier-free service.

[0176] The agent constructs natural language prompts based on historical operation information and corresponding GUI information, as well as preset prompt templates. The natural language prompts include: task prompts, current GUI status (i.e., XML information of the current page), historically executed actions, context prompts, and specified output formats. The prompts are input into the visual language model.

[0177] Example 1:

[0178] First, install the preset app on the client side and set it to the barrier-free mode. Then deploy the cloud service on the cloud server to communicate and interact with the client side app.

[0179] The user wakes up the client-side app through voice and issues a command to it, such as opening an app on the current screen.

[0180] When the client app receives the command, it first obtains the package list of all third-party apps installed on the current terminal through the accessibility service interface, and then sends the user command, package list, and current screenshot to the cloud server through the socket channel.

[0181] After receiving the user's instructions, package list, and current screenshot, the cloud server first converts the user's voice instructions into text through the voice recognition model. Then the user's instructions and package list are assembled into prompt words through the prompt project, and then sent to large model services such as GPT4 or Tongyi Qianwen. In this solution, there is no restriction on which large model to use. The powerful natural language understanding of the large model is used to identify the current user's intention. After successful recognition, the package name of the application that the user needs to open is returned. And the instruction to open the application is generated. Finally, it is sent to the terminal app through the socket channel.

[0182] The terminal receives the instruction to open the application and opens the corresponding APP software according to the package name.

[0183] If the big model cannot identify the corresponding package in the above process, resulting in a failed command, the above screenshot needs to be sent to the visual big model. The visual big model combines the current screenshot with the user's command description to finally determine which app the user wants to open, and then sends the coordinates of the app to be opened, and opens the corresponding app software through the simulated click capability of the barrier-free mode.

[0184] Example 2:

[0185] The user wakes up the client app through voice and has opened an app interface, such as a file management list. Figure 7 As shown in the screenshot in the lower right corner, the user issues a command such as "list all files in the DCIM directory".

[0186] When the client app receives the instruction, it will first use the Android accessibility service to extract the screen layout information (NodeInfoMap), and according to the strategy described above, save the collection of all clickable or input operation components of the current page (NodeInfo: {className:android.widget.TextView; text:DCIM; id:folders; clickable:true}) in the Map in memory. The key is the index number of the current element, and the value is the instance object of the current element. At the same time, the screen layout information is mapped to XML description information. Figure 7 This is shown in the XML description in the middle left.

[0187] Take a screenshot of the current interface of the third-party APP in barrier-free mode, and finally send the screenshot and the above XML description information to the cloud server through the socket service.

[0188] The cloud server receives the user's command and the XML description of the current screen, and assembles it into the following through the prompt word project: Figure 7 The prompt word shown. Figure 7 This is a schematic diagram of the prompt words provided by the present invention. Among them, the topmost box at the beginning is the overall task guidance, the middle box on the left shows the XML information representation of the UI elements of the current third-party APP interface layout, the bottom box on the left shows the output requirements for the data returned by the large model, and the bottom right box shows a screenshot of the interface where the current third-party APP is located.

[0189] After the prompt word project is assembled, the prompt word is sent to the big model service, and the big model service returns the instruction data in the required format according to the current prompt. The cloud service sends the returned instruction data to the terminal APP through the socket service.

[0190] After receiving the instruction, the client APP obtains the corresponding component instance from the above nodeMap according to the returned index, and then calls the component's simulated click, input, or slide methods to simulate click input and other operations. In this example, the corresponding action is click Action.

[0191] Through the above technology, the "see and speak" function can be realized without the support of third-party APP development interface or SDK, which simplifies the access process and reduces the cost and development cycle. This method develops an APP running in barrier-free mode to obtain XML information and screenshots of application pages in real time, interact with the cloud server through voice commands, and realize intelligent interactive operations. By using the powerful understanding ability of the multimodal large model, the user's operation intention is identified, and instructions are generated and issued to the terminal APP to realize complex operations such as clicking, inputting, and sliding.

[0192] The following is a diagram to introduce how to generate XML information from the current display page.

[0193] Figure 8 This is a schematic diagram of the correspondence between the current page and the XML information provided by the present invention. Figure 8 As shown,

[0194] Map all controls in the page to XML elements. According to the interactive properties of the controls, editable controls are editable elements (editable view), and non-editable controls are elements (view). The text and description attributes of the controls in the page are used as the content of the XML element (i.e. the content in text). Display all interactive properties of the control in each XML message. Each control may have multiple interactive properties, such as clickable, editable, etc. Number the interactive functions of all XML elements in turn, so that the large model only needs to feedback a number to find which operation is performed on which control. For example, if the control instruction returned by the large model is 5, then according to the XML and the diagram, click the collection control. With this method, the XML information in this format is easier for the large model to understand, and it can also reduce unnecessary text interactions in the control instructions.

[0195] Fig. 9 A schematic diagram of a vehicle voice interaction device embodiment provided by the present invention, such as Fig. 9 As shown, the vehicle voice interaction device 900 includes:

[0196] The acquisition module 911 is used to acquire the user's voice text and the XML information of the current display page of the vehicle computer;

[0197] The prompt word module 912 is used to construct a prompt word according to the user voice text, the XML information and a preset prompt word template;

[0198] The large model module 913 is used to input the prompt word into the visual language large model to obtain a control instruction for the vehicle computer operation;

[0199] The action simulation module 914 is used to execute the control instruction.

[0200] further,

[0201] The acquisition module 911 is also used to acquire new XML information of a new display page of the vehicle computer after executing the control instruction;

[0202] The prompt word module 912 is also used to construct an interactive prompt word according to the user voice text, the control instruction, the new XML information and a preset interactive prompt word template, wherein the interactive prompt word includes a control instruction of a historical operation;

[0203] The large model module 913 is also used to input the interactive prompt words into the visual language large model to obtain new control instructions for vehicle computer operation;

[0204] The action simulation module 914 is also used to execute the new control instruction until the new control instruction includes an identification of completing the user's requirement.

[0205] further,

[0206] The acquisition module 911 is also used to obtain a screenshot of the current display page of the vehicle computer;

[0207] The large model module 913 is also used to input the screenshot and the prompt word into the visual language large model to obtain new control instructions and interaction coordinates for vehicle computer operation;

[0208] The action simulation module 914 is also used to execute the new control instruction on the control of the interactive coordinates.

[0209] Furthermore, the prompt word module 912 is specifically used for:

[0210] The user voice text is written in the first position of the prompt word template, and the XML information is written in the second position of the prompt word template to obtain the prompt word; wherein the XML information includes the identifier of each control in the current display page of the vehicle computer and the attribute description of each control; the prompt word template includes at least one of the following control instructions indicating output: click operation, long press operation, input operation, sliding operation and completion operation.

[0211] Furthermore, the prompt word template also includes the package names of all application programs of the vehicle computer.

[0212] Furthermore, the acquisition module 911 is specifically used for:

[0213] Get the control set of the current display page of the vehicle computer;

[0214] For each control, use the control's text and description attributes as the content of the XML element, and use the control's interaction attributes as the tag of the XML element;

[0215] Assign index numbers to XML elements as identifiers of controls;

[0216] The XML information is obtained according to the content of the XML element, the tag of the XML element and the index number.

[0217] Furthermore, the acquisition module 911 is also used for:

[0218] Determine whether the content of the xml element includes a preset blacklist word;

[0219] If the content of the XML element includes the blacklist vocabulary, the control corresponding to the XML element is closed and the XML information is regenerated.

[0220] The vehicle voice interaction device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and this embodiment will not be described in detail here.

[0221] Fig.10 This is a schematic diagram of the structure of the electronic device provided by the present invention. The electronic device 50 can be a vehicle computer, a cloud server, or a mobile phone. Fig.10 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0222] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0223] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0224] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0225] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0226] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present invention is not limited to only one bus or one type of bus.

[0227] The present invention also provides a vehicle, the vehicle includes a vehicle computer, and the vehicle computer can implement the method described in the above embodiment.

[0228] The present invention also provides a computer program product, comprising a computer program, and the computer program implements the above method when executed by a processor.

[0229] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0230] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0231] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0232] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0233] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0235] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0236] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0237] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.< / button>

Claims

1. A vehicle voice interaction method, characterized in that: The method comprises: Get the user's voice text and the XML information of the current display page of the vehicle computer; Constructing prompt words according to the user voice text, the XML information and a preset prompt word template; Input the prompt words into the visual language model to obtain control instructions for vehicle computer operation; Execute the control instruction.

2. The method according to claim 1, characterized in that The method further comprises: After executing the control instruction, new XML information of a new display page of the vehicle computer is obtained; Constructing an interactive prompt word according to the user voice text, the control instruction, the new XML information and a preset interactive prompt word template, wherein the interactive prompt word includes a control instruction of a historical operation; Inputting the interactive prompt words into the visual language macromodel to obtain new control instructions for vehicle computer operation; The new control instruction is executed until the new control instruction includes an identification of completing the user requirement.

3. The method according to claim 1, characterized in that If the control instruction does not contain an identifier of an interactive control, the method further includes: Obtaining a screenshot of the current display page of the vehicle computer; Inputting the screenshot and the prompt word into the visual language macromodel to obtain new control instructions and interaction coordinates for vehicle computer operation; The new control instruction is executed on the control of the interactive coordinates.

4. The method according to any one of claims 1 to 3, characterized in that: The step of constructing a prompt word according to the user voice text, the XML information and a preset prompt word template includes: The user voice text is written in the first position of the prompt word template, and the XML information is written in the second position of the prompt word template to obtain the prompt word; wherein the XML information includes the identifier of each control in the current display page of the vehicle computer and the attribute description of each control; the prompt word template includes at least one of the following control instructions indicating output: click operation, long press operation, input operation, sliding operation and completion operation.

5. The method according to claim 4, characterized in that The prompt word template also includes the package names of all application programs of the vehicle computer.

6. The method according to any one of claims 1 to 3, characterized in that: Get the XML information of the current display page of the vehicle computer, including: Get the control set of the current display page of the vehicle computer; For each control, use the control's text and description attributes as the content of the XML element, and use the control's interaction attributes as the tag of the XML element; Assign index numbers to XML elements as identifiers of controls; The XML information is obtained according to the content of the XML element, the tag of the XML element and the index number.

7. The method according to claim 6, characterized in that The method further comprises: Determine whether the content of the xml element includes a preset blacklist word; If the content of the XML element includes the blacklist vocabulary, the control corresponding to the XML element is closed and the XML information is regenerated.

8. A vehicle voice interaction device, characterized in that: The device comprises: The acquisition module is used to obtain the user's voice text and the XML information of the current display page of the vehicle computer; A prompt word module, used for constructing prompt words according to the user voice text, the XML information and a preset prompt word template; A large model module, used for inputting the prompt words into the visual language large model to obtain control instructions for vehicle computer operations; The action simulation module is used to execute the control instruction.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A vehicle, characterized in that: It comprises a vehicle computer, and the vehicle computer is used to execute the method according to any one of claims 1-7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images