Voice control method and device

Abstract

The invention provides a voice control method and device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, image processing, natural language understanding, deep learning and the like, and can be applied to a what you see is what you speak voice interaction scene. A specific embodiment of the method comprises the following steps: acquiring voice information input by a user; converting the voice information into a corresponding text instruction; inputting the text instruction and the current interface into an image understanding model to obtain a control matched with the text instruction in the current interface; and executing a control interaction operation on the control matched with the text instruction to obtain an operation result. According to the embodiment, image understanding and control interaction can be enhanced by utilizing the image understanding model, so that a user can operate various controls on an interface more conveniently and efficiently.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, image processing, natural language understanding, deep learning and other technical fields, and can be applied to see-and-speak voice interaction scenarios. Background Art

[0002] With the development of operating systems, the user input method has gradually evolved from the earliest command-line operation to mouse visual operation, and then to touch screen operation. However, in some scenarios, it is inconvenient for users to touch the screen, input keyboard, mouse and other operations, such as driving, cooking, etc. Therefore, the "what you see is what you say function" has gradually taken shape. The what you see is what you say function allows users to speak any text they see on the screen directly to the computer in natural language. The what you see is what you say module obtains the user's intention based on speech recognition technology and natural language processing technology, and finds the most matching interface control for automatic operation.

[0003] At present, the What You See Is What You Say module needs to parse the text content of each control, configure the commands that each control needs to support, and other customized contents, and then parse the text content spoken by the user according to each customized command, and perform related operations according to the parsed commands. Summary of the invention

[0004] The embodiments of the present disclosure provide a voice control method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes a method for training an image understanding model, comprising: obtaining training samples and a pre-trained language model, wherein the training samples include sample text instructions, a sample interface, and a sample control, and the sample control is a control in the sample interface that matches the sample text instruction; taking the sample text instruction and the sample interface as input, and the sample control as output, and performing training based on the language model to obtain an image understanding model.

[0006] In the second aspect, an embodiment of the present disclosure proposes a voice control method, including: obtaining voice information input by a user; converting the voice information into corresponding text instructions; inputting the text instructions and the current interface into an image understanding model to obtain controls in the current interface that match the text instructions, wherein the image understanding model is trained using the method described in the first aspect; performing control interaction operations on the controls that match the text instructions to obtain operation results.

[0007] In the third aspect, an embodiment of the present disclosure proposes an image understanding model training device, comprising: an acquisition module, configured to acquire training samples and a pre-trained language model, wherein the training samples include sample text instructions, sample interfaces and sample controls, and the sample controls are controls in the sample interface that match the sample text instructions; a training module, configured to take the sample text instructions and sample interfaces as inputs and the sample controls as outputs, and to perform training based on the language model to obtain an image understanding model.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a voice control device, comprising: a first acquisition module, configured to acquire voice information input by a user; a conversion module, configured to convert the voice information into corresponding text instructions; an understanding module, configured to input the text instructions and the current interface into an image understanding model, and obtain controls in the current interface that match the text instructions, wherein the image understanding model is trained using the device described in the third aspect; and an operation module, configured to perform control interaction operations on the controls that match the text instructions, and obtain operation results.

[0009] In a fifth aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method described in any implementation manner in the first aspect, or execute a method described in any implementation manner in the second aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a method as described in any implementation manner in the first aspect, or to execute a method as described in any implementation manner in the second aspect.

[0011] In a seventh aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect, or implements the method described in any implementation manner in the second aspect.

[0012] The voice control method provided by the embodiment of the present disclosure can automatically operate any control on any page when using the what-you-see-is-speak function of the voice interaction system. By using the image understanding model, image understanding and control interaction can be enhanced, allowing users to operate various controls on the interface more conveniently and efficiently. In addition, by using the powerful text understanding ability of the image understanding model, more natural and accurate voice command conversion and text matching can be achieved, thereby improving the intelligence and flexibility of voice control.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, purposes and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:

[0015] Figure 1 is a flowchart of an embodiment of an image understanding model training method according to the present disclosure;

[0016] Figure 2 is a flowchart of another embodiment of the image understanding model training method according to the present disclosure;

[0017] Figure 3 It is a structural diagram of the image understanding model;

[0018] Figure 4 is a flow chart of an embodiment of a voice control method according to the present disclosure;

[0019] Figure 5 It is a schematic diagram of the mobile phone screen interface;

[0020] Figure 6 This is a schematic diagram of the in-vehicle central control screen interface;

[0021] Figure 7 is a flow chart of another embodiment of the voice control method according to the present disclosure;

[0022] Figure 8 is a scene graph that can implement the voice control method of the present disclosure;

[0023] Fig. 9 is a structural schematic diagram of an embodiment of an image understanding model training device according to the present disclosure;

[0024] Fig.10 is a structural schematic diagram of an embodiment of a voice control device according to the present disclosure;

[0025] Fig.11 It is a block diagram of an electronic device used to implement the voice control method or image understanding model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] Figure 1 A process 100 of an embodiment of an image understanding model training method according to the present disclosure is shown. The image understanding model training method comprises the following steps:

[0029] Step 101, obtaining training samples and a pre-trained language model.

[0030] In this embodiment, the execution subject of the image understanding model training method can obtain training samples and a pre-trained language model.

[0031] The training sample may include a sample text instruction, a sample interface, and a sample control. The sample interface may include multiple controls. The sample text instruction may be an instruction to operate a control in the sample interface. The sample control may be a control in the sample interface that matches the sample text instruction. For example, the sample interface may include a name text box control, a gender text box control, and an age text box control. If the sample text instruction is to fill in a name text box, then the sample control may be a name text box control in the sample interface.

[0032] Step 102, taking sample text instructions and sample interfaces as input, taking sample controls as output, training based on a language model, and obtaining an image understanding model.

[0033] In this embodiment, the above-mentioned execution entity can take sample text instructions and sample interfaces as input, and sample controls as output, and perform training based on a language model to obtain an image understanding model.

[0034] For example, sample text instructions and sample interfaces are input into the language model to obtain output controls; based on the sample controls and output controls, the loss is calculated; based on the loss, the parameters of the language model are adjusted until the model converges to obtain an image understanding model.

[0035] The image understanding model training method provided by the embodiment of the present disclosure can obtain the image understanding model by fine-tuning the pre-trained language model, thereby reducing the training cost of the image understanding model. In addition, the trained image understanding model can perform multimodal analysis and understanding of the input interface and identify various elements and controls in the interface. Introducing multimodality in the visible and speakable field, that is, introducing text and image information and combining it with the language model for training, directly gives the visible and speakable command information that needs to be executed.

[0036] Continue to refer Figure 2 , which shows a process 200 of another embodiment of the image understanding model training method according to the present disclosure. The image understanding model training method includes the following steps:

[0037] Step 201, obtaining training samples and a pre-trained language model.

[0038] In this embodiment, the specific operation of step 201 is already Figure 1 In the illustrated embodiment, step 101 is introduced in detail and will not be repeated here.

[0039] Step 202: input the sample text instruction and the sample interface into the language model, and add prompt information in front of the input sample text instruction to obtain an output control.

[0040] In this embodiment, the execution subject can input the sample text instruction and the sample interface into the language model, and add prompt information (prompt) in front of the sample text instruction to obtain the output control. Adding prompt in front of the sample text instruction can be used to distinguish tasks that can be seen and spoken.

[0041] Step 203, calculate the loss based on the sample control and the output control.

[0042] In this embodiment, the above-mentioned execution entity can calculate the loss based on the sample control and the output control.

[0043] Step 204: Based on the loss, adjust the parameters of the prompt information to obtain an image understanding model.

[0044] In this embodiment, the above-mentioned execution entity can adjust the parameters of the prompt information based on the loss until the model converges, thereby obtaining an image understanding model.

[0045] Here, the loss is calculated based on the output control and sample control of the model, and then the prompt parameters are adjusted until the loss is minimized and the model converges, and the image understanding model is obtained. In the field of visible and speakable, multimodality is introduced, that is, text and image information is introduced and combined into the language model for training, and the command information that needs to be executed in visible and speakable is directly given.

[0046] In some embodiments, an embedding layer can be added before the multi-head attention layer of the language model to encode the prompt. In this way, the sample text instruction is encoded using the language model, and the prompt is encoded using the embedding layer, so that the prompt is encoded separately in the model. Then, based on the loss, only the parameters of the prompt and embedding layers are adjusted, and the image understanding model can be obtained without adjusting the original parameters of the language model, which greatly reduces the training cost of the image understanding model.

[0047] Usually, an embedding layer is added before each multi-head attention layer of the language model to train the prompt and embedding layers. While keeping the original parameters of the language model unchanged, an additional task related to "see it, say it" is added. In terms of training objectives, the "see it, say it"-related instruction task is used to obtain the name of the control inside the interface, whether a click command or a swipe command is required, etc.

[0048] For ease of understanding, Figure 3 Figure 2 shows the structure of the image understanding model. Figure 3 As shown in the figure, the sample text instructions and sample interface are input into the language model, prompt is added in front of the sample text instructions, the prompt embedding layer is added to the language model, and the control name is output. The loss is calculated based on the controls output by the model and the controls marked on the sample interface, and then the parameters of prompt and prompt embedding layers are adjusted to obtain the image understanding model.

[0049] from Figure 2 It can be seen that Figure 1 Compared with the corresponding embodiment, the image understanding model training method in this embodiment highlights the training steps. Therefore, the solution described in this embodiment adds prompt in front of the sample text instruction and only adds a small number of parameters. During the training process, the parameters of the language model are kept unchanged, and only the parameters of prompt are adjusted, which reduces the workload of parameter adjustment and improves the efficiency of model training.

[0050] Further references Figure 4 , which shows a process 400 of an embodiment of a voice control method according to the present disclosure. The voice control method comprises the following steps:

[0051] Step 401: Acquire voice information input by the user.

[0052] In this embodiment, the execution subject of the voice control method can obtain voice information input by the user.

[0053] Here, a voice application is installed on the client to collect voice information input by the user. Generally, the user can tell the system the task or operation to be performed through voice input, such as "click the button on the left" or "fill in the name text box".

[0054] Step 402: Convert the voice information into corresponding text instructions.

[0055] In this embodiment, the above execution entity can convert the voice information into corresponding text instructions.

[0056] Among them, speech information can be converted into corresponding text instructions using speech recognition technology. ASR (Automatic Speech Recognition) is a commonly used speech recognition technology that can convert speech into text by performing acoustic modeling and language model matching on input speech signals.

[0057] Step 403: input the text instruction and the current interface into the image understanding model to obtain the controls in the current interface that match the text instruction.

[0058] In this embodiment, the above-mentioned execution subject can input the text instruction and the current interface into the image understanding model to obtain the control in the current interface that matches the text instruction.

[0059] Here, the current interface can be the interface currently displayed on the client's screen. In the code, each element is represented by an object, and the entire interface has an object in the code, representing the root node. All elements on the current page are nodes belonging to the root node, that is, child nodes of the root node. Figure 5 A schematic diagram of a mobile phone screen interface is shown.

[0060] [Discover Nearby], [Favorites], [Connectivity], [Group Travel], [Offline Map], [Event Reporting], [Settings] and [About Us] are all elements on the interface.

[0061] Figure 6 The following figure shows the schematic diagram of the in-vehicle central control screen interface. Among them, [Movies], [Entertainment],

[0062] [TV Series], [Children], [Games] and [Sports] are all elements on the interface. In addition, there are sliding elements on the interface, which makes the interface slidable.

[0063] Among them, the image understanding model can be obtained by fine-tuning the language model, and can perform multimodal analysis and understanding of the input text instructions and the current interface, and identify the controls in the current interface that match the text instructions. Specifically, the image understanding model can use object detection technology to detect and identify objects and controls in the current interface; use interface analysis technology to parse and understand the layout of elements and controls in the current interface; and use semantic analysis technology to understand and explain the meaning and function of text and symbols in the current interface.

[0064] It should be noted that the image understanding model can be used Figure 1 or Figure 2 The method shown in the figure is used for training and will not be described here.

[0065] Step 404: perform a control interaction operation on the control that matches the text instruction to obtain an operation result.

[0066] In this embodiment, the execution subject can perform a control interaction operation on the control that matches the text instruction to obtain an operation result.

[0067] Here, the corresponding control interaction operations are performed according to the matching results, such as clicking, inputting, sliding, selecting, etc. In addition, automation tools and simulators can be used to simulate the user's operation of controls and execution of tasks to achieve automation and efficiency. In addition, the results of the executed operations can be fed back to the user, such as displaying a success or failure message, or updating the corresponding control status on the interface. In addition, if the task execution fails, the user can be prompted with possible reasons or solutions.

[0068] The voice control method provided by the embodiment of the present disclosure can automatically operate any control on any page when using the what-you-see-is-speak function of the voice interaction system. By using the image understanding model, image understanding and control interaction can be enhanced, allowing users to operate various controls on the interface more conveniently and efficiently. In addition, by using the powerful text understanding ability of the image understanding model, more natural and accurate voice command conversion and text matching can be achieved, thereby improving the intelligence and flexibility of voice control.

[0069] Further references Figure 7 , which shows a process 700 of another embodiment of the voice control method according to the present disclosure. The voice control method includes the following steps:

[0070] Step 701, in response to the voice application being started, obtaining a child node corresponding to an element in the current interface.

[0071] In this embodiment, when the voice application is started, the execution subject may obtain the child node corresponding to the element in the current interface.

[0072] Here, a voice application is installed on the client. After the voice application is started, it will apply for permission to open the accessibility service. After the user authorizes the voice application to obtain the accessibility service permission, the voice accessibility service (VoiceAccessibilityService) inherited from the accessibility service (AccessibilityService) will be started. In this way, the client can capture all the elements of the current interface and the corresponding child node information.

[0073] Here, the current interface can be the interface currently displayed on the client's screen. In the code, each element is represented by an object, and the entire interface has an object in the code, representing the root node. All elements on the current page are nodes belonging to the root node, that is, child nodes of the root node.

[0074] Since the nodes of the current interface adopt a tree structure, first obtain the root node corresponding to the current interface, and then obtain the child nodes contained in the root node through recursive traversal to improve the efficiency of node information acquisition. Usually, the getRootInActiveWindow() method can be used to obtain the AccessibilityNodeInfo information of the root node of the current interface. Then, the information of all child nodes contained in the root node can be obtained through recursive traversal.

[0075] Step 702: Associating the text information of the child node with the corresponding control in the current interface.

[0076] In this embodiment, the execution subject may associate the text information of the sub-node with the corresponding control in the current interface, thereby completing the element registration. The text information may include text description information and information about the area where the text is located.

[0077] Here, the text information of the sub-node corresponding to the element in the current interface is associated with the corresponding control in the current interface, and the text information is analyzed using the image understanding model, which helps to analyze the associated controls and improve the accuracy of control understanding.

[0078] Step 703: Acquire the voice information input by the user.

[0079] Step 704: convert the voice information into corresponding text instructions.

[0080] Step 705: input the text instruction and the current interface into the image understanding model to obtain the controls in the current interface that match the text instruction.

[0081] Step 706: perform a control interaction operation on the control that matches the text instruction to obtain an operation result.

[0082] In this embodiment, the specific operations of steps 703-706 are already in Figure 4 In the illustrated embodiment, steps 401 - 404 are described in detail and will not be repeated here.

[0083] from Figure 7 It can be seen that Figure 4 Compared with the corresponding embodiment, the voice control method in this embodiment adds an element registration step. Therefore, the scheme described in this embodiment completes the element registration by associating the sub-nodes corresponding to the elements in the current interface with the corresponding controls, thereby improving development efficiency and avoiding missed registrations. In addition, associating the text information of the sub-nodes corresponding to the elements in the current interface with the corresponding controls in the current interface, and analyzing the text information using an image understanding model helps to analyze the associated controls and improve the accuracy of control understanding.

[0084] For ease of understanding, Figure 8 A scene diagram is shown in which the voice control method of the present disclosure can be implemented.

[0085] Step 801, start accessibility service.

[0086] After the voice application is started, it will apply for permission to open the accessibility service. After the user authorizes the voice application to obtain the accessibility service permission, the VoiceAccessibilityService accessibility service inherited from the AccessibilityService implementation will be started.

[0087] Step 802: Get the root node of the current active window view.

[0088] The getRootInActiveWindow() method can be used to obtain the AccessibilityNodeInfo information of the root node of the current interface.

[0089] Step 803, recursively traverse all child nodes under the root node.

[0090] Since the nodes of the current active window view adopt a tree structure, the information of all child nodes contained in the root node can be obtained by recursive traversal.

[0091] Step 804, obtaining text description information and text area information of all child nodes.

[0092] Step 805: Add a specific label to the sliding control.

[0093] Based on the text description information of the child node and the information of the area where the text is located, a specific label is added to the sliding control, and the element can be associated with the sliding control to complete the element registration.

[0094] Step 806, audio input.

[0095] The client is installed with a voice application that can collect voice information input by the user. Usually, the user can tell the system the task or operation to be performed through voice input, such as "swipe left one page".

[0096] Step 807, speech recognition.

[0097] Step 808, obtaining ASR results.

[0098] By using ASR to perform speech recognition on the voice information input by the user, the corresponding text instructions can be obtained.

[0099] Step 809, parsing the sliding command.

[0100] Step 810, analyzing the sliding direction and sliding step length.

[0101] Step 811, whether the sliding control is hit.

[0102] The image understanding model is used to determine whether the sliding control is hit. If the sliding control is hit, step 812 is executed; if the sliding control is not hit, step 814 is executed.

[0103] Step 812, according to the hit sliding control, return the hit control ID, sliding direction and sliding step length.

[0104] Step 813, executing the associated behavior according to the hit sliding control.

[0105] Step 814, voice broadcast TTS (Text To Speech), I did not understand.

[0106] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image understanding model training device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0107] like Fig. 9 As shown, the image understanding model training device 900 of this embodiment may include: an acquisition module 901 and a training module 902. The acquisition module 901 is configured to acquire training samples and a pre-trained language model, wherein the training samples include sample text instructions, sample interfaces and sample controls, and the sample controls are controls in the sample interfaces that match the sample text instructions; the training module 902 is configured to take the sample text instructions and the sample interfaces as inputs, take the sample controls as outputs, perform training based on the language model, and obtain an image understanding model.

[0108] In this embodiment, in the image understanding model training device 900, the specific processing of the acquisition module 901 and the training module 902 and the technical effects thereof can be referred to in Figure 1 The relevant descriptions of steps 101 - 102 in the corresponding embodiment are not repeated here.

[0109] In some optional implementations of the present embodiment, the training module 902 includes: an understanding submodule, configured to input sample text instructions and a sample interface into a language model, and to add prompt information in front of the input sample text instructions to obtain an output control; a calculation submodule, configured to calculate the loss based on the sample control and the output control; and an adjustment submodule, configured to adjust the parameters of the prompt information based on the loss to obtain an image understanding model.

[0110] In some optional implementations of this embodiment, an embedding layer is added in front of the multi-head attention layer of the language model to encode the prompt information, and the adjustment submodule is further configured to: based on the loss, adjust the parameters of the prompt information and the embedding layer to obtain an image understanding model.

[0111] Further references Fig.10 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a voice control device, and the device embodiment is Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0112] like Fig.10 As shown, the voice control device 1000 of this embodiment may include: a first acquisition module 1001, a conversion module 1002, a comprehension module 1003 and an operation module 1004. The first acquisition module 1001 is configured to acquire the voice information input by the user; the conversion module 1002 is configured to convert the voice information into corresponding text instructions; the comprehension module 1003 is configured to input the text instructions and the current interface into the image comprehension model to obtain the controls in the current interface that match the text instructions, wherein the image comprehension model is a Fig. 9 The device shown is trained; the operation module 1004 is configured to perform control interaction operations on controls that match text instructions to obtain operation results.

[0113] In this embodiment, in the voice control device 1000, the specific processing of the first acquisition module 1001, the conversion module 1002, the understanding module 1003 and the operation module 1004 and the technical effects thereof can be referred to respectively. Figure 4 The relevant descriptions of steps 401-404 in the corresponding embodiment are not repeated here.

[0114] In some optional implementations of this embodiment, the voice control device 1000 also includes: a second acquisition module, configured to acquire a sub-node corresponding to an element in the current interface in response to the start of a voice application; and an association module, configured to associate the text information of the sub-node with the corresponding control in the current interface.

[0115] In some optional implementations of this embodiment, the second acquisition module is further configured to: acquire the root node corresponding to the current interface; and acquire the child nodes contained in the root node by recursive traversal.

[0116] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0117] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0118] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0119] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0120] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0121] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the voice control method. For example, in some embodiments, the voice control method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the voice control method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the voice control method in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0127] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training an image understanding model, comprising: Acquire a training sample and a pre-trained language model, wherein the training sample includes a sample text instruction, a sample interface, and a sample control, and the sample control is a control in the sample interface that matches the sample text instruction; The sample text instruction and the sample interface are taken as input, the sample control is taken as output, and training is performed based on the language model to obtain an image understanding model.

2. The method according to claim 1, wherein: The method of taking the sample text instruction and the sample interface as input, taking the sample control as output, and training based on the language model to obtain an image understanding model includes: Inputting the sample text instruction and the sample interface into the language model, and adding prompt information before the input sample text instruction to obtain an output control; Calculating a loss based on the sample control and the output control; Based on the loss, the parameters of the prompt information are adjusted to obtain an image understanding model.

3. The method according to claim 2, wherein: Adding an embedding layer before the multi-head attention layer of the language model to encode the prompt information, and The step of adjusting the content of the prompt information based on the loss to obtain an image understanding model includes: Based on the loss, the prompt information and parameters of the embedding layer are adjusted to obtain an image understanding model.

4. A voice control method, comprising: Get the voice information input by the user; Converting the voice information into corresponding text instructions; Inputting the text instruction and the current interface into an image understanding model to obtain a control in the current interface that matches the text instruction, wherein the image understanding model is trained using the method of any one of claims 1 to 3; A control interaction operation is performed on the control matching the text instruction to obtain an operation result.

5. The method according to claim 4, wherein: The method of obtaining the voice information input by the user further includes: In response to the voice application being started, obtaining a child node corresponding to an element in the current interface; The text information of the sub-node is associated with the corresponding control in the current interface.

6. The method according to claim 5, wherein: The obtaining of the child node corresponding to the element in the current interface includes: Get the root node corresponding to the current interface; The child nodes contained in the root node are obtained by recursive traversal.

7. An image understanding model training device, comprising: An acquisition module is configured to acquire a training sample and a pre-trained language model, wherein the training sample includes a sample text instruction, a sample interface, and a sample control, and the sample control is a control in the sample interface that matches the sample text instruction; The training module is configured to take the sample text instructions and the sample interface as input, take the sample control as output, perform training based on the language model, and obtain an image understanding model.

8. The method according to claim 7, wherein: The training module includes: The understanding submodule is configured to input the sample text instruction and the sample interface into the language model, and add prompt information in front of the input sample text instruction to obtain an output control; A calculation submodule, configured to calculate a loss based on the sample control and the output control; The adjustment submodule is configured to adjust the parameters of the prompt information based on the loss to obtain an image understanding model.

9. The device according to claim 8, wherein: Adding an embedding layer before the multi-head attention layer of the language model to encode the prompt information, and The adjustment submodule is further configured to: Based on the loss, the prompt information and parameters of the embedding layer are adjusted to obtain an image understanding model.

10. A voice control device, comprising: A first acquisition module is configured to acquire voice information input by a user; A conversion module, configured to convert the voice information into corresponding text instructions; A comprehension module, configured to input the text instruction and the current interface into an image comprehension model to obtain a control in the current interface that matches the text instruction, wherein the image comprehension model is trained using the device described in any one of claims 7 to 9; The operation module is configured to perform a control interaction operation on the control matching the text instruction to obtain an operation result.

11. The device according to claim 10, wherein: The device also includes: A second acquisition module is configured to acquire a subnode corresponding to an element in the current interface in response to the voice application being started; The association module is configured to associate the text information of the sub-node with the corresponding control in the current interface.

12. The device according to claim 11, wherein The second acquisition module is further configured to: Get the root node corresponding to the current interface; The child nodes contained in the root node are obtained by recursive traversal.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-3, or execute the method of any one of claims 4-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method of any one of claims 1 to 3, or to execute the method of any one of claims 4 to 6.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 3, or executes the method according to any one of claims 4 to 6.

Images