Interaction method and device of user interface and storage medium

Abstract

Along with the development of the Internet technology, application programs change rapidly, and people are busy more and more. At ordinary times, when a computer or other display devices are operated, there is no time or energy to use a mouse or a specific input device (such as a keyboard) to carry out data input and operation. For example, people with physical disabilities cannot use the mouse and can normally operate equipment only through special hardware. Therefore, the problem that the input method is not abundant and convenient enough exists in the prior art. In order to solve the problems in the prior art, the invention creatively provides an interaction method and device of a user interface and storage equipment, and aims to solve the problem that an input method is not rich and convenient enough in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a user interface interaction method, device and storage medium. Background Art

[0002] With the development of Internet technology, applications are changing with each passing day, and people are getting busier and busier. When operating a computer or other display device, people may not have the time or energy to use a mouse or a specific input device (such as a keyboard) to input data and operate.

[0003] For another example, people with physical disabilities cannot use a mouse and can only operate the device normally through special hardware.

[0004] Therefore, there is a problem in the prior art that the input methods are not rich and convenient enough. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present application creatively proposes a user interface interaction method, apparatus and storage device to solve the problem that the input methods in the prior art are not rich and convenient enough.

[0006] In a first aspect, a user interface interaction method is provided, comprising the following steps: Step S101: displaying a plurality of objects, each object having an identifier that can be described by a natural language; Step S102: receiving a text instruction in natural language; Step S103: parsing the text instruction through natural language understanding to determine the identifier of the operation verb and the operation object; Step S104: performing corresponding operations according to the operation verb, the type attribute and / or the position information of the operation object.

[0007] In a second aspect, based on the first aspect, the display object is a visual element on or near the visual element of the user interface itself.

[0008] In a third aspect, based on the first aspect, the identifier includes at least one of text, color, and pattern.

[0009] In a fourth aspect, based on the first aspect, the text instruction may include multiple operation objects, and the method performs the operation represented by the operation verb on each operation object.

[0010] In a fifth aspect, based on the first aspect, the text instruction may include multiple operation steps, and the method executes each operation step sequentially or randomly.

[0011] In a sixth aspect, based on the first aspect, the method further includes defining macro instructions, wherein the macro instructions include a series of predefined operations, and the execution is triggered by a single instruction.

[0012] In a seventh aspect, based on the first aspect, the object has owner information, and the method only allows the user to operate the objects that belong to him / her.

[0013] In an eighth aspect, based on the first aspect, the receiving step includes converting the user's other modal input into text instructions through voice recognition.

[0014] In a ninth aspect, based on the first aspect, the method supports multiple users, each user has his own set of objects, and the user can only operate his own objects.

[0015] In the tenth aspect, based on the first aspect, there is a position association between the multiple objects.

[0016] In an eleventh aspect, based on the first aspect, at least some objects have one or more additional identifiers in their vicinity, and the user can specify operations to be performed on these additional identifiers through text instructions.

[0017] In a twelfth aspect, based on the first aspect, the operation object may be a relative position based on a displayed identifier.

[0018] In the thirteenth aspect, based on the first aspect, one or more operation objects are used as reference points to indirectly operate other one or more operation objects.

[0019] In a fourteenth aspect, a user interface interaction device is provided, comprising: A display unit, for displaying a plurality of objects, each object having an identifier that can be described by a natural language, and a type attribute and / or position information; An input unit for receiving text instructions using natural language; A processing unit configured to parse the text instruction through natural language understanding to determine an operation verb and an operation object; The execution unit is used to execute the corresponding operation according to the operation verb, the type attribute and / or the position information of the operation object.

[0020] In the fifteenth aspect, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, one or more methods as described in the first to thirteenth aspects of the present application are implemented.

[0021] The present application provides a user interface interaction method and device, which enable a user to interact with the user interface through text instructions through natural language processing technology.

[0022] By using the technical solution described in this application, one or more of the following beneficial effects can be achieved.

[0023] First, it improves the flexibility and efficiency of user interaction: users can input instructions through natural language without having to remember complex commands or perform tedious mouse operations, thus completing tasks faster.

[0024] Second, it enhances the accessibility of the user interface: For visually impaired users or users who are not familiar with graphical user interfaces, this text-command-based interaction method provides a more convenient access path.

[0025] Third, it supports multi-object and multi-step operations: users can specify multiple operation objects or a series of operation steps at one time, reducing the complexity and time of the operation.

[0026] Fourth, by introducing the macro instruction function, users can define and use macro instructions to combine a series of operations into one instruction, further improving operational efficiency.

[0027] Fifth, by considering the ownership and location association of objects, in a multi-user environment, it ensures that users can only operate their own objects and can perform operations based on the location relationship of the objects, thereby enhancing the accuracy, convenience and security of the interaction.

[0028] Sixth, support voice recognition: voice input is converted into text instructions through voice recognition, making the interaction more natural and intuitive, especially suitable for scenarios where typing is inconvenient.

[0029] Seventh, enrich the object identification methods: By using identifiers such as numbers, colors, letters, and identification based on relative positions, the accuracy and flexibility of object recognition are improved.

[0030] In summary, the present invention provides a more flexible, efficient and easy-to-use interaction method and device by combining natural language processing and user interface design, which is suitable for a variety of application scenarios, especially environments that require frequent interaction with the user interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only schematic introductions or descriptions of the invention content or embodiments recorded in this application. For ordinary technicians in this field, other drawings can be easily obtained based on these drawings.

[0032] It should be noted that the text, shapes, patterns, or designs in all the drawings are only for illustration purposes and may be presented differently in specific embodiments. For example, if the words "Button 1" appear in a wireframe in the figure, it only means that there are button elements in the area, and in the specific implementation process, its features may be a specific color, specific text, specific pattern, or a combination of one or more specific patterns.

[0033] Attached Figure 1 This is a schematic diagram of Embodiment 1 of the present application.

[0034] Attached Figure 2 This is a schematic diagram of the second embodiment of the present application.

[0035] Attached Figure 3 This is a schematic diagram of Embodiment 3 of the present application.

[0036] Attached Figure 4 This is a schematic diagram of Embodiment 4 of the present application.

[0037] Attached Figure 5 This is a schematic diagram of Embodiment 5 of the present application.

[0038] Attached Figure 6 This is a schematic diagram of Example 6 of the present application.

[0039] Attached Figure 7 This is a schematic diagram of the twelfth embodiment of the present application.

[0040] Attached Figure 8 This is a schematic diagram of the fourteenth embodiment of the present application.

[0041] Attached Fig. 9 This is a schematic diagram of the sixteenth embodiment of the present application.

[0042] Attached Fig.10 The present invention is a flowchart of the method described in the present application.

[0043] Attached Fig.11 is a block diagram of the device described in this application.

[0044] Attached Fig.12 It is a conceptual schematic diagram of a data processing device, which can generally be used to use the storage device described in this application or use the method described in this application to implement the device described in this application. DETAILED DESCRIPTION

[0045] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification.

[0046] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0047] There are many ways to receive the "text instructions using natural language" mentioned in this application. Depending on the specific situation, place of use, method of use, and implementation method, it can be through the direct reception of user input text at the implementer's local site, or natural language text received through communication media such as the Internet, or preset text read from a storage device file, or even text data generated in real time or in advance through generative artificial intelligence (such as a large language model). The "text instructions" described in all embodiments of this application are schematic in nature. In actual use, there may be completely different ways of expression, or the use of punctuation marks, or even no punctuation marks at all. The description of "user said" mentioned in this application means "receiving text instructions using natural language from the user."

[0048] The received "text instructions in natural language" mentioned in the present application may also be parsed or identified by other modal information received locally by the implementer or through other communication media such as the Internet.

[0049] For example, if a sound signal is received, the specific text command can be recognized through voice recognition technology (for example, the Whisper open source voice recognition model can be used). Since voice recognition technology is already a type of technology well known to those skilled in the art, it will not be described in detail in this application.

[0050] For another example, if an image signal is received, computer vision technology can be used to identify the intent of the image and infer text instructions (for example, a deep neural network combined with a multimodal generative language model can be used to identify a photo of a person pointing at a doorbell as the text instruction "press the doorbell").

[0051] The "user" mentioned in this application can be a natural person, a network communication connection, or any channel that can input information.

[0052] The first embodiment of the present application is introduced below.

[0053] This embodiment describes a simple user interface interaction involving two buttons and text input. This embodiment is intended to show how to operate objects in the user interface through natural language text instructions, in particular, to control buttons through the instructions of "click button 1" and "click button 2".

[0054] In this embodiment, the user interface includes two buttons, labeled "Button 1" and "Button 2". The two buttons are objects that the user can interact with, and their identifiers are their text labels, namely "Button 1" and "Button 2". The user can operate these buttons by entering text instructions in natural language without directly clicking on them.

[0055] Display object: The user interface displays two buttons, "Button 1" and "Button 2". Each button has an identifier that can be described by natural language, that is, their text label.

[0056] Receive text instructions: Users enter natural language text instructions through an input box. For example, users can enter "click button 1" or "click button 2".

[0057] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "click" and the identifier of the operation object is "button 1" or "button 2".

[0058] Execute operation: Based on the analysis result, the system performs the corresponding behavior. If the instruction is "click button 1", the system simulates the action of clicking button 1; if it is "click button 2", it clicks button 2.

[0059] This embodiment uses the technical solution described in the first aspect of the present application to demonstrate how to control objects in the user interface through text instructions, providing another interactive method besides directly clicking on graphic elements. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on a touch screen.

[0060] The second embodiment of the present application is introduced below.

[0061] This embodiment describes a user interface interaction involving a button and text input. This embodiment is intended to show how to operate objects in the user interface through natural language text instructions, especially in the first state ( Figure 2211) The button is controlled by the text instruction "click button 1", and a new button is displayed after clicking, switching to the second state ( Figure 2 212).

[0062] In this example, the user interface includes an initial button labeled "Button 1" ( Figure 2 201). This button is an object that the user can interact with, and its identifier is its text label, i.e., "Button 1". The user can operate this button by entering text instructions in natural language without clicking it directly.

[0063] Display object: The user interface displays a button. The button has an identifier that can be described by natural language, namely its text label "Button 1".

[0064] Receiving text instructions: Users enter natural language text instructions through an input box. For example, users can enter "click button 1".

[0065] Parsing instructions: The system parses the user's text instructions through natural language understanding technology. In this process, the system recognizes that the operation verb is "click" and the identifier of the operation object is "button 1".

[0066] Execute operation: Based on the parsing result, the system executes the corresponding behavior. If the instruction is "click button 1", the system simulates the action of clicking button 1 and displays a new button "button 2" at the bottom of the interface after clicking ( Figure 2 In 202 ), the interface is converted from the first state to the second state.

[0067] This embodiment uses the technical solution described in the first aspect of the present application to control objects in the user interface through text instructions and display new objects after the operation, providing another interaction method besides directly clicking on graphic elements.

[0068] The third embodiment of the present application is introduced below.

[0069] This embodiment describes a user interface interaction involving a button ( Figure 3 301), there is a star icon on the top side of the button ( Figure 3 302), there is a sun icon on the bottom side of the button ( Figure 3 303). This embodiment is intended to demonstrate how to operate objects in a user interface through natural language text instructions, in particular, to control buttons and their icons through text instructions such as "click the star icon above the button" and "click the sun icon below the button".

[0070] In addition, this embodiment also demonstrates that the object movement operation is achieved through the instruction of "drag the star above the button to the graphic icon below the button" to prove that the technical solution of the present application can support fuzzy search and use computer vision technology to identify icons.

[0071] Display objects: The user interface displays a button and two icons. The button is an object that the user can interact with, with a star icon on the upper side of the button and a sun icon on the lower side of the button. Each object has an identifier that can be described in natural language, that is, a description of their location, such as "star icon above the button" and "sun icon below the button".

[0072] Receive text instructions: Users enter natural language text instructions through an input box. For example, users can enter "click the star icon above the button" or "click the sun icon below the button."

[0073] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "click" and the identifier of the operation object is "the star icon above the button" or "the sun icon below the button." For the instruction "drag the star above the button to the graphic icon below the button," the system recognizes that the "graphic icon" refers to the "sun icon" through computer vision technology.

[0074] Execute operation: Based on the parsing result, the system executes the corresponding behavior. If the instruction is "click the star icon above the button", the system simulates the action of clicking the star icon; if it is "click the sun icon below the button", it clicks the sun icon. For the instruction "drag the star above the button to the graphic icon below the button", the system simulates the action of dragging the star icon to the sun icon.

[0075] This embodiment uses the technical solution described in this application to demonstrate how to control objects in a user interface through text instructions, especially to identify objects through fuzzy search and computer vision technology. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on a touch screen. This embodiment achieves at least the first and second aspects, and achieves the beneficial effects of controlling user interface objects through natural language instructions and supporting fuzzy search.

[0076] The fourth embodiment of the present application is introduced below.

[0077] This embodiment describes the user interface interaction, involving three buttons ( Figure 4401, 402 and 403) and text input. This embodiment is intended to demonstrate how to operate objects in a user interface through natural language text instructions, in particular, to control buttons through text instructions such as "click the blue button", "click the 'click me' button", "click the heart-shaped icon button" and "click all buttons from left to right in sequence".

[0078] In this embodiment, the user interface includes three buttons. The left button is labeled "Click Me" ( Figure 4 401), the middle button is schematically dotted, representing a certain color fill ( Figure 4 402), for example, in one embodiment it can be blue. The button on the right has a heart-shaped icon ( Figure 4 Each button is an object that the user can interact with, and their identifiers are "click me", "blue", and "heart icon".

[0079] Display object: The user interface displays three buttons. Each button has an identifier that can be described by natural language, namely "click me", "blue", and "heart icon".

[0080] Receive text instructions: Users enter natural language text instructions through an input box. For example, users can enter "click the blue button", "click the 'click me' button", "click the heart-shaped icon button", or "click all buttons from left to right in order".

[0081] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "click" or "click in sequence", and the identifier of the operation object is "blue", "click me" or "heart icon".

[0082] Execute operation: Based on the parsing result, the system executes the corresponding behavior. If the instruction is "click the blue button", the system simulates the action of clicking the middle button (402); if it is "click the 'click me' button", the system clicks the left button (401); if it is "click the heart-shaped icon button", the system clicks the right button (403). If the instruction is "click all buttons from left to right in order", the system clicks the left button (401), the middle button (402), and the right button (403) in order.

[0083] This embodiment implements at least the first and third aspects, and demonstrates how to control objects of different forms in a user interface through text instructions by using the technical solution of the present application.

[0084] The fifth embodiment of the present application is introduced below.

[0085] This embodiment describes a user interface interaction involving five check boxes ( Figure 5 This embodiment is intended to demonstrate how to operate objects in a user interface through natural language text instructions, and in particular, to control a check box through a text instruction such as "uncheck 2 and check 5".

[0086] In this embodiment, the user interface includes five check boxes, which are marked as "1", "2", "3", "4" and "5". Among them, the check boxes "1", "2" and "3" have been checked ( Figure 5 511). Each checkbox is an object that the user can interact with, and their identifier is the number next to them. The user can operate these checkboxes by entering text instructions in natural language without clicking on them directly.

[0087] Display object: The user interface displays five check boxes. Each check box has an identifier that can be described in natural language, which is the number next to them.

[0088] Receive text instructions: Users enter natural language text instructions through an input box. For example, users can enter "uncheck 2 and check 5".

[0089] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verbs are "uncheck" and "check", and the identifiers of the operation objects are "2" and "5".

[0090] Execute operation: Based on the parsing result, the system performs the corresponding action. If the instruction is "uncheck check 2", the system unchecks check box 2; if it is "check check 5", the system checks check box 5.

[0091] After executing the above instructions, the user interface enters the second state ( Figure 5 At this point, checkboxes "1" and "3" are still checked, checkbox "2" is no longer checked, checkbox "4" is still unchecked, and checkbox "5" is checked.

[0092] This embodiment demonstrates how to control objects in a user interface through text instructions by using the technical solution described in this application, and provides another way of interaction besides directly clicking on graphic elements. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on a touch screen. This embodiment implements at least the technical solutions described in the first and fourth aspects, realizes the function of controlling multiple objects in a user interface through natural language text instructions, and improves the flexibility and efficiency of interaction.

[0093] The sixth embodiment of the present application is introduced below.

[0094] This embodiment describes a user interface interaction involving three stars ( Figure 6 This embodiment is intended to demonstrate how to operate objects in a user interface through natural language text instructions, and in particular, to control the stars through text instructions such as "mouse slides over stars 1, 3, and 2 in sequence".

[0095] In this embodiment, the user interface includes three stars, each of which is marked as “Star 1” ( Figure 6 601), "Star 2" ( Figure 6 602) and "Star 3" ( Figure 6 603). These stars are objects that users can interact with, and their identifiers are their numbers, i.e., "1", "2", and "3". Users can operate these stars by inputting text instructions in natural language without directly clicking or sliding them.

[0096] Display object: The user interface displays three stars. Each star has an identifier that can be described in natural language, that is, their number.

[0097] Receive text instructions: Users enter natural language text instructions through an input box. For example, users can enter "mouse over stars 1, 3, and 2 in order".

[0098] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "slide" and the identifiers of the operation objects are "star 1", "star 3" and "star 2".

[0099] Execute operation: Based on the analysis result, the system executes the corresponding behavior. If the instruction is "mouse slides over stars 1, 3, and 2 in order", the system simulates the action of the mouse pointer 611 sliding over stars 1, 3, and 2 in order. The movement track 612 of the mouse pointer is as follows: Figure 6 shown.

[0100] In this embodiment, the position of a star responds to a mouse sliding event in two ways: first, it will not change due to the absolute or relative position relationship with other stars; second, the position of each star and / or its relative position with other stars will affect the effect of mouse sliding, for example, other stars around the star being swiped may dodge. The specific mouse sliding effect can be set by the implementer, and since there are almost infinite ways to set the specific sliding effect, this embodiment will not elaborate on it.

[0101] Of course, the user may not specify the order, in which case the implementer may default to sliding in a certain order (e.g., from left to right) or sliding randomly; the user may also specify random sliding, in which case the implementer may randomly generate the sliding order through a random number generator or other means.

[0102] This embodiment uses the technical solution described in this application to demonstrate how to control objects in the user interface through text instructions, providing another way of interaction besides directly sliding graphic elements. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on the touch screen. This embodiment achieves at least the first and fifth aspects, and achieves the beneficial effect of improving the flexibility and efficiency of user interaction.

[0103] The seventh embodiment of the present application is introduced below.

[0104] This embodiment describes a user interface interaction involving a slider and a button, and implements repeated loop operations by defining macro instructions. This embodiment aims to show how to control objects in the user interface through natural language text instructions, and define a series of macro instructions for predefined operations, which are triggered by a single instruction.

[0105] In this embodiment, the user interface includes a slider and a button. The slider is an object that can be dragged to change its position, and the button is an object that can be clicked to trigger an action. These two objects are objects that the user can interact with, and their identifiers are "slider" and "button" respectively.

[0106] Display objects: The user interface displays a slider and a button. Each object has an identifier that can be described in natural language, that is, their text labels: "slider" and "button".

[0107] Receive text instructions: Users enter natural language text instructions through an input box. For example, users can enter "Define operation A, drag the slider to the far right, then press the button, and finally drag the slider back to the far left."

[0108] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verbs are "drag" and "press", and the identifiers of the operation objects are "slide bar" and "button".

[0109] Execute operation: Based on the parsing result, the system executes the corresponding behavior. If the instruction is "drag the slider to the far right", the system simulates the action of dragging the slider to the far right; if it is "press the button", it simulates the action of clicking the button; if it is "drag the slider back to the far left", it drags the slider back to the far left.

[0110] Define macros: Users can define a macro "Operation A", which includes a series of predefined operations: drag the slider to the far right, press the button, and finally drag the slider back to the far left. Users can trigger this macro by entering the command "Perform Operation A three times", and the system will perform Operation A three times in sequence.

[0111] This embodiment uses the technical solution described in this application to demonstrate how to control objects in a user interface through text instructions and define macro instructions to implement repetitive operations. For users who need to frequently perform the same sequence of operations, the operating efficiency can be greatly improved. This embodiment implements at least the technical solutions described in the first and sixth aspects, or may also be the technical solutions described in the first and fifth aspects, and achieves the beneficial effects of controlling user interface objects through text instructions and simplifying repetitive operations through macro instructions.

[0112] The eighth embodiment of the present application is introduced below.

[0113] This example describes a user interface interaction involving three buttons and two users. The text on the buttons is "A button" (belongs to user 1), "B button" (belongs to user 1), and "C button" (belongs to user 2). This example is intended to show how to operate objects in a user interface through natural language text instructions, and only allows users to operate objects that they own.

[0114] Display objects: Three buttons are displayed on the user interface, namely "A button", "B button" and "C button". Each button has an identifier that can be described in natural language. Among them, "A button" and "B button" belong to user 1, and "C button" belongs to user 2. The ownership of the buttons can be indicated by internal markings in the computer program, visual or auditory sensory distinctions, for example, the buttons of user 1 (A and B) can be filled with a line, and the button of user 2 (C) can be filled with a dotted line.

[0115] Receiving text instructions: Users enter natural language text instructions through the input box. For example, user 1 can enter "press button A" or "press button B", and user 2 can enter "press button C".

[0116] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "press" and the identifier of the operation object is "A button", "B button" or "C button".

[0117] Execute operation: Based on the parsing result, the system executes the corresponding behavior. If user 1 inputs "press button A" or "press button B", the system simulates the action of clicking the corresponding button; if user 2 inputs "press button C", button C is clicked. If user 1 tries to press a button that does not belong to them (such as "C button"), or user 2 tries to press a button that does not belong to them (such as "A button" or "B button"), the system will prompt an error message or directly ignore the instruction.

[0118] In another setting of this embodiment, the "owner" of the "user's own object" described in the ninth aspect of the application can be changed. For example, when a user tries to click a button that does not belong to him, the button can be changed to one that he "owns", thereby allowing the user to operate the button. This setting can improve the flexibility of interaction in a multi-user environment.

[0119] This embodiment may also have more users. For the sake of convenience in explaining this application, two users are described; the buttons owned by the users may also vary.

[0120] This embodiment uses the technical solution described in this application to demonstrate how to control objects in the user interface through text instructions, and only allows users to operate objects that belong to them, providing another way of interaction besides directly clicking on graphic elements. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on the touch screen. This embodiment implements at least the technical solutions described in the first and seventh aspects of this application, and achieves the beneficial effect of ensuring that users can only operate their own objects in a multi-user environment.

[0121] The ninth embodiment of the present application is introduced below.

[0122] This embodiment describes the user interface interaction, and the user can input natural language text instructions in a variety of ways, including voice calls, audio, symbols, speech recognition, TCP socket receiving text, and WebSocket receiving text. This embodiment is intended to show how to operate objects in the user interface through flexible instruction input methods.

[0123] In this embodiment, the user interface displays multiple objects, each of which has an identifier that can be described by natural language. The user can input text instructions in natural language through voice calls, audio input, symbols, voice recognition, TCP socket receiving text, or WebSocket receiving text. For example, the user can say "click button 1" through a voice call or an audio input device, or input a corresponding symbolic instruction through a symbolic input device, or receive a pre-set text instruction through a TCP socket or WebSocket.

[0124] Receiving text instructions: Users can input text instructions in a variety of ways. For example, a user can say "Click button 1" through a voice call, and the system converts the voice signal into the text instruction "Click button 1" through the voice recognition module. Users can also input audio signals through an audio input device, and the system converts the audio signal into the text instruction "Click button 1" through the voice recognition module. In addition, users can also input corresponding symbols through a symbol input device, and the system converts the symbols into the text instruction "Click button 1" through symbol recognition technology. Users can also receive pre-set text instructions, such as "Click button 1", through TCP sockets or WebSockets.

[0125] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "click" and the identifier of the operation object is "button 1".

[0126] Execute operation: Based on the analysis result, the system executes the corresponding behavior. If the instruction is "click button 1", the system simulates the action of clicking button 1.

[0127] This embodiment, by using the technical solution described in this application, demonstrates how to receive natural language text instructions in a variety of ways to control objects in the user interface, and provides another way of interaction besides directly clicking on graphic elements. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on a touch screen. This embodiment achieves at least the first and eighth aspects, and achieves the beneficial effect of improving the flexibility and efficiency of user interaction.

[0128] The tenth embodiment of the present application is introduced below.

[0129] This embodiment describes the interaction of the user interface, receiving text instructions using natural language through text comments received from a third-party platform (such as a live broadcast platform), and parsing these instructions through natural language understanding technology to perform corresponding operations. This embodiment specifically shows how to use text instructions to control objects in the user interface in a live broadcast environment.

[0130] In this embodiment, the user interface displays multiple objects, each of which has an identifier that can be described by natural language. These objects can be virtual items, control buttons or other interactive elements on the live broadcast platform.

[0131] Display objects: The user interface displays multiple objects, each with an identifier that can be described in natural language. For example, these objects can be virtual items, control buttons, or other interactive elements.

[0132] Receiving text instructions: Users send text comments as natural language text instructions through third-party platforms (such as live broadcast platforms, public broadcasting, and online chat software). In the live broadcast platform scenario, these instructions can be sent by viewers, anchors, or other participants during the live broadcast.

[0133] The receiving step includes converting the user's other modal input into text instructions through voice recognition: In this embodiment, in addition to text comments, the user's voice input can also be converted into text instructions through voice recognition technology. For example, the audience can send instructions through voice, and the anchor can control certain functions in the live broadcast platform through voice.

[0134] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system identifies the operation verb and the identifier of the operation object. For example, if the instruction is "click button 1", the system recognizes that the operation verb is "click" and the identifier of the operation object is "button 1".

[0135] Execute operation: Based on the parsing result, the system executes the corresponding behavior. If the instruction is "click button 1", the system simulates the action of clicking button 1; if it is "click button 2", it clicks button 2. These operations can be the movement of virtual objects, the clicking of control buttons, or the execution of other interactive functions.

[0136] This embodiment uses the technical solution described in this application to demonstrate how to receive text instructions in natural language through text comments on a third-party platform (such as a live broadcast platform), and parse these instructions through natural language understanding technology to perform corresponding operations. This provides new possibilities for interaction and control in a live broadcast environment, allowing viewers and other participants to interact with the user interface in the live broadcast platform more conveniently. This embodiment implements at least the first and eighth aspects, and achieves the beneficial effects of receiving natural language instructions through a text source and parsing and executing them on a third-party platform.

[0137] The eleventh embodiment of the present application is introduced below.

[0138] This embodiment describes a user interface interaction in a game scene, involving two teams: Team A and Team B, and three users: User 1, User 2, and User 3. This embodiment aims to show how to control objects (i.e., tanks) in the game scene through natural language text instructions, and in particular, to achieve control of tanks through text instructions such as "all tanks attack Team B tanks", "all tanks retreat", and "two of the tanks retreat".

[0139] In this embodiment, three tanks are placed on the left and right sides of the game scene. These tanks are objects that users can interact with, and their identifiers can be distinguished by users through their fill style, position, or icon. Users can operate these tanks by inputting text instructions in natural language, without directly operating through clicks or other graphical interfaces.

[0140] Display object: The user interface displays three tanks on the left and three tanks on the right. Each tank has an identifier that can be described by natural language, for example, the tank on the left can be described by "the tank on the left" and the tank on the right can be described by "the tank on the right".

[0141] Receiving text instructions: Users input natural language text instructions through an input box. For example, user 1 inputs "all tanks attack team B tanks", user 2 inputs "all tanks retreat", and user 3 inputs "all tanks launch an attack".

[0142] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "attack", "retreat" or "launch attack", and the identifier of the operation object is "all tanks", "two of them", etc.

[0143] Execute operation: Based on the parsing result, the system executes the corresponding behavior. If user 1's instruction is "all tanks attack team B's tank", the system controls all tanks of team A to find the nearest tank of team B to attack; if user 2's instruction is "all tanks retreat", the system controls all tanks of team A to leave the battle; if user 2's instruction is "two of the tanks retreat", the system controls two tanks of team A to retreat and the other to continue fighting; if user 3's instruction is "all tanks attack", the system controls all tanks of team B to attack the tank of team A.

[0144] This embodiment demonstrates how to control objects in a game scene through text instructions by using the technical solution described in this application, and provides another way of interaction besides directly clicking on graphic elements. This may be more convenient for some users or in specific scenarios, and is particularly suitable for complex scenarios that require multi-step or multi-object operations. The user identity authentication function in this embodiment (user 1 and user 2 belong to team A, and user 3 belongs to team B) further enhances the security and accuracy of the interaction, and achieves the beneficial effect of allowing users to operate only tanks belonging to their own team. This embodiment achieves at least the first and seventh aspects, and achieves the beneficial effects of controlling objects through natural language instructions and allowing users to operate only tanks belonging to their own team.

[0145] The twelfth embodiment of the present application is introduced below.

[0146] This embodiment relates to user interface interaction, and the method is applied to a game scene, where a user can operate objects in the game interface through natural language instructions. Specifically, this embodiment describes how to build a factory and produce tanks through text instructions, and these operations will be performed next to the corresponding base according to the user's team affiliation information.

[0147] In this embodiment, the game interface includes two bases, representing two different teams: the base on the left has a star (Team A's base), and the base on the right has a lightning (Team B's base). The elements of the game interface schematically include icons representing the teams, and the specific implementer can use different methods to implement it, or even not express the team affiliation information through visual elements. Figure 7 In the figure, the star schematically represents team A and the lightning schematically represents team B.

[0148] Initial state 701 is as follows Figure 7 As shown, no text instructions are executed at this time, and the game interface only shows two bases, the left base with a star and the right base with lightning.

[0149] User 1 belongs to Team A. When User 1 enters "build a factory next to the base" and then "use the factory to produce 3 tanks", the system will interpret these instructions and build a factory next to Team A's base on the left and build three tanks based on User 1's team affiliation information. Both the factory and the tanks will have stars, indicating that they belong to Team A. The execution results are as follows: Figure 7 As shown in state 702.

[0150] User 2 belongs to Team B. When User 2 enters the same content "Build a factory next to the base" and then "Use the factory to produce 3 tanks", the system will interpret these instructions and build a factory and three tanks next to Team B's base on the right according to User 2's team affiliation information. Both the factory and the tank will have lightning, indicating that they belong to Team B. The execution result is as follows: Figure 7 As shown in state 703.

[0151] This embodiment shows how to control objects in the game interface through natural language text instructions, and these operations will be performed next to the corresponding base according to the user's team affiliation information. This not only improves the flexibility and efficiency of user interaction, but also enhances the accessibility and operation accuracy of the user interface, especially in a multi-user environment. This embodiment achieves at least the first and ninth aspects, and achieves the beneficial effects of operating through natural language instructions and supporting multiple users, each user can only operate their own objects.

[0152] The thirteenth embodiment of the present application is introduced below.

[0153] This embodiment describes a game scene involving two tanks, and involves an interactive method for controlling the tanks through natural language text instructions. This embodiment aims to show how to control the tanks through text instructions such as "control tank No. 1 to attack tank No. 2", and especially considers the influence of the position relationship between the tanks on the execution effect of the instruction.

[0154] In this embodiment, the game scene contains two tanks, labeled "Tank 1" and "Tank 2". The two tanks are objects that users can interact with, and their identifiers are their text labels, namely "Tank 1" and "Tank 2". Users can operate these tanks by entering text instructions in natural language without directly clicking on them.

[0155] Display objects: The user interface displays two tanks. Each tank has an identifier that can be described in natural language, i.e., their text labels.

[0156] Receiving text instructions: Users enter natural language text instructions through an input box. For example, the user can enter "control tank No. 1 to attack tank No. 2".

[0157] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "control", the identifier of the operation object is "Tank 1", and the target operation is "attack Tank 2".

[0158] Execute operation: Based on the analysis result, the system executes the corresponding behavior. If the distance between the two tanks is within the attack range of Tank 1 (the first scenario), Tank 1 will directly start attacking Tank 2. If the distance between the two tanks exceeds the attack range of Tank 1 (the second scenario), Tank 1 will first drive to the range of Tank 2 before starting the attack. Due to the different distances between the tanks, in the second scenario, Tank 1 will take longer to reach the attack position, which may cause other factors in the game to change, such as other tanks providing fire support during the journey.

[0159] This embodiment demonstrates how to control objects in a user interface through text instructions by using the technical solution described in this application, and specifically considers the influence of the positional relationship between objects on the execution effect of instructions. This method not only provides another way of interaction besides directly clicking on graphic elements, but also ensures the accurate execution of instructions in complex interactive scenarios. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on a touch screen. This embodiment achieves at least the first and tenth aspects, and achieves the beneficial effects of controlling objects through natural language instructions and considering the positional relationship of objects.

[0160] The fourteenth embodiment of the present application is introduced below.

[0161] This embodiment relates to a base in a game scene ( Figure 8 801), the base's wireframe has four direction identifiers, "ZN", "ZS", "ZW", and "ZE", respectively. Users can operate these objects through text commands. It should be noted that for the sake of explanation, other elements in the game (such as tanks) are omitted in the figure for ease of explanation.

[0162] Display object: A base (801) is displayed on the user interface. The base identifier is 811, and the north, west, south, and east sides of the base have identifiers "ZN" (812), "ZW" (813), "ZS" (814), and "ZE" (815), respectively.

[0163] Receive text instructions: Users enter natural language text instructions through an input box. For example, users can enter "build a factory above the base" or "build a factory at ZN", or "move the tank to the right of the base" or "move the tank to ZE".

[0164] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes the operation verbs (such as "build", "move") and the identifiers of the operation objects (such as "above the base", "at ZN", "to the right of the base", "ZE").

[0165] Execute operation: Based on the parsing result, the system executes the corresponding behavior. For example, if the instruction is "build a factory above the base" or "build a factory at ZN", the system simulates building a factory above the base; if the instruction is "move the tank to the right side of the base" or "move the tank to ZE", the tank is moved to the right side of the base.

[0166] This embodiment uses the technical solution described in this application to demonstrate how to control objects in the user interface through text instructions, especially in combination with direction identifiers to achieve more flexible operations. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on the touch screen. This embodiment achieves at least the first and eleventh aspects, and achieves the beneficial effects of operating user interface objects through natural language instructions and direction identifiers, so that users can easily use text to describe relative coordinates, or positions, or areas.

[0167] The fifteenth embodiment of the present application is introduced below.

[0168] This embodiment describes a user interface interaction involving a star object and text input. This embodiment is intended to show how to operate an object in a user interface through natural language text instructions, in particular, to control a star through a text instruction like "mouse slides around a star and then clicks the star", and the instruction involves an identifier based on the relative position of a displayed identifier.

[0169] In this embodiment, a star object is displayed on the user interface. The star object has an identifier that can be described by natural language, such as "star". The user can operate the star object by inputting text instructions in natural language, without directly operating it with a mouse.

[0170] Display object: The user interface displays a star object. The star object has an identifier that can be described by natural language, namely "star".

[0171] Receive text instructions: Users enter natural language text instructions through an input box. For example, the user can enter "mouse around the star and then click the star".

[0172] Parsing instructions: The system parses the user's text instructions through natural language understanding (NLU) technology. In this process, the system recognizes that the operation verb is "click" and the identifier of the operation object is "star". At the same time, the system also recognizes that the instruction describing the relative position is "after the mouse slides around the star in a circle."

[0173] Execute the operation: Based on the analysis result, the system executes the corresponding behavior. First, the system simulates the action of sliding the mouse around the star, and then clicks the star. Since the instruction involves the description of relative position, the system needs to determine the trajectory of the mouse sliding by identifying the position of the star.

[0174] This embodiment demonstrates how to control an object in a user interface by using the technical solution described in this application, and the instruction involves an identifier based on the relative position of the displayed identifier. This may be more convenient for some users or in specific scenarios, such as when the user uses voice input or when it is inconvenient to operate on a touch screen. This embodiment achieves at least the twelfth aspect and achieves the beneficial effect of describing the operation object by relative position.

[0175] The sixteenth embodiment of the present application is introduced below.

[0176] This embodiment describes the user interface interaction (such as Fig. 9 ), involving three bases (901 to 903) and five tanks (911 to 915). There are absolute or relative positional relationships between the objects, such as Fig. 9 shown.

[0177] The user determines that the operation object can use relative relationships.

[0178] For example, “three tanks near the base on the left” can be interpreted as the user wants to operate tanks 911, 912, and 913.

[0179] For another example, “a tank between two bases” can be parsed as tank 914, because there is only one tank between bases 901 and 902.

[0180] For another example, "the base farthest from the tank" can be parsed as base 903, because according to Fig. 9 Judging from the coordinates of the bases and tanks, all the tanks in base 903 are the farthest away.

[0181] During one use, the user inputs "tanks near the left base attack the base in the lower right corner". At this time, tanks 911 to 913 go back to attack base 903. Because tanks 911 to 913 are closest to the base, tank 914 is judged to be not near the base because the distance to the base is greater than a certain threshold. The threshold can be hard-coded or obtained by dynamic calculation based on other tanks to maximize the user experience.

[0182] The specific implementation process of determining the above positional relationship can be calculated / estimated by humans, or calculated / estimated using screen coordinates using trigonometric functions, or expressed in internal coordinates of application software / game software, which can be one or more of one-dimensional, two-dimensional, or three-dimensional coordinates.

[0183] This embodiment uses the technical solution described in this application to demonstrate that a user can directly or indirectly operate one or more objects through text instructions, and can indirectly determine the object the user wants to operate by using one or part of the target as a reference object. This embodiment at least implements the technical solution described in the thirteenth aspect.

[0184] The following is an introduction to embodiments of the device described in the present application.

[0185] like Fig.11The device 1101 shown includes four units: a display unit 1111, an input unit 1112, a processing unit 1113, and an execution unit 1114. The display unit 1111 is used to display multiple objects, each of which has an identifier that can be described by natural language. The input unit 1112 is used to receive text instructions using natural language. The processing unit 1113 is configured to parse the text instructions through natural language understanding to determine the operating verb and the identifier of the operating object. The execution unit 1114 is used to perform corresponding operations based on the operating verb, the identifier of the operating object, the type attribute of the object and / or the location information. In summary, the device 1101 at least implements the device described in the thirteenth aspect of the present application, and realizes the beneficial effect of enabling the user to interact with the user interface through text instructions through natural language processing technology.

[0186] By using the technical solutions described in the first to thirteenth aspects of the present application, a person skilled in the art can easily implement the device described in the fourteenth aspect of the present application, so it will not be described in detail below.

[0187] On the basis of the above, the present application also provides a storage medium (the fifteenth aspect of the present application), in which a computer program is stored, and when the computer program is executed, the technical solution described in the first to thirteenth aspects of the present application or any embodiment of the present application is implemented.

[0188] Computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages, such as Java, C++, PHP, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0189] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the unit itself in some cases. For example, an attribute calculation unit may also be described as a "calculation unit".

[0191] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0192] In the context of the present disclosure, a storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A storage medium may be a machine-readable signal medium or a machine-readable storage medium. Storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CDROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0193] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0194] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Claims

1. A user interface interaction method, comprising: Step S101, displaying a plurality of objects, each object having an identifier that can be described by natural language; Step S102, receiving a text instruction using a natural language; Step S103, parsing the text instruction by natural language understanding to determine the identifier of the operation verb and the operation object; Step S104: performing corresponding operations according to the operation verb, the type attribute and / or the position information of the operation object.

2. The method according to claim 1, characterized in that The display object is a visual element on or near the visual element of the user interface itself.

3. The method according to claim 1, characterized in that The identifier includes at least one of text, color, and pattern.

4. The method according to claim 1, characterized in that The text instruction may include multiple operation objects, and the method performs the operation represented by the operation verb on each operation object.

5. The method according to claim 1, characterized in that The text instruction may include multiple operation steps, and the method executes each operation step sequentially or randomly.

6. The method according to claim 1, characterized in that The method further includes defining a macro instruction, wherein the macro instruction includes a series of predefined operations and is executed by triggering a single instruction.

7. The method according to claim 1, characterized in that The object has owner information, and the method only allows the user to operate the object owned by the user.

8. The method according to claim 1, characterized in that The receiving step includes converting the user's other modality input into text instructions through voice recognition.

9. The method according to claim 1, characterized in that The method supports multiple users, each user has his own object set, and the user can only operate his own objects.

10. The method according to claim 1, characterized in that The multiple objects are associated with each other in positions, and the association affects the final execution effect of the method.

11. The method according to claim 1, characterized in that At least some of the objects have one or more additional identifiers in their vicinity, and the user can specify operations to be performed on these additional identifiers through text instructions.

12. The method according to claim 1, characterized in that The operation object may be based on a relative position of a displayed identifier.

13. The method according to claim 1, characterized in that Use one or more operands as reference points to indirectly operate on one or more other operands.

14. An interactive device for a user interface, comprising: A display unit, for displaying a plurality of objects, each object having an identifier that can be described by a natural language, and a type attribute and / or position information; An input unit for receiving text instructions using natural language; A processing unit configured to parse the text instruction through natural language understanding to determine an operation verb and an operation object; The execution unit is used to execute the corresponding operation according to the operation verb, the type attribute and / or the position information of the operation object.

15. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images