Interaction instruction processing method and device, electronic equipment and storage medium

Abstract

The invention provides an interaction instruction processing method and device, electronic equipment and a storage medium. The method comprises the steps that a simulation mouse is displayed on a graphical user interface of a television, the television is connected with a cloud server and used for providing a touch type cloud application program through the cloud server, and the simulation mouse is controlled through a remote controller in signal connection with the television; in response to a touch simulation triggering operation for the graphical user interface, setting the television to enter a touch screen simulation state; in the state, responding to a first setting operation aiming at a first key on the remote controller, taking the current position of the simulated mouse in the application scene interface as a simulated touch point, and sending a simulated touch event corresponding to the simulated touch point to the cloud server according to the first setting operation of the first key; and displaying a processing result returned by the cloud server based on the simulated touch event on the graphical user interface. Through the technology disclosed by the invention, the cloud application program can be popularized on the television side on the basis of not increasing the hardware cost.

Description

Technical Field

[0001] The present disclosure relates to the technical field of cloud computing, and in particular to an interactive instruction processing method, device, electronic device and storage medium. Background Art

[0002] With the development of cloud computing technology, cloud applications are becoming more popular. Cloud applications developed based on touch terminals such as mobile phones are usually touch-type, and users can realize the functions of cloud applications through simple finger operations. Taking cloud games as an example, cloud games refer to games running on cloud servers. Cloud servers are a type of computing service that can provide simple and efficient processing capabilities and elastically scalable. Cloud servers are responsible for the game's logical operations, screen rendering and other functions. Users only need to use the game display components (such as a display with touch function) to play the game.

[0003] Traditional TVs usually do not have touch functions and need to be controlled by external components such as remote controls. If cloud applications developed based on touch devices such as mobile phones are directly transplanted to TVs, these cloud applications require touch operation drivers and therefore cannot recognize the button click input of the remote control, making it difficult to popularize touch-type cloud applications on TVs. Summary of the invention

[0004] The purpose of the present disclosure is to provide an interactive instruction processing method, device, electronic device and storage medium, so as to popularize cloud applications on the TV side without increasing hardware costs.

[0005] An embodiment of the present disclosure provides an interactive instruction processing method, the method comprising: displaying a simulated mouse on a graphical user interface of a television; wherein the television is connected to a cloud server for providing a touch-type cloud application for a user to use through the cloud server, and the simulated mouse is controlled by a remote controller connected to the television signal; in response to a touch simulation trigger operation on the graphical user interface, setting the television to enter a touch screen simulation state; wherein the graphical user interface displays an application scenario interface of the cloud application, and the simulated mouse is located in the application scenario interface; in the touch screen simulation state, in response to a first setting operation on a first button on the remote control, taking the current position of the simulated mouse in the application scenario interface as a simulated touch point, sending a simulated touch event corresponding to the simulated touch point to the cloud server according to the first setting operation of the first button; and displaying on the graphical user interface a processing result returned by the cloud server based on the simulated touch event.

[0006] In a second aspect, an embodiment of the present disclosure further provides an interactive instruction processing device, the device comprising: a first display module, used to display a simulated mouse on a graphical user interface of a television; wherein the television is connected to a cloud server, and is used to provide a touch-type cloud application for a user to use through the cloud server, and the simulated mouse is controlled by a remote control connected to the television signal; a state setting module, used to set the television to enter a touch screen simulation state in response to a touch simulation trigger operation on the graphical user interface; wherein the graphical user interface displays an application scenario interface of the cloud application, and the simulated mouse is located in the application scenario interface; an interactive module, used to, in the touch screen simulation state, respond to a first setting operation on a first button on the remote control, use the current position of the simulated mouse in the application scenario interface as a simulated touch point, and send a simulated touch event corresponding to the simulated touch point to the cloud server according to the first setting operation of the first button; a second display module, used to display on the graphical user interface the processing result returned by the cloud server based on the simulated touch event.

[0007] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the above-mentioned interactive instruction processing method.

[0008] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned interactive instruction processing method.

[0009] The embodiments of the present disclosure provide an interactive instruction processing method, device, electronic device and storage medium. The method displays a simulated mouse on a graphical user interface of a television, uses the simulated mouse position as a touch point, and uses the operation on a remote control as a finger gesture of tapping the screen, thereby simulating the operation of finger touching the screen through the remote control to interact with a cloud server. This technology does not require modification of the touch driving method in the cloud application, so that touch-type cloud applications can be applied to the television. This expands the functions of the television without increasing the hardware cost, and broadens the application scenarios of touch-type cloud applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 A flowchart of an interactive instruction processing method provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of simulating the movement of a mouse provided in an embodiment of the present disclosure; Figure 3 A flowchart of an interactive instruction processing method provided by an embodiment of the present disclosure; Figure 4 A structural block diagram of an interactive instruction processing device provided in an embodiment of the present disclosure; Figure 5 A structural block diagram of another interactive instruction processing device provided in an embodiment of the present disclosure; Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] The technical solution of the present disclosure will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0013] Today's televisions can access broadband networks through set-top boxes or computers to provide services such as digital television, time-shifted television, and interactive television. The emergence of Internet television has brought people a new way to watch television. Users can choose the programs they are interested in to play on TV, realizing functions such as on-demand viewing and stop-and-see.

[0014] In order to further expand the functions of the TV, this embodiment provides an interactive instruction processing method, device, electronic device and storage medium. This technology simulates touch operation through a remote control, so that the TV can apply the touch driving method of the cloud application, effectively reducing the development cost of porting the cloud application to the TV.

[0015] The remote control of this embodiment is also called a TV remote control, which has functions such as controlling TV programs, for example, it can be used to control the power on and off of the TV, switch channels, etc. The remote control includes basic buttons, such as a power button, a menu button, a confirmation button, and direction buttons, and the direction buttons may specifically include buttons for the four directions of up, down, left, and right.

[0016] See also Figure 1 The flowchart of an interactive instruction processing method shown in FIG. 1 is a diagram of a TV set in the embodiment of the present invention. The TV set in the embodiment of the present invention is a terminal device with a display function, which can be configured with a remote controller for controlling TV programs. The method includes the following steps: Step S102, displaying a simulated mouse on a graphical user interface of a television; wherein the television is connected to a cloud server, and is used to provide a touch-type cloud application for a user through the cloud server, wherein the cloud application may be a game, chat, live broadcast, or shopping application based on cloud computing technology. The simulated mouse is controlled by a remote controller connected to a television signal, wherein the signal connection is a wireless signal.

[0017] The above-mentioned touch-type cloud application refers to a cloud application driven by touch operation. The touch operation includes touch object and operation method. The touch object can be represented by the coordinates of the touch point, or by a virtual object or area corresponding to the touch point. The operation method includes click operation, long press operation, sliding operation, double-click operation, triple-click operation, etc.

[0018] The simulated mouse can be displayed on the graphical user interface of the TV by default when the TV is turned on, or it can be displayed on the graphical user interface of the TV when the TV starts the cloud application. Alternatively, the simulated mouse can also be triggered to display by operating the buttons on the remote control. For example: a mouse button is set on the remote control, and clicking the mouse button can trigger the simulated mouse to be displayed on the graphical user interface of the TV. Alternatively, long pressing the menu button on the remote control can also trigger the simulated mouse to be displayed on the graphical user interface of the TV.

[0019] After the simulated mouse is displayed on the graphical user interface of the TV, the movement of the simulated mouse can be controlled by the direction buttons of the remote control, see Figure 2 The simulated mouse 20 is initially located at the center of the graphical user interface 21 of the TV. The remote controller has four direction keys, namely, up, down, left and right. If a direction key in the remote controller is pressed, the simulated mouse 20 moves a certain distance in the direction corresponding to the direction key. The distance can be pre-configured by the system. Figure 2The dotted lines in the upper, lower, left and right directions of the simulated mouse 20 are simulated mouse icons, which are the effects of pressing the up key, down key, left key or right key on the remote control. If a direction key is pressed for a long time, the simulated mouse 20 will continue to move in the direction corresponding to the direction key until the simulated mouse 20 moves to the edge of the graphical user interface and stops.

[0020] Step S104, in response to the touch simulation trigger operation on the graphical user interface, setting the television to enter a touch screen simulation state; wherein the graphical user interface displays an application scenario interface of the cloud application, and the simulated mouse is located in the application scenario interface.

[0021] The above-mentioned touch simulation trigger operation can be an operation on a remote control, or an operation of selecting a virtual control on a graphical user interface by simulating a mouse, or, if the TV supports voice control, it can also be a voice control operation, etc.

[0022] Before the TV enters the touch screen simulation state, the TV is equivalent to the function of an ordinary TV, that is, it is in the TV remote control state. In the TV remote control state, the TV's own system will respond to the button operation of the remote control. In this TV remote control state, the remote control has the function of an ordinary TV remote control and is used to trigger TV program related processing, such as: switching of display content or adjustment of volume, color tone, etc.

[0023] After the TV enters the touch screen simulation state, the remote control no longer triggers TV program-related processing. The remote control acts as a simulated touch operation device. The TV receives the key events of the remote control, simulates touch operations based on the key events, interacts with the cloud server corresponding to the cloud application, and completes the operations corresponding to the cloud application.

[0024] The television of this embodiment, whether in a touch screen simulation state or a non-touch screen simulation state (i.e., a television remote control state), can identify the corresponding buttons and button operation modes of the remote control according to the information carried in the events reported by the remote control. For example, when a user clicks the confirmation button in the remote control, the remote control reports the confirmation button identifier and the number of presses as 1. Based on the confirmation button identifier and the number of presses, it can be identified that the current operation is clicking the confirmation button.

[0025] Step S106, in the touch screen simulation state, in response to the first setting operation for the first button on the remote control, the current position of the simulated mouse in the application scenario interface is used as the simulated touch point, and a simulated touch event corresponding to the simulated touch point is sent to the cloud server according to the first setting operation of the first button.

[0026] In the touch screen simulation state, if the first setting operation is performed on the first button on the remote control, the TV side will receive an event corresponding to the first setting operation of the first button. The TV will identify the first button and the first setting operation mode according to the event, as well as the current position of the simulated mouse, which can be represented by the coordinate information of the simulated mouse in the above application scenario interface. After identifying this information, the TV sends a simulated touch event to the cloud server, which is used to indicate that the simulated touch operation corresponding to the first setting operation has occurred at the current position of the simulated mouse.

[0027] The first button may be a direction button, a confirmation button or a menu button, and the first set operation may be a long press or a click operation. The long press operation corresponds to a button duration greater than a set press threshold, and the click operation is also called a single-click operation, and the button duration corresponding to the long press operation is shorter than the set press threshold.

[0028] The simulated touch events may include sliding events, pressing events, lifting events, etc. There is a correspondence between the simulated touch events and the buttons and button operation modes on the remote controller, and the correspondence may be pre-configured in the TV system or the cloud application of the TV.

[0029] The above method can perform gesture recognition on the five buttons of the remote control "up, down, left, right, and confirm", and convert the recognition results into simulated touch operations such as sliding, confirming, long pressing, pressing or lifting, interact with the cloud server, and then control the cloud application.

[0030] Step S108: displaying the processing result returned by the cloud server based on the simulated touch event on the graphical user interface.

[0031] After receiving the above-mentioned simulated touch event, the cloud server will process it according to the simulated touch event. Different simulated touch events may have different corresponding processing methods. For example: Taking the cloud application as a cloud game, if the current position of the simulated mouse is a jump control, and the first set operation is a click operation of the confirmation key on the remote control, then the corresponding simulated touch event is a finger touch event corresponding to the jump control, and the cloud server will control the player's virtual character in the application scenario interface of the above-mentioned cloud application to perform a jump action. If the current position of the simulated mouse is a virtual direction joystick, and the first set operation is a click operation of the left button on the remote control, then the corresponding simulated touch event is a touch event of the finger on the left side of the virtual direction joystick, and the cloud server will control the simulated mouse in the above-mentioned application scenario interface to move left and the player's virtual character to move left.

[0032] The above method simulates a mouse by displaying it on the graphical user interface of the TV, using the simulated mouse position as the touch point, and the operation on the remote control as the gesture of finger tapping the screen, and then simulates the operation of finger touching the screen through the remote control to interact with the cloud server. This technology does not need to modify the touch driving method in the cloud application, so that touch-type cloud applications can be used on the TV. Without increasing the hardware cost, it expands the function of the TV and broadens the application scenarios of touch-type cloud applications.

[0033] In order to facilitate the user to switch between the TV program and the cloud application, the above method further includes the following steps (1) and (2): (1) In response to a touch simulation exit operation for the touch screen simulation state, switching the touch screen simulation state to a television remote control state; The touch simulation exit operation is similar to the above-mentioned touch simulation trigger operation. It can be an operation on the remote control, or an operation of selecting a virtual control on the graphical user interface by simulating a mouse, or, if the TV supports voice control, it can also be a voice control operation, etc.

[0034] (2) When the TV is in remote control mode, it receives the key commands sent by the remote control and processes the TV programs according to the key commands.

[0035] The above-mentioned TV program processing is similar to conventional TV processing. For details, please refer to the relevant technical implementation, which will not be described in detail here.

[0036] The above method can convert the TV from the touch screen simulation state to the TV remote control state through the touch simulation exit operation, so that the normal functions of the remote control can be used to operate the TV interface. This method can make the remote control have both the functions of an ordinary remote control and the function of simulating finger touch screen operation without increasing the hardware cost.

[0037] As a possible implementation, the above touch simulation trigger operation includes: a second operation (such as a long press operation) on a second button (such as a confirmation button) on a remote control, a third operation (such as a long press operation) on a third button (such as a menu button) on a remote control, a selection operation on a first simulated touch control in a graphical user interface, or a first voice control operation on a TV with a voice control function; accordingly, the above touch simulation exit operation includes: a fourth operation (such as a long press operation) on a fourth button (such as a confirmation button) on a remote control, a fifth operation (such as a long press operation) on a fifth button (such as a menu button) on a remote control, a selection operation on a second simulated touch control in a graphical user interface, or a second voice control operation on a TV with a voice control function. Among them, the first simulated touch control and the second simulated touch control can be the same virtual control. If the TV is currently in a TV remote control state, clicking the virtual control triggers the TV to enter a touch screen simulation state; if the TV is currently in a touch screen simulation state, clicking the virtual control triggers the TV to enter a TV remote control state. The first simulated touch control and the second simulated touch control can also be two different virtual controls, respectively realizing the two trigger operations of entering and exiting the touch screen simulation state. The first voice control operation and the second voice control operation can both be composed of a voice trigger command and a voice control content. For example, the content of the first voice control operation is: TV-TV, enter the touch screen simulation state; the content of the first voice control operation is: TV-TV, exit the touch screen simulation state. Through the above multiple ways of triggering the TV to enter or exit the touch screen simulation state, the user can flexibly select according to actual needs to control the current state of the TV.

[0038] As a possible implementation, the above-mentioned setting of the TV to enter the touch screen simulation state in response to the touch simulation trigger operation for the graphical user interface may include: in response to the second setting operation for the sixth button on the remote control, setting the state identifier of the TV to the first setting value, and sending a simulated finger press event to the cloud server; wherein the second setting operation of the sixth button is a touch simulation trigger operation, the first setting value (for example, 0) is used to indicate that the TV is in the touch screen simulation state, and the simulated finger press event (action: touch_down) carries the first coordinate information of the simulated mouse in the application scene interface. The origin of the coordinate system of the application scene interface can be located at the upper left vertex of the interface, the x-axis extends to the right along the upper side of the interface, and the y-axis extends downward along the left side of the interface. Based on this, the first coordinate information includes the x-coordinate and y-coordinate of the simulated mouse in the coordinate system.

[0039] In specific applications, the resolution of the TV screen may be different from the resolution of the cloud server corresponding to the cloud application. After receiving the above-mentioned first coordinate information, the cloud server needs to convert the first coordinate information according to the resolution of the TV screen and the resolution of the cloud server to determine the actual position of the simulated mouse in the application scenario interface of the cloud application. For example, if the TV screen resolution is 1920×1080, the application scenario interface of the cloud application overlaps with the graphical user interface of the TV, that is, the application scenario interface of the cloud application is also 1920×1080, the resolution of the cloud server is 1280×720, the x coordinate of the simulated mouse is 382, ​​and the y coordinate is 540. Then, the height and width percentage of the current simulated mouse on the TV screen is (382÷1920=20%, 540÷1080=50%). This value is multiplied by the resolution of the cloud server (1280, 720). The calculated coordinates of the simulated mouse corresponding to the application scenario interface on the cloud server side are (0.2, 0.5)*(1280, 720)=(256, 360). Based on this, the cloud server can determine which virtual control or virtual object in the application scenario interface the position of the simulated mouse corresponds to, and then perform subsequent control operations.

[0040] If the resolution of the TV screen is the same as the resolution of the cloud server corresponding to the cloud application, the cloud server can directly apply the first coordinate information to determine the virtual control or virtual object corresponding to the simulated mouse and perform subsequent control operations.

[0041] The above method can record the current state of the TV as a touch screen simulation state by setting the state identifier of the TV to the first set value. The operations of the remote control in this state are all processed as simulated touch operations. By sending a simulated finger press event to the cloud server, the cloud server side can directly respond to the press event. The TV can be regarded as having a touch function. In this way, the touch screen function can be realized without upgrading the hardware of the TV, effectively broadening the application on the TV.

[0042] As a possible implementation, the sending of the simulated touch event corresponding to the simulated touch point to the cloud server according to the first setting operation of the first button may include: if the first setting operation of the first button is a single-click operation on the direction key, sending a simulated finger movement event (action: touch_move) to the cloud server; wherein the simulated finger movement event carries the second coordinate information corresponding to the simulated touch point and the direction corresponding to the direction key; accordingly, the processing result returned by the cloud server based on the simulated touch event includes: controlling the simulated mouse to move a set distance in the direction corresponding to the direction key. The set distance is a pre-configured distance.

[0043] In the above method, each time the user clicks the direction key on the remote control, the TV simulates the single-click operation into a sliding operation on the screen, thereby generating a simulated finger movement event. The direction of the sliding operation is consistent with the direction corresponding to the direction key. For example, if the user clicks the left button of the remote control, the direction of the sliding operation is to slide to the left. The direction corresponding to the direction key carried by the simulated finger movement event is the left direction. The current position of the simulated mouse is the simulated touch point. The corresponding second coordinate information is similar to the first coordinate information, which is the x-coordinate and y-coordinate of the current position of the simulated mouse. The above process sends a simulated finger movement event to the cloud server by clicking the direction key on the remote control, triggering the cloud server to control the simulated mouse to move a set distance in the direction corresponding to the direction key based on the simulated finger movement event, thereby achieving the effect of simulating finger sliding on the TV screen.

[0044] As a possible implementation, the touch simulation exit operation is the second setting operation for the sixth button, that is, the touch simulation exit operation is the same as the touch simulation trigger operation; accordingly, switching the touch screen simulation state to the TV remote control state includes: setting the state identifier of the TV to the second setting value, and sending a simulated finger lift event (action: touch_up) to the cloud server; wherein the second setting value (for example, 1) is used to indicate that the TV is in the TV remote control state, and the simulated finger press event carries the third coordinate information of the simulated mouse in the application scene interface. The third coordinate information is similar to the first coordinate information or the second coordinate information, and is the x coordinate and y coordinate of the simulated mouse in the coordinate system corresponding to the application scene interface. By using the same operation (second setting operation) of the same button (sixth button) to trigger the TV to enter or exit the touch screen simulation state, the user's memory cost can be reduced and the user's operation of the remote control can be simplified; by setting the state identifier of the TV to the second setting value, the current state of the TV can be recorded as the TV remote control state, and the operation of the remote control in this state is processed according to the ordinary remote control operation. By sending the simulated finger lift event to the cloud server, the cloud server side can directly end the current round of simulated touch processing process to avoid unnecessary long-term occupation of cloud server resources. The second setting operation of the sixth button can be a long press operation on the sixth button. The identification method of the long press operation can be determined by counting the number of press event notifications sent by the remote control, or by counting the length of time the sixth button is pressed by the remote control. Based on this, the method further includes the following steps (1) and (2): (1) In response to a pressing operation on a sixth button on the remote control, counting the number of press event notifications (i.e., event.getRepeatCount()) received from the remote control within a first time period; When a button on the remote control is pressed, the remote control sends an event notification to the TV to indicate whether the button is pressed. For example, if event.getAction() == KeyEvent.ACTION_DOWN, it indicates that a button is pressed.

[0045] (2) In response to the number of press event notifications being greater than a first threshold (eg, 3), it is determined that a second setting operation of the sixth key occurs.

[0046] The above method can quickly determine the long press operation of the sixth button by counting the number of sixth button press event notifications received within the first time period, and then quickly respond to the long press operation, thereby improving the sensitivity of the TV to the remote control.

[0047] In order to facilitate the user to select the virtual object in the application scene interface, the above method also includes: in response to the click operation on the confirmation key of the remote control, the click operation is processed according to the current position of the simulated mouse. If the current position of the simulated mouse corresponds to the virtual control, the click operation is to select the virtual control and perform the processing corresponding to the virtual control. In this way, whether the TV is in the TV remote control state or the touch screen simulation state, the existing function of the confirmation key of the remote control can be retained, thereby improving the convenience of the user to operate the remote control.

[0048] As a possible implementation, the cloud application is a cloud game, and the initial interface of the application scenario interface is a login interface; the method further includes: in a touch screen simulation state, in response to a login verification operation on the login interface, performing login verification on the user of the TV, and after the verification is passed, setting the user's virtual character to enter the application scenario interface of the cloud game. The login verification here can be an operation of selecting characters and / or icons, or an operation of sliding a tile to a specified position to prevent illegal login to the cloud game. In this way, cloud games can be played on the TV through a remote control, and the user can be logged in and verified by simulating touch to maintain the security of the cloud game.

[0049] If the above-mentioned cloud application is a cloud game, the above-mentioned TV can also be connected to a game controller to support controlling the cloud game by means of a game controller. Based on this, before setting the TV to enter the touch screen simulation state, the above-mentioned method also includes: detecting whether the TV is connected to a game controller; if the TV is not connected to a game controller, executing the step of setting the TV to enter the touch screen simulation state. If the TV is connected to a game controller, a text or voice prompt can be provided to the user that the TV is connected to a game controller, and the touch screen simulation state will no longer be entered. The TV is set to enter the game controller control state, and the cloud game can be controlled by the game controller. In this way, users can use a remote control to play cloud games on the TV, or they can use a game controller to play cloud games on the TV, which improves the convenience of cloud games.

[0050] This embodiment takes the cloud game provided by the TV as an example to provide an interactive instruction processing method. The cloud game refers to a game mode based on cloud computing. In the operation mode of the cloud game, the operation subject of the game program and the game screen presentation subject are separated. The storage and operation of the above-mentioned interactive instruction processing method are completed on the cloud game server. The role played by the TV when running the cloud game is used for data reception, transmission and presentation of the game screen. The TV is equivalent to a display device with data transmission function close to the user side, but the device for information processing is a cloud server in the cloud. When playing the game, the player operates the remote control and sends instructions (or events) for simulating touch operations to the cloud server through the TV. The cloud server runs the game according to the instruction, encodes and compresses the game screen and other data, and returns it to the TV through the network. Finally, the game screen is decoded and output through the TV, and the TV display presents the game screen through a graphical user interface.

[0051] The TV in the method is connected to a cloud server and can provide cloud games through the cloud server. The TV is also equipped with a remote controller. In addition to a power button, the remote controller also includes a confirmation button, an up button, a down button, a left button, and a right button. The up button, the down button, the left button, and the right button are respectively located around the confirmation button. Figure 3 The flowchart of the interactive instruction processing method shown in the figure shows that the TV is turned on and the graphical user interface displays a simulated mouse. The graphical user interface also includes a game icon of a cloud game. The method includes the following steps: Step S301: The remote control receives a signal from a user that an up key is clicked on a direction key, and sends a first event notification of the up key being pressed to the television.

[0052] The first event notification can be achieved through the following three command messages: (1) Whether it is pressed. If event.getAction() == KeyEvent.ACTION_DOWN, it is pressed; (2) Whether it is a long press: If event.getRepeatCount()>3, it is a long press; otherwise, it is a click; (3) Key type: event.getKeyCode() KeyEvent.KEYCODE_DPAD_DOWN, corresponding to the down key; KeyEvent.KEYCODE_DPAD_UP, corresponding to the up key; KeyEvent.KEYCODE_DPAD_LEFT, corresponding to the left button; KeyEvent.KEYCODE_DPAD_RIGHT, corresponding to the right button; KeyEvent.KEYCODE_DPAD_CENTER, corresponding to the confirmation key; Step S302: The TV receives the first event notification and moves the simulated mouse on the graphical user interface upward. After the movement, the simulated mouse is located at the game icon position of the cloud game.

[0053] The user can use the arrow keys on the remote control to control the simulated mouse movement multiple times in a row. For example, if the TV receives a "left" button from the remote control, if the simulated mouse is at the leftmost side of the screen, it will be ignored; otherwise, the simulated mouse can be controlled to move 2% of the screen distance to the left. For example, if the TV receives a "up" button from the remote control, if the simulated mouse is at the top of the screen, it will be ignored; otherwise, the simulated mouse can be controlled to move 2% of the screen distance upward.

[0054] Step S303: the remote control receives a signal that the user clicks the confirmation key, and sends a second event notification of the confirmation key being pressed to the television.

[0055] Step S304, the TV responds to the second event notification, selects the game icon at the simulated mouse position, and opens the login interface of the cloud game.

[0056] The login interface displays a login verification box, and the user needs to drag the tile in the login verification box to the correct graphic box.

[0057] Step S305: When the remote control receives a long press of the confirmation key by the user, the remote control sends a third event notification indicating that the confirmation key is long pressed to the television.

[0058] Step S306: The TV responds to the third event notification, sets the TV to enter a touch screen simulation state, and sends a touch_down event to the cloud server. The touch_down event carries coordinate information of the simulated mouse.

[0059] Step S307: After receiving the touch_down event, the cloud server monitors the user's login verification operation.

[0060] Step S308: When the remote control receives a long press of the right button of the direction buttons by the user, the remote control sends a fourth event notification indicating that the right button is long pressed to the TV.

[0061] Step S309: In response to the fourth event notification, the TV sends a touch_move event to the cloud server. The touch_move event carries right-button long-press information and coordinate information of the simulated mouse.

[0062] Step S310: After receiving the touch_move event, the cloud server controls the simulated mouse to continue moving a certain distance to the right.

[0063] The user can use the direction keys of the remote control and the simulated mouse to drag the tiles in the login verification box to the correct graphic box. The cloud server determines whether the user's login verification is passed based on the final position of the tiles. The specific dragging process may require multiple clicks or long presses on the direction keys to achieve, which will not be repeated here.

[0064] Step S311: When the remote control receives a long press of a confirmation key by the user, the remote control sends a fifth event notification indicating that the confirmation key is long pressed to the television.

[0065] Step S312: The TV responds to the fifth event notification, sets the TV to exit the touch screen simulation state, and sends a touch_up event to the cloud server.

[0066] Step S313: After receiving the touch_up event, the cloud server ends monitoring the cloud application.

[0067] The above steps S305 to S313 are an example of a touch cycle. By setting the TV to enter the touch screen simulation state, the event notifications of the remote control received by the TV in this state are converted into simulated touch events, namely the above touch_down event, touch_move event or touch_up event. Through the simulated touch event, the TV interacts with the cloud server, thereby triggering the cloud server to control and process the cloud game and present its processing results on the TV.

[0068] Corresponding to the above method, this embodiment also provides an interactive instruction processing device, see Figure 4 , the device includes the following modules: A first display module 42 is used to display a simulated mouse on a graphical user interface of a television; wherein the simulated mouse is controlled by a remote controller; the television is connected to a cloud server, and is used to provide a touch-type cloud application for a user through the cloud server, and the simulated mouse is controlled by a remote controller connected to the television signal; A state setting module 44, configured to set the television to enter a touch screen simulation state in response to a touch simulation trigger operation on a graphical user interface, wherein the graphical user interface displays an application scenario interface of a cloud application program, and the simulated mouse is located in the application scenario interface; The interaction module 46 is used to respond to a first setting operation on a first button on the remote control in a touch screen simulation state, using the current position of the simulated mouse in the application scenario interface as a simulated touch point, and sending a simulated touch event corresponding to the simulated touch point to the cloud server according to the first setting operation on the first button; The second display module 48 is used to display the processing result returned by the cloud server based on the simulated touch event on the graphical user interface.

[0069] The above-mentioned device simulates a mouse by displaying it on the graphical user interface of the TV, using the simulated mouse position as the touch point, and using the operation on the remote control as the gesture of finger tapping the screen, and then simulating the operation of finger touching the screen through the remote control to interact with the cloud server. This technology does not require modifying the touch driving method in the cloud application, so that touch-type cloud applications can be used on the TV. Without increasing the hardware cost, it expands the function of the TV and broadens the application scenarios of touch-type cloud applications.

[0070] As a possible implementation, the state setting module 44 is also used to: in response to a touch simulation exit operation for the touch screen simulation state, switch the touch screen simulation state to a TV remote control state; in the TV remote control state, receive a key command sent by the remote control, and process the TV program according to the key command. Through the touch simulation exit operation, the TV can be converted from the touch screen simulation state to the TV remote control state, so that the normal functions of the remote control can be used to operate the TV interface. This method can enable the remote control to have both the functions of an ordinary TV remote control and the function of simulating finger touch screen operation without increasing the hardware cost.

[0071] As a possible implementation, the touch simulation trigger operation includes: a second operation (such as a long press operation) on a second button (such as a confirmation button) on the remote control, a third operation (such as a long press operation) on a third button (such as a menu button) on the remote control, a selection operation on a first simulated touch control in the graphical user interface, or a first voice control operation on the TV with a voice control function; correspondingly, the touch simulation exit operation includes: a fourth operation (such as a long press operation) on a fourth button (such as a confirmation button) on the remote control, a fifth operation (such as a long press operation) on a fifth button (such as a menu button) on the remote control, a selection operation on a second simulated touch control in the graphical user interface, or a second voice control operation on the TV with a voice control function. Protect multiple ways of triggering the TV to enter or exit the touch screen simulation state, and users can flexibly choose according to actual needs.

[0072] As a possible implementation, the state setting module 44 is also used to: in response to the second setting operation for the sixth button on the remote control, set the state identifier of the TV to the first setting value, and send a simulated finger press event to the cloud server; wherein the second setting operation of the sixth button is a touch simulation trigger operation, the first setting value is used to indicate that the TV is in a touch screen simulation state, and the simulated finger press event carries the first coordinate information of the simulated mouse in the application scenario interface. By setting the state identifier of the TV to the first setting value, the current state of the TV can be recorded as a touch screen simulation state, and the operations of the remote control in this state are all processed according to the simulated touch operation. By sending a simulated finger press event to the cloud server, the cloud server side can directly respond to the press event, and the TV can be regarded as having a touch function. The touch screen function can be realized without upgrading the hardware of the TV, which effectively broadens the application on the TV.

[0073] As a possible implementation, the interaction module 46 is also used to: if the first set operation of the first button is a single-click operation on the direction key, send a simulated finger movement event to the cloud server; wherein the simulated finger movement event carries the second coordinate information corresponding to the simulated touch point and the direction corresponding to the direction key, and the processing result returned by the cloud server based on the simulated touch event includes: controlling the simulated mouse to move a set distance in the direction corresponding to the direction key. In this way, by clicking the direction key on the remote control, a simulated finger movement event is sent to the cloud server, triggering the cloud server to control the simulated mouse to move a set distance in the direction corresponding to the direction key based on the simulated finger movement event, thereby achieving the effect of simulating finger sliding on the TV screen.

[0074] As a possible implementation, the touch simulation exit operation is the second setting operation for the sixth button; the state setting module 44 is also used to: set the state identifier of the TV to the second setting value, and send a simulated finger lift event to the cloud server; wherein, the second setting value is used to indicate that the TV is in the TV remote control state, and the simulated finger press event carries the third coordinate information of the simulated mouse in the application scenario interface. By using the same operation (second setting operation) of the same button (sixth button) to trigger the TV to enter or exit the touch screen simulation state, the user's memory cost can be reduced and the user's operation of the remote control can be simplified; this method of setting the state identifier of the TV to the second setting value can record the current state of the TV as the TV remote control state, and the operation of the remote control in this state is processed according to the ordinary remote control operation. By sending a simulated finger lift event to the cloud server, the cloud server side can directly end the current round of simulated touch processing process to avoid unnecessary long-term occupation of cloud server resources.

[0075] As a possible implementation, see Figure 5 The schematic diagram of another interactive instruction processing device shown in FIG. Figure 4 Based on the device shown, it also includes: The operation determination module 52 is used to, in response to the pressing operation of the sixth button on the remote control, count the number of press event notifications of the remote control continuously received within a first time period; in response to the number of press event notifications being greater than a first threshold, determine that the second set operation of the sixth button has occurred. By counting the number of press event notifications of the sixth button received within the first time period, a long press operation of the sixth button can be quickly determined, and then the long press operation can be quickly responded to, thereby improving the sensitivity of the television in responding to the remote control.

[0076] As a possible implementation, the interaction module 46 is further configured to: in response to a click operation on the confirmation key of the remote controller, process the click operation according to the current position of the simulated mouse. In this manner, the existing function of the confirmation key of the remote controller can be retained, thereby improving the convenience of the user in operating the remote controller.

[0077] As a possible implementation, the cloud application is a cloud game, and the initial interface of the application scenario interface is a login interface; see Figure 5 The device further includes: a login verification module 54, which is used to perform login verification on the user of the TV in response to the login verification operation on the login interface in the touch screen simulation state, and after the verification is passed, set the user's virtual character to enter the application scenario interface of the cloud game. In this way, cloud games can be played on the TV through the remote control, and the user can be logged in and verified by simulating touch, thereby maintaining the security of cloud games.

[0078] As a possible implementation, see Figure 5 The above device further includes: a game controller detection module 56, which is used to detect whether the TV is connected to a game controller before setting the TV to enter the touch screen simulation state; if the TV is not connected to the game controller, the trigger state setting module 44 sets the TV to enter the touch screen simulation state. In this way, users can use the remote control to play cloud games on the TV, or use the game controller to play cloud games on the TV, which improves the convenience of cloud games.

[0079] The above-mentioned TV has not entered the touch screen simulation state, that is, the TV is still in the TV remote control state. In this state, the touch-type cloud application will not respond to the operation of the buttons on the remote control, because these cloud applications require touch drive, which cannot recognize the button operation on the remote control. Correspondingly, the simulated mouse cannot act as a simulated touch point, but is a simple interface operation mark, indicating the target object corresponding to the current operation of the remote control, that is, the TV does not have a touch drive function, so it cannot control the touch-type cloud application, which makes it difficult to popularize the touch-type cloud application on the TV. The above-mentioned method and device, by setting the TV to enter the touch screen simulation state, can transform the functions of the simulated mouse and the remote control, regard the simulated mouse as a simulated touch point, and transform the button operation on the remote control into the touch operation of the touch-type cloud application, and then still use the touch drive method to control the cloud application, so that the touch-type cloud application can be easily transplanted from touch terminals such as mobile phones to TVs.

[0080] The interactive instruction processing device provided in the embodiment of the present disclosure has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for parts not mentioned in the embodiment of the interactive instruction processing device, reference may be made to the corresponding contents in the aforementioned interactive instruction processing method embodiment.

[0081] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0082] The present disclosure also provides an electronic device, such as Figure 6 As shown, it is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 61 and a memory 60, the memory 60 stores computer executable instructions that can be executed by the processor 61, and the processor 61 executes the computer executable instructions to implement the above-mentioned interactive instruction processing method.

[0083] exist Figure 6 In the illustrated embodiment, the electronic device further includes a bus 62 and a communication interface 63 , wherein the processor 61 , the communication interface 63 and the memory 60 are connected via the bus 62 .

[0084] Among them, the memory 60 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 62 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] The processor 61 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 61 or the instruction in the form of software. The above processor 61 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present disclosure can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 61 reads the information in the memory and completes the steps of the interactive instruction processing method of the above embodiment in combination with its hardware.

[0086] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned interactive instruction processing method. The specific implementation can be found in the aforementioned method embodiment, which will not be repeated here.

[0087] The computer program product of the interactive instruction processing method, device and electronic device provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0088] Unless otherwise specifically stated, the relative steps, numerical expressions and numerical values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0089] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that can be executed by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0090] In the description of the present disclosure, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present disclosure. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0091] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for processing interactive instructions, characterized in that: The method comprises: Displaying a simulated mouse on a graphical user interface of a television; wherein the television is connected to a cloud server, and is used to provide a touch-type cloud application for a user through the cloud server, and the simulated mouse is controlled by a remote controller connected to a television signal; In response to a touch simulation trigger operation on the graphical user interface, setting the television to enter a touch screen simulation state; wherein the graphical user interface displays an application scenario interface of the cloud application, and the simulated mouse is located in the application scenario interface; In the touch screen simulation state, in response to a first setting operation on a first button on the remote control, taking the current position of the simulated mouse in the application scenario interface as a simulated touch point, sending a simulated touch event corresponding to the simulated touch point to the cloud server according to the first setting operation of the first button; The processing result returned by the cloud server based on the simulated touch event is displayed on the graphical user interface.

2. The method according to claim 1, characterized in that The method further comprises: In response to a touch simulation exit operation for the touch screen simulation state, switching the touch screen simulation state to a television remote control state; In the television remote control state, a key command sent by the remote controller is received, and television program processing is performed according to the key command.

3. The method according to claim 2, characterized in that The touch simulation trigger operation includes: a second operation on a second button on the remote control, a third operation on a third button on the remote control, a selection operation on a first simulated touch control in the graphical user interface, or a first voice control operation on the television with a voice control function; The touch simulation exit operation includes: a fourth operation on the fourth key on the remote control, a fifth operation on the fifth key on the remote control, a selection operation on the second simulated touch control in the graphical user interface, or a second voice control operation on the TV with voice control function.

4. The method according to claim 1, characterized in that The step of setting the television to enter a touch screen simulation state in response to a touch simulation trigger operation on the graphical user interface includes: In response to the second setting operation for the sixth button on the remote control, the status identifier of the TV is set to the first setting value, and a simulated finger press event is sent to the cloud server; wherein the second setting operation of the sixth button is a touch simulation trigger operation, the first setting value is used to indicate that the TV is in a touch screen simulation state, and the simulated finger press event carries the first coordinate information of the simulated mouse in the application scenario interface.

5. The method according to claim 4, characterized in that The sending the simulated touch event corresponding to the simulated touch point to the cloud server according to the first setting operation of the first button includes: If the first set operation of the first button is a single-click operation on the direction key, a simulated finger movement event is sent to the cloud server; wherein, the simulated finger movement event carries the second coordinate information corresponding to the simulated touch point and the direction corresponding to the direction key, and the processing result returned by the cloud server based on the simulated touch event includes: controlling the simulated mouse to move a set distance in the direction corresponding to the direction key.

6. The method according to claim 4, characterized in that The touch simulation exit operation is the second setting operation for the sixth button; Switching the touch screen simulation state to the TV remote control state includes: The state identifier of the TV is set to a second setting value, and a simulated finger lift event is sent to the cloud server; wherein the second setting value is used to indicate that the TV is in a TV remote control state, and the simulated finger press event carries the third coordinate information of the simulated mouse in the application scenario interface.

7. The method according to any one of claims 4 to 6, characterized in that The method further comprises: In response to a pressing operation on a sixth button on the remote controller, counting the number of times that the pressing event notifications of the remote controller are continuously received within a first time period; In response to the number of press event notifications being greater than a first threshold, it is determined that a second setting operation of the sixth key occurs.

8. The method according to claim 1, characterized in that The method further comprises: In response to a click operation on a confirmation key of the remote controller, the click operation is processed according to a current position of the simulated mouse.

9. The method according to claim 1, characterized in that: The cloud application is a cloud game, and the initial interface of the application scenario interface is a login interface; the method further includes: In the touch screen simulation state, in response to the login verification operation for the login interface, the user of the TV is logged in and verified. After the verification is passed, the user's virtual character is set to enter the application scenario interface of the cloud game.

10. The method according to claim 9, characterized in that Before setting the television to enter the touch screen simulation state, the method further includes: Detecting whether the television is connected to a game controller; If the television is not connected to the game controller, the step of setting the television to enter a touch screen simulation state is performed.

11. An interactive instruction processing device, characterized in that: The device comprises: A first display module, configured to display a simulated mouse on a graphical user interface of a television; wherein the television is connected to a cloud server, configured to provide a touch-type cloud application for a user through the cloud server, and the simulated mouse is controlled by a remote controller connected to a television signal; a state setting module, configured to set the television to enter a touch screen simulation state in response to a touch simulation trigger operation on the graphical user interface; wherein the graphical user interface displays an application scenario interface of the cloud application, and the simulated mouse is located in the application scenario interface; An interaction module, configured to, in the touch screen simulation state, respond to a first setting operation on a first button on the remote controller, use the current position of the simulated mouse in the application scenario interface as a simulated touch point, and send a simulated touch event corresponding to the simulated touch point to the cloud server according to the first setting operation on the first button; The second display module is used to display the processing result returned by the cloud server based on the simulated touch event on the graphical user interface.

12. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 10.

Images