Cross-end interaction method and device, computer equipment, storage medium and program product

Abstract

The embodiment of the invention provides a cross-end interaction method and device, computer equipment, a storage medium and a program product. The cross-end interaction method comprises the steps of obtaining a target shortcut instruction input into a first terminal in a process that the first terminal runs a target application in a cross-end manner; the terminal type of the first terminal is the first terminal type, the target application is developed based on a second terminal type, and the first terminal type and the second terminal type are two different terminal types; obtaining a cross-end running configuration file, wherein the configuration file is used for recording a mapping relation between the shortcut instruction and the cross-end interaction sequence; determining a target cross-end interaction sequence mapped with the target shortcut instruction on the basis of the mapping relationship, wherein the target cross-end interaction sequence is used for indicating a plurality of application operations executed in the target application subjected to cross-end running; and executing a plurality of application operations indicated by the target cross-end interaction sequence in the target application. By adopting the embodiment of the invention, application cross-end interaction can be efficiently and conveniently carried out.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a cross-end interaction method, a cross-end interaction device, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In order to improve the convenience of application experience, a cross-end operation solution for applications is proposed. Cross-end operation of applications means running applications of another platform on one platform, so that applications of another platform can be conveniently experienced on one platform. However, the cross-end interaction experience of applications that are run across different platforms is not good. This is because cross-end operations have limitations in mapping, which makes cross-end interaction of applications inefficient and inconvenient. Summary of the invention

[0003] The embodiments of the present application provide a cross-terminal interaction method, apparatus, computer equipment, storage medium, and program product, which can efficiently and conveniently perform cross-terminal application interaction.

[0004] On the one hand, an embodiment of the present application provides a cross-end interaction method, the cross-end interaction method comprising:

[0005] In the process of the first terminal running the target application across terminals, a target shortcut command input into the first terminal is obtained; the terminal type of the first terminal is a first terminal type, the target application is developed based on a second terminal type, and the first terminal type and the second terminal type are two different terminal types;

[0006] Get the configuration file for cross-end operation. The configuration file is used to record the mapping relationship between shortcut commands and cross-end interaction sequences.

[0007] Determine a target cross-end interaction sequence mapped to the target shortcut command based on the mapping relationship, where the target cross-end interaction sequence is used to indicate multiple application operations executed in the target application being run across ends;

[0008] A plurality of application operations indicated by the target cross-end interaction sequence are executed in the target application.

[0009] Accordingly, an embodiment of the present application provides a cross-end interaction device, the cross-end interaction device comprising:

[0010] an acquisition unit, configured to acquire a target shortcut command input into the first terminal during the process of the first terminal running the target application across terminals; the terminal type of the first terminal is a first terminal type, the target application is developed based on a second terminal type, and the first terminal type and the second terminal type are two different terminal types;

[0011] The acquisition unit is also used to acquire a configuration file for cross-end operation, where the configuration file is used to record the mapping relationship between shortcut commands and cross-end interaction sequences;

[0012] A processing unit, configured to determine a target cross-end interaction sequence mapped to a target shortcut instruction based on a mapping relationship, wherein the target cross-end interaction sequence is used to indicate a plurality of application operations executed in a target application being run across ends;

[0013] The processing unit is further used to execute multiple application operations indicated by the target cross-end interaction sequence in the target application.

[0014] In one implementation, the target cross-end interaction sequence includes a plurality of cross-end interaction instructions arranged in sequence; the processing unit is configured to execute the following steps when executing the plurality of application operations indicated by the target cross-end interaction sequence in the target application:

[0015] Obtaining a sequence entry corresponding to a target cross-end interaction sequence in a configuration file; the sequence entry is used to instruct execution of the target cross-end interaction sequence, and the sequence entry includes application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence;

[0016] The application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence are executed in sequence in the target application.

[0017] In one implementation, the sequence entry also includes execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the display screen of the target application;

[0018] The processing unit is used to sequentially execute the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the target application, specifically to perform the following steps:

[0019] According to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, the application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence are sequentially executed in the display screen of the target application.

[0020] In one implementation, the sequence entry also includes the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence;

[0021] The processing unit is used to sequentially execute the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the display screen of the target application according to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, specifically to perform the following steps:

[0022] According to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence and according to the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence is executed sequentially in the display screen of the target application.

[0023] In one implementation, the sequence entry also includes reference display window information of the target application; any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; and the processing unit is used to execute the following steps when executing the application operation corresponding to the target interaction instruction in the display screen of the target application:

[0024] If the current display window information of the target application matches the reference display window information, then according to the execution coordinates corresponding to the target interaction instruction, the application operation corresponding to the target interaction instruction is executed in the display screen of the target application;

[0025] If the current display window information of the target application does not match the reference display window information, coordinate transformation is performed on the execution coordinates corresponding to the target interaction instruction, and the application operation corresponding to the target interaction instruction is performed in the display screen of the target application according to the first transformed coordinates obtained by the coordinate transformation;

[0026] The reference display window information includes default display window information, or display window information when entering a target interaction instruction.

[0027] In one implementation, the execution coordinates corresponding to the target interaction instruction are adapted to the reference display window information; the processing unit is used to perform coordinate conversion on the execution coordinates corresponding to the target interaction instruction, specifically to perform the following steps:

[0028] Determine the coordinate conversion ratio according to the reference display window information and the current display window information;

[0029] According to the coordinate conversion ratio, the execution coordinates corresponding to the target interactive instruction are converted into coordinates adapted to the current display window information to obtain first converted coordinates.

[0030] In one implementation, any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; the simulator in the first terminal serves as a running container to run the target application across the terminals; and the processing unit is used to execute the application operation corresponding to the target interaction instruction in the display screen of the target application, specifically to execute the following steps:

[0031] Execute the application operation corresponding to the target interaction instruction on the screen canvas of the simulator;

[0032] The execution of the application operation corresponding to the target interaction instruction is transparently transmitted to the display screen of the target application.

[0033] In one implementation, the acquisition unit is further configured to perform the following steps:

[0034] Obtain the entered target cross-end interaction sequence;

[0035] The processing unit is further configured to perform the following steps:

[0036] Assign target shortcuts to target cross-end interaction sequences;

[0037] Map the target cross-end interaction sequence and the target shortcut command and store them in the configuration file;

[0038] Among them, entering the target cross-end interaction sequence includes any of the following: entering the target cross-end interaction sequence by key editing, entering the target cross-end interaction sequence by operation recording, entering the target cross-end interaction sequence by voice entry, and entering the target cross-end interaction sequence by natural language entry.

[0039] In one implementation, the processing unit is used to perform the following steps when recording the target cross-end interaction sequence in an operation recording manner:

[0040] Recording interactive operations performed on a target application; the interactive operations include at least one of a key input operation and a trigger operation on a display screen of the target application;

[0041] Mapping the interactive operation to a corresponding cross-end interactive instruction; the mapped cross-end interactive instruction includes at least one of a key instruction corresponding to the key input operation and a trigger instruction corresponding to the trigger operation;

[0042] Generate a target cross-end interaction sequence according to the cross-end interaction instructions mapped by the interaction operation.

[0043] In one implementation, the processing unit is used to perform the following steps when inputting the target cross-end interaction sequence in a natural language input manner:

[0044] Perform sentence classification and recognition on the input natural language to obtain one or more interaction description sentences in the natural language, wherein any interaction description sentence in the natural language is represented as a target sentence;

[0045] Perform semantic recognition processing on the target sentence to obtain multiple operation description information;

[0046] Map each operation description information to the corresponding cross-end interaction instruction;

[0047] According to the cross-end interaction instructions mapped to each operation description information, a target cross-end interaction sequence is generated.

[0048] In one implementation, the processing unit is further configured to perform the following steps:

[0049] Generate a sequence entry corresponding to the target cross-end interaction sequence, where the sequence entry is used to indicate the execution of the target cross-end interaction sequence;

[0050] Stores sequence entries into a configuration file.

[0051] In one implementation, the target cross-end interaction sequence includes a plurality of cross-end interaction instructions arranged in sequence; the processing unit is configured to generate a sequence entry corresponding to the target cross-end interaction sequence, specifically configured to perform the following steps:

[0052] Obtaining an execution time interval between any two adjacent cross-end interaction instructions in the target cross-end interaction sequence, and determining an execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence according to the execution time interval;

[0053] Obtain the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence;

[0054] Obtaining the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence;

[0055] A sequence entry corresponding to the target cross-end interaction sequence is generated according to the execution time, execution coordinates, and application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence.

[0056] In one implementation, any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; the processing unit is used to obtain the execution coordinates corresponding to the target interaction instruction, specifically to perform the following steps:

[0057] Obtain the reference coordinates corresponding to the target interaction instruction;

[0058] If the reference coordinates corresponding to the target interaction instruction match the reference display window information of the target application, the reference coordinates corresponding to the target interaction instruction are determined as the execution coordinates corresponding to the target interaction instruction;

[0059] If the reference coordinates corresponding to the target interaction instruction are not compatible with the reference display window information of the target application, a coordinate transformation is performed on the reference coordinates corresponding to the target interaction instruction to obtain second converted coordinates that are compatible with the reference display window information of the target application, and the second converted coordinates are determined as the execution coordinates corresponding to the target interaction instruction.

[0060] Accordingly, an embodiment of the present application provides a computer device, the computer device comprising:

[0061] a processor suitable for implementing a computer program;

[0062] Computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned cross-end interaction method.

[0063] Accordingly, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned cross-terminal interaction method.

[0064] Accordingly, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned cross-end interaction method.

[0065] In the embodiment of the present application, the application is run across terminals by the first terminal, the first terminal is a terminal of the first terminal type, the application is developed based on the second terminal type, and the first terminal type and the second terminal type are two different terminal types; the mapping relationship between the shortcut command and the cross-terminal interaction sequence is recorded in the cross-terminal operation configuration file; based on the mapping relationship, in the process of the application being run across terminals by the first terminal, the target cross-terminal interaction sequence mapped to the target shortcut command input into the first terminal can be determined, and multiple application operations indicated by the target cross-terminal interaction sequence can be performed in the application run across terminals by the first terminal. It can be seen that in the embodiment of the present application, in the process of the application being run across terminals by the first terminal, based on the mapping relationship between the shortcut command and the cross-terminal interaction sequence, the input operation of a shortcut command in the first terminal can be quickly mapped to a series of application operations in the application run across terminals by the first terminal, which can improve the interaction efficiency between the application and the cross-terminal running in the first terminal, that is, the application cross-terminal interaction can be efficiently performed; and the input operation of the shortcut command is convenient, which can improve the convenience of the cross-terminal interaction with the application in the first terminal, that is, the application cross-terminal interaction can be conveniently performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 This is a schematic diagram of an interactive mode for cross-terminal interaction of an application provided in an embodiment of the present application;

[0068] Figure 2 This is a schematic diagram of another interactive mode of cross-terminal application interaction provided in an embodiment of the present application;

[0069] Figure 3 It is a schematic diagram of the architecture of a cross-terminal interaction system provided in an embodiment of the present application;

[0070] Figure 4 It is a flowchart of a cross-terminal interaction method provided in an embodiment of the present application;

[0071] Figure 5 This is a schematic diagram of transparent transmission of cross-terminal interaction of an application provided in an embodiment of the present application;

[0072] Figure 6 This is a flow chart of cross-terminal interaction of an application provided in an embodiment of the present application;

[0073] Figure 7 It is a flowchart of another cross-terminal interaction method provided by an embodiment of the present application;

[0074] Figure 8 This is a schematic diagram of an editing function for enabling cross-terminal interactive mapping documents provided in an embodiment of the present application;

[0075] Fig. 9 This is a schematic diagram of an interface for entering a cross-terminal interaction sequence in a key position editing manner provided in an embodiment of the present application;

[0076] Fig.10 It is a flowchart of a key editing method for entering a cross-terminal interaction sequence provided in an embodiment of the present application;

[0077] Fig.11 This is a schematic diagram of an interface for recording a cross-terminal interaction sequence in an operation recording manner provided in an embodiment of the present application;

[0078] Fig.12 This is a flowchart of recording a cross-terminal interaction sequence in an operation recording method provided in an embodiment of the present application;

[0079] Fig.13 This is a schematic diagram of an interface for entering a cross-terminal interaction sequence in a natural language input manner provided in an embodiment of the present application;

[0080] Fig.14 This is a flow chart of a natural language input method for inputting a cross-terminal interaction sequence according to an embodiment of the present application;

[0081] Fig.15 It is a flow chart of recording a cross-terminal interaction sequence by a voice recording method provided in an embodiment of the present application;

[0082] Fig.16This is a schematic diagram of a cross-terminal interaction sequence input process provided by an embodiment of the present application;

[0083] Fig.17 is a structural diagram of a cross-terminal interaction device provided in an embodiment of the present application;

[0084] Fig.18 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0086] In order to more clearly understand the technical solutions provided by the embodiments of the present application, the technical terms involved in the present application are first introduced here:

[0087] 1. Application runs across terminals:

[0088] Cross-terminal application operation refers to the simulation operation of an application developed based on a second terminal type in a first terminal of a first terminal type, and the first terminal type and the second terminal type are two different terminal types. Among them, the application developed based on the second terminal type refers to an application that can be run in a second terminal of the second terminal type. In the embodiment of the present application, the second terminal is a mobile terminal as an example. The application developed based on the second terminal type can be called a mobile terminal application, and the mobile terminal application can include but is not limited to any of the following: APP (mobile terminal software), cloud application, and mini program. Cloud application refers to a software that is mainly accessed through the Internet, so this means that at least part of it is managed by a server (rather than a local computer); for example, cloud games are cloud applications of the game type, and cloud games can use powerful cloud services deployed in data centers to render game screens.

[0089] The difference between the first terminal type and the second terminal type may at least include: the operating system used by the first terminal type is different from the operating system used by the second terminal type; the embodiment of the present application takes the first terminal as a PC (Personal Computer) terminal and the second terminal as a mobile terminal as an example for explanation, and the operating system used by the PC terminal is different from the operating system used by the mobile terminal. The difference between the first terminal type and the second terminal type may also include: the operation method for the first terminal of the first terminal type is different from the operation method for the second terminal of the second terminal type; for example, the first terminal is a PC terminal and the second terminal is a mobile terminal, the operation method for the PC terminal may include keyboard operation and mouse operation, and the operation method for the mobile terminal may include screen trigger operation. It can be seen that the operation method for the PC terminal is different from the operation method for the mobile terminal. Among them, the PC terminal may include but is not limited to any of the following: desktop computers and laptop computers; the mobile terminal may include but is not limited to any of the following: smart phones, tablet computers, smart watches, smart home appliances, and car terminals.

[0090] Furthermore, in the first terminal of the first terminal type, the application developed based on the second terminal type is simulated and run specifically through the application download client in the first terminal. The application download client refers to a client for downloading applications, and the application download client can be used to download applications that can be installed in the first terminal of the first terminal type, and the application download client can also be used to experience applications developed based on the second terminal type. Among them, the client (Client) refers to a program that corresponds to the server and provides local services to customers; except for some programs that only run locally, it is generally installed on an ordinary client and needs to cooperate with the server to run. If the first terminal is a PC, the client type of the application download client is a desktop client, and if the first terminal is a mobile terminal, the client type of the application download client is a mobile client; the desktop client refers to a client version that can be run in a PC, and the mobile client refers to a client version that can be run in a mobile terminal.

[0091] In addition, in the first terminal of the first terminal type, the application download client in the first terminal can simulate the operation of the application developed based on the second terminal type by calling the simulator. A simulator is a software that can simulate the hardware or software environment of another platform (another platform here can be understood as the second terminal of the second terminal type) on a platform (a platform here can be understood as the first terminal of the first terminal type), so that an application of another platform can be run on one platform. In the embodiment of the present application, the first terminal is a PC end, and the simulator is an Android simulator as an example. The main function of the Android simulator is to simulate the operating system of an Android phone on the PC end; through the Android simulator, Android applications can be run on the PC end to experience various functions of Android applications. In addition, the Android simulator can also support the installation, use and uninstallation of Android applications, so that various operations can be performed without a real mobile phone device.

[0092] 2. Application cross-end interaction:

[0093] Application cross-terminal interaction refers to mapping the terminal operation input to the first terminal to the application operation executed in the application cross-terminal running by the first terminal (i.e., the application developed based on the second terminal type is cross-terminal running by the first terminal of the first terminal type) during the process of cross-terminal running of the application. Taking the first terminal as an example of a PC terminal and the application cross-terminal running by the first terminal as an example of a mobile terminal application, the terminal operation input to the first terminal may include keyboard operation, mouse operation, game controller operation, voice operation, etc., and the application operation mapped to be executed in the application cross-terminal running by the first terminal may include click operation, slide operation, input operation, etc.

[0094] For example, when a mobile game application is run across PCs, the PC keyboard is used to input, select, navigate, and other operations. These operations can be mapped to the mobile application in real time, achieving efficient cross-application interaction. Figure 1 As shown, by inputting preset key commands such as W, A, S, D, P, and K on the PC keyboard, the mobile terminal can touch and click the corresponding coordinates of the mobile application. The keyboard mapping can ensure that the input key commands can be accurately executed on the mobile application in the simulator.

[0095] For example, the operation of inputting a key command on the PC side through the keyboard can be mapped to executing a click operation in the mobile application. For example, the operation of inputting a key command "W" on the PC side through the keyboard can be mapped to the operation of clicking the coordinates of the up button 101 in the mobile application to control the game character to move upward.

[0096] Alternatively, clicking a control button in the display screen of the mobile terminal application in the PC terminal with a mouse can be mapped to executing a click operation in the mobile game application. For example, clicking an up button 101 in the display screen of the mobile game application with a mouse can be mapped to an operation of clicking the coordinates of the up button 101 in the mobile game application to control the game character to move upward.

[0097] Based on the above introduction to cross-terminal operation and cross-terminal interaction of applications, in order to improve the efficiency and convenience of cross-terminal interaction of applications, an embodiment of the present application provides a cross-terminal interaction method. The principle of the cross-terminal interaction method is introduced below:

[0098] ① In the process of cross-terminal operation of the application (i.e., the application developed based on the second terminal type is cross-terminal operated by the first terminal of the first terminal type), in the configuration stage of the cross-terminal interaction of the application, the cross-terminal interaction sequence can be entered, and the shortcut command that can be input through the first terminal is assigned to the cross-terminal interaction sequence. The cross-terminal interaction sequence can be used to indicate a series (here, a series refers to multiple) of application operations performed in the application being cross-terminal operated; the cross-terminal interaction sequence and the shortcut command assigned to it can be mapped and stored in the configuration file of the cross-terminal operation. In addition, the cross-terminal interaction sequence can be modified, and the mapping relationship between the cross-terminal interaction sequence and the shortcut command can also be modified. A cross-terminal interaction guidance document that records the mapping relationship between the cross-terminal interaction sequence and the shortcut command can also be generated, so as to facilitate the interaction (here, the interaction refers to the object that performs cross-terminal interaction with the application being cross-terminal operated by the first terminal through the first terminal, and the shortcut command is input into the first terminal by the interaction) to consult and use during the process of cross-terminal interaction of the application.

[0099] like Figure 2 As shown, taking the case where a mobile game application is run across PC terminals as an example, executing a special game skill usually requires the game player to enter a series of complex key combinations. The player may need to quickly press the key combination corresponding to "right, down, left, fist" to execute the special game skill. In an embodiment of the present application, such a complex key combination can be defined as a cross-terminal interaction sequence, and the simulator can allow the cross-terminal interaction sequence to be mapped with a shortcut command consisting of one or more key position commands. For example, a shortcut command "1" can be defined, and mapped with the cross-terminal interaction sequence "DSAP" corresponding to "right, down, left, fist".

[0100] ② In the process of cross-end interaction of applications, the cross-end interaction method can obtain the shortcut command input into the first terminal, can determine the cross-end interaction sequence mapped to the shortcut command based on the mapping relationship in the configuration file, and can execute a series of application operations indicated by the cross-end interaction sequence mapped to the shortcut command in the application run across the end. It can be seen that the embodiment of the present application realizes the input operation of a shortcut command in the first terminal, which is mapped to the execution of a series of application operations in the application run across the end by the first terminal, thereby improving the efficiency and convenience of cross-end interaction of applications.

[0101] Continue to see Figure 2 When the game player is ready to use special game skills in the game, the game player only needs to enter the pre-set shortcut command "1" through the keyboard, and the simulator will automatically execute a series of application operations indicated by the cross-end interaction sequence "DSAP" mapped to the shortcut command "1" to control the game character to perform "right, down, left, punch", thereby triggering the game character's special game skills in the game. Whether it is operations such as moving, shooting, and selecting menus in game-type applications, or input, editing, and navigation in office-type applications, operations can be completed quickly and accurately by entering shortcut commands on the PC. In this way, you can enjoy the efficient triggering of complex cross-end interaction sequences in mobile applications through customized shortcut commands on the PC, thereby improving the efficiency and convenience of cross-end interaction of applications.

[0102] ③ In addition, in the process of entering the cross-end interaction sequence, you can also customize the execution time interval between any two adjacent application operations; thus, when executing multiple application operations indicated by the cross-end interaction sequence, you can automatically execute the multiple application operations indicated by the cross-end interaction sequence in sequence according to the execution time interval to achieve the preset interaction speed effect, further improving the convenience and friendliness of the cross-end interaction of the application. For example, for the cross-end interaction sequence "SPD", you can customize the execution time interval between the application operation corresponding to the cross-end interaction instruction "S" and the application operation corresponding to the cross-end interaction instruction "P" to 0 seconds, and you can customize the execution time interval between the application operation corresponding to the cross-end interaction instruction "P" and the application operation corresponding to the cross-end interaction instruction "D" to 0.01 seconds; then, the application operation corresponding to the cross-end interaction instruction "S" and the application operation corresponding to the cross-end interaction instruction "P" can be executed simultaneously in the application being run across the ends, and the application operation corresponding to the cross-end interaction instruction "D" will be executed 0.01 seconds after the application operation corresponding to the cross-end interaction instruction "P" is executed.

[0103] The following introduces the execution subject of the cross-terminal interaction method. The cross-terminal interaction method can be executed by a computer device, which can be, for example, a first terminal of a first terminal type, and the first terminal of the first terminal type can be, for example, a PC terminal.

[0104] The first terminal may be in Figure 3 In the cross-end interactive system shown, the cross-end interactive system may include a first terminal 301 of a first terminal type and a server 302. A direct communication connection may be established between the first terminal 301 and the server 302 via wired communication, or an indirect communication connection may be established between the first terminal 301 and the server 302 via wireless communication. The embodiment of the present application does not limit the communication connection method between the first terminal 301 and the server 302.

[0105] In the cross-terminal interactive system composed of the first terminal 301 and the server 302, an application download client and a simulator are running in the first terminal 301, and the application download client in the first terminal 301 can call the simulator to run the application developed based on the second terminal type across the terminal. The server 302 can be used to provide background services for the application download client. The server can be a separate physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms, which are explained here.

[0106] The application cross-end interaction process corresponding to the cross-end interaction system may include: in the configuration stage of the application cross-end interaction, after the application download client in the first terminal 301 enters the cross-end interaction sequence and obtains the shortcut sequence assigned to the cross-end interaction instruction, the cross-end interaction sequence and the shortcut instruction assigned thereto may be sent to the server 302; the server 302 may map and store the cross-end interaction sequence and the shortcut instruction assigned thereto in the configuration file for cross-end operation. In the process of application cross-end interaction, after receiving the shortcut instruction input into the first terminal 301, the configuration file may be downloaded from the server 302, and the cross-end interaction sequence mapped to the shortcut instruction may be determined based on the mapping relationship recorded in the configuration file; the application download client in the first terminal 301 may call the simulator to execute multiple application operations indicated by the cross-end interaction sequence in the application being run across the end. It should be noted that after the cross-end interaction sequence is updated (for example, the application operations indicated by the cross-end interaction sequence are updated, and the execution order between the application operations indicated by the cross-end interaction sequence is updated, etc.), or the mapping relationship between the cross-end interaction sequence and the shortcut instructions is updated (for example, the shortcut instructions mapped to the cross-end interaction sequence are updated), the server 302 can adaptively update the configuration file, and the first terminal 301 can download the updated configuration file from the server 302.

[0107] Based on the cross-terminal interaction system, a single input operation of a shortcut command in the first terminal can trigger the automatic execution of a complex series of application operations in the application running across terminals in the first terminal, thereby improving the efficiency and convenience of cross-terminal interaction of applications. It can be understood that the cross-terminal interaction system described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. It is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0108] The cross-terminal interaction method provided in the embodiment of the present application is introduced in detail below.

[0109] The embodiment of the present application provides a cross-terminal interaction method, and the content introduced in the cross-terminal interaction method includes: the execution mode of multiple application operations indicated by the cross-terminal interaction sequence in the application being run across the terminals. The cross-terminal interaction method is executed by a computer device, and the computer device can be, for example, a first terminal of a first terminal type, and the first terminal can be, for example, a PC terminal. Figure 4 As shown, the cross-end interaction method may include but is not limited to the following steps S401 to S404:

[0110] S401, during the process of running the target application across terminals on the first terminal, obtaining a target shortcut command input into the first terminal; the terminal type of the first terminal is a first terminal type, the target application is developed based on a second terminal type, and the first terminal type and the second terminal type are two different terminal types.

[0111] In step S401, the first terminal of the first terminal type can support cross-terminal operation of multiple applications developed based on the second terminal type, and the target application can be any application supported by the first terminal for cross-terminal operation. The application download client in the first terminal can run the target application across terminals by calling the simulator.

[0112] The shortcut command may be a single key command, or may be a combination command obtained by combining multiple key commands. For example, the shortcut command may be a single key command "1", or the shortcut command may be a combination command "Ctrl+A" obtained by combining the key command "Ctrl" and the key command "A". The input method of the shortcut command may include but is not limited to at least one of the following: keyboard input, mouse input, game controller input, and voice input.

[0113] S402, obtaining a configuration file for cross-end operation, where the configuration file is used to record a mapping relationship between shortcut commands and cross-end interaction sequences.

[0114] In step S402, the configuration file for cross-terminal operation can be used to record the mapping relationship between the shortcut command and the cross-terminal interaction sequence. The configuration file can be stored locally in the first terminal, and the configuration file can be obtained locally from the first terminal; or the configuration file can be stored in a server corresponding to the application download client, and the configuration file can be downloaded from the server corresponding to the application download client; or the configuration file can be stored in a cloud storage, and the configuration file can be downloaded from the cloud storage.

[0115] S403: Determine a target cross-end interaction sequence mapped to the target shortcut instruction based on the mapping relationship, where the target cross-end interaction sequence is used to indicate a plurality of application operations executed in the target application being run across ends.

[0116] In step S403, based on the mapping relationship between the shortcut command and the cross-end interaction sequence, the target cross-end interaction sequence mapped to the target shortcut command is determined in the configuration file. The target cross-end interaction sequence can be used to indicate multiple application operations performed in the target application run across the end. The target cross-end interaction sequence may include multiple cross-end interaction instructions arranged in order, each cross-end interaction instruction corresponding to a respective application operation. The type of application operation may include, but is not limited to, at least one of the following: click type, slide type, and input type.

[0117] by Figure 2 For example, the target cross-end interaction sequence "DASP" mapped to the target shortcut instruction "1" includes 4 cross-end interaction instructions arranged in sequence. The type of application operation corresponding to the cross-end interaction instruction "D" is a click type, and the application operation is specifically clicking the right button to control the game character to move to the right; the type of application operation corresponding to the cross-end interaction instruction "A" is a click type, and the application operation is specifically clicking the left button to control the game character to move to the left; the type of application operation corresponding to the cross-end interaction instruction "S" is a click type, and the application operation is specifically clicking the down button to control the game character to move downward; the type of application operation corresponding to the cross-end interaction instruction "P" is a click type, and the application operation is specifically clicking the punch button to control the game character to punch. It can be seen that the application operations indicated by the target cross-end interaction sequence "DASP" are all click operations.

[0118] S404: Execute multiple application operations indicated by the target cross-end interaction sequence in the target application.

[0119] In step S404, the multiple application operations indicated by the target cross-end interaction sequence are executed in sequence. The sequential execution means that the target cross-end interaction sequence includes multiple cross-end interaction instructions arranged in sequence, and the application operation corresponding to each cross-end interaction instruction can be executed in sequence according to the arrangement order corresponding to each cross-end interaction instruction in the target cross-end interaction sequence.

[0120] In detail, the configuration file for cross-end operation may also include a sequence entry corresponding to the target cross-end interaction sequence, and the sequence entry may be used to instruct the target cross-end interaction sequence to execute, and the execution instruction refers to an instruction on the execution method of the target cross-end interaction sequence. The sequence entry may include application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence. In this case, the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence may be executed in sequence in the target application.

[0121] Further, the application operation is performed based on the corresponding execution coordinates in the display screen of the target application. Specifically, the sequence entry may also include the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the display screen of the target application. In this case, the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence may be sequentially executed in the display screen of the target application according to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence.

[0122] For example, for an application operation whose operation type is a click type (hereinafter referred to as a click operation), the execution coordinates corresponding to the click operation are single coordinates, and the click operation can be performed at the execution coordinates corresponding to the click operation in the display screen of the target application. For an application operation whose operation type is a sliding type (hereinafter referred to as a sliding operation), the execution coordinates corresponding to the sliding operation may include a starting coordinate and an ending coordinate, and the sliding operation may be performed between the starting and ending coordinates and the ending coordinates in the display screen of the target application. For an application operation whose operation type is an input type (hereinafter referred to as an input operation), the cross-end interaction instruction corresponding to the input operation may include the input target content, and the execution coordinates corresponding to the input operation may include the coordinates corresponding to the input box, and the input operation may be performed in the input box in the display screen of the target application, and the input content is the target content.

[0123] Optionally, the multiple application operations indicated by the target cross-end interaction sequence can be executed rhythmically in sequence according to the corresponding execution time. Specifically, the sequence entry may also include the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence. In this case, according to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, and according to the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, the application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence can be executed in sequence in the display screen of the target application. In this way, the execution time in the configuration file is determined according to the configured execution time interval, and the multiple application operations indicated by the cross-end interaction sequence can be executed in order according to the preset execution interval time when executed, and the preset execution speed can be achieved, thereby achieving the expected application cross-end interaction speed effect.

[0124] In the execution method of the application operation in the target cross-end interaction sequence introduced in the above step S404, the following two points need to be noted:

[0125] The first point is coordinate transformation.

[0126] The sequence entry may also include the reference display window information of the target application, and the execution coordinates recorded in the sequence entry are compatible with the reference display window information. The execution coordinates being compatible with the reference display window information means that the execution coordinates are coordinates in the coordinate system associated with the reference display window information; the coordinate system associated with the reference display window information refers to a coordinate system established based on the reference display window information. If the current display window information of the target application does not match the reference display window information of the target application, the execution coordinates recorded in the sequence entry are incompatible with the current display window information of the target application; in this case, it is necessary to convert the execution coordinates recorded in the sequence entry into converted coordinates that are compatible with the current display window information of the target application, and then perform the corresponding application operation based on the converted coordinates.

[0127] Among them, the display window information can be used to indicate the properties of the display window of the target application. For example, the display window information may include the size information of the display window of the target application, and the resolution information of the display window of the target application, etc. The reference display window information may include the default display window information, and the default display window information may be set according to the empirical value. Alternatively, the reference display window information may include the display window information when each cross-end interaction instruction in the target cross-end interaction sequence is entered.

[0128] Regarding the inclusion of the reference display window information, it should be noted that the display window information may change during the process of entering the target cross-end interaction sequence. For example, when entering one of the cross-end interaction instructions in the target cross-end interaction sequence, the display window size of the target application is enlarged. For another example, when entering another cross-end interaction instruction in the target cross-end interaction sequence, the display window resolution of the target application is adjusted. Therefore, the display window information when each cross-end interaction instruction in the target cross-end interaction sequence is entered can be recorded in the sequence entry as the reference display window information.

[0129] Taking any cross-end interaction instruction (which can be represented as a target interaction instruction) in the target cross-end interaction sequence as an example, the reference display window information may include the default display window information, or the display window information when the target interaction instruction is entered. The process of executing the application operation corresponding to the target interaction instruction in the display screen of the target application may include:

[0130] If the current display window information of the target application matches the reference display window information, the application operation corresponding to the target interaction instruction can be executed in the display screen of the target application according to the execution coordinates corresponding to the target interaction instruction. In this case, if the current display window information of the target application matches the reference display window information, it can be said that the execution coordinates corresponding to the target interaction instruction are compatible with the current display window information of the target application, and coordinate conversion may not be performed, and the application operation corresponding to the target interaction instruction can be executed based on the execution coordinates corresponding to the target interaction instruction. For example, if the reference display window information of the target application is the default display window information, and the current display window information of the target application is the default display window information, the application operation corresponding to the target interaction instruction can be executed in the display screen of the target application according to the execution coordinates corresponding to the target interaction instruction.

[0131] If the current display window information of the target application does not match the reference display window information, the execution coordinates corresponding to the target interaction instruction can be transformed to obtain the first transformed coordinates corresponding to the target interaction instruction; the application operation corresponding to the target interaction instruction can be executed in the display screen of the target application according to the first transformed coordinates corresponding to the target interaction instruction. In this case, if the current display window information of the target application does not match the reference display window information, it can be explained that the execution coordinates corresponding to the target interaction instruction are not compatible with the current display window information of the target application, and a coordinate transformation can be performed to convert them into the first transformed coordinates compatible with the current display window information of the target application, and the application operation corresponding to the target interaction instruction can be executed based on the first transformed coordinates. For example, if the reference display window information is the default display window information, and the current display window of the target application adjusts the window size based on the default display window, the execution coordinates corresponding to the target interaction instruction can be transformed, and according to the first transformed coordinates obtained by the coordinate transformation, the application operation corresponding to the target interaction instruction can be executed in the display screen of the target application.

[0132] In particular, the coordinate conversion method may include: determining a coordinate conversion ratio based on reference display window information and current display window information, and the coordinate conversion ratio may be, for example, equal to the ratio between the reference display window information and the current display window information; and converting the execution coordinates corresponding to the target interaction instruction into coordinates that are compatible with the current display window information based on the coordinate conversion ratio to obtain first converted coordinates, and the first converted coordinates may be, for example, equal to the coordinate conversion ratio divided by the execution coordinates corresponding to the target interaction instruction.

[0133] It can be seen that when the reference display window information matches the current display window information, the execution coordinates are the exact coordinates in the current display window of the target application; when the reference display window information does not match the current display window information, the execution coordinates can be converted into the exact coordinates in the current display window of the target application through coordinate conversion, so that the application operation can be accurately executed in the display screen of the target application based on the exact coordinates.

[0134] The second point is interactive transparent transmission.

[0135] In the process of cross-terminal interaction of applications, the target shortcut command input into the first terminal can be mapped to the target cross-terminal interaction sequence based on the mapping relationship between the shortcut command and the cross-terminal interaction sequence in the configuration file. Thus, the input operation of the target shortcut command in the first terminal can be converted into executing multiple application operations indicated by the target cross-terminal interaction sequence in the target application being run across terminals. It is worth noting that the simulator in the first terminal acts as a running container to run the target application across terminals, such as Figure 5 As shown, taking the case where the first terminal is a PC and the target application is a mobile application (for example, an Android application), the application download client in the first terminal runs on the operating system of the PC, and the APK (Android Application Package) of the target application needs to run on the Android OS (Android system, a mobile operating system) environment; therefore, the application download client host running on the operating system of the PC (the operating system of the PC can be understood as the host OS) needs to create a virtual machine of a mobile operating system (the mobile operating system can be understood as the guest OS) as a running container for the mobile application in order to run the mobile application, and runs the mobile application on the virtual machine and performs interactive operations. The virtual machine here is the simulator.

[0136] like Figure 5 As shown, the application operation can be transparently transmitted to the screen canvas of the mobile terminal application being run across terminals through the screen canvas interaction on the virtual machine container (i.e., the simulator). It can be further understood that the simulator in the first terminal provides a display window of the target application, which may include the simulator's screen canvas, and the display screen of the target application (i.e., the screen canvas of the target application) can be displayed on the simulator's screen canvas, and the display screen of the target application is the same size as the simulator's screen canvas; based on this, the application operation is executed at the corresponding execution coordinates on the simulator's screen canvas, and the execution of the application operation can be transparently transmitted to the same execution coordinates of the target application's display screen.

[0137] Taking any cross-end interaction execution in the target cross-end interaction sequence (which can be represented as a target interaction instruction) as an example, the execution process of the application operation corresponding to the target interaction instruction in the display screen of the target application may include: executing the application operation corresponding to the target interaction instruction on the screen canvas of the simulator; and transparently transmitting the execution of the application operation corresponding to the target interaction instruction to the display screen of the target application. A more detailed execution process may include: executing the application operation corresponding to the target interaction instruction on the screen canvas of the simulator according to the execution coordinates corresponding to the target interaction instruction and the execution time corresponding to the target interaction instruction, and transparently transmitting the execution of the application operation corresponding to the target interaction instruction to the display screen of the target application.

[0138] The above steps S401 to S404 introduce the execution process of cross-terminal interaction of applications. The following further introduces the execution process of cross-terminal interaction of applications in combination with the module design in the first terminal. Figure 6 As shown, the execution process of cross-end interaction of applications includes the following steps (1)-(7):

[0139] (1) Enter the target shortcut command:

[0140] Enter the target shortcut command in the first terminal.

[0141] (2) Configuration file for locating record mapping relationships:

[0142] File management system module: The first terminal can call the file management system module to locate the configuration file for cross-terminal operation. The configuration file includes a mapping relationship between shortcut commands and cross-terminal interaction sequences.

[0143] (3) Triggering cross-end interaction sequence:

[0144] Detection module: After the detection module in the first terminal detects the input target shortcut command, it triggers the target cross-terminal interaction sequence mapped to the target shortcut command in the configuration file.

[0145] Sequence recognition module: The sequence recognition module in the first terminal can recognize each cross-end interaction instruction in the target cross-end interaction sequence that is intended to be triggered, and can use each cross-end interaction instruction in the target cross-end interaction sequence as the input of the next module in the first terminal.

[0146] (4) Key-value pair search:

[0147] Mapping parser module: The mapping parser module in the first terminal can search for the key-value pairs corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the sequence entry corresponding to the target cross-end interaction sequence in the configuration file. The key-value pairs corresponding to each cross-end interaction instruction define the correspondence between the cross-end interaction instruction and the application operation (for example, click operation, sliding operation, and input operation, etc.), execution time and execution coordinates.

[0148] (5) Coordinate transformation:

[0149] Coordinate converter module: Since the screen size and resolution of the mobile terminal and the PC terminal may be different, the coordinate converter module in the first terminal can convert the execution coordinates in the configuration file into converted coordinates that are compatible with the simulator's screen canvas (i.e., the simulator's current display window information) according to the simulator's current settings and the target application's interface layout.

[0150] (6) Execute the application operations at the corresponding coordinates in order and time:

[0151] Operation executor module: The operation executor module in the first terminal can execute the corresponding application operation at the coordinates corresponding to each cross-end interaction instruction in turn according to the order and execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence. The application operation may include:

[0152] Simulate click operation: simulate mouse click or touch screen click at the specified coordinate position.

[0153] Simulate a sliding operation: Perform a sliding operation on the screen, which may involve starting and ending coordinates.

[0154] Simulate input operations: If the interaction sequence includes text input, you can simulate keyboard input operations.

[0155] (7) Complete the cross-end interaction sequence trigger:

[0156] After the application operations corresponding to each cross-end interaction instruction in the cross-end interaction sequence are executed, a notification can be issued to notify that the cross-end interaction sequence is triggered. The notification can be in the form of a log, that is, the log can be printed after the cross-end interaction sequence is triggered.

[0157] In an embodiment of the present application, an input operation of a shortcut command in the first terminal can be quickly mapped to execute a series of application operations in an application run across terminals by the first terminal, which can improve the efficiency of cross-terminal and application interaction in the first terminal; and the input operation of the shortcut command is convenient, which can improve the convenience of cross-terminal and application interaction in the first terminal. Each cross-terminal interaction instruction in the cross-terminal interaction sequence has an execution order, and each cross-terminal interaction instruction corresponds to an execution time, so that each cross-terminal interaction instruction in the cross-terminal interaction sequence can be executed in sequence and on time to achieve the target interaction speed effect. The application operation corresponding to each cross-terminal interaction instruction in the cross-terminal interaction sequence can be executed according to the coordinates corresponding to each cross-terminal interaction instruction, and the coordinates are always adapted to the current display window information of the target application, so that the application operation can be ensured to be executed at the correct location.

[0158] The embodiment of the present application provides a cross-terminal interaction method, and the content introduced in the cross-terminal interaction method includes: the configuration process of applying cross-terminal interaction. The cross-terminal interaction method is executed by a computer device, and the computer device can be, for example, a first terminal of a first terminal type, and the first terminal can be, for example, a PC terminal. Figure 7 As shown, the cross-end interaction method may include but is not limited to the following steps S701 to S707:

[0159] S701, obtaining an input target cross-end interaction sequence.

[0160] In step S701, the target cross-end interaction sequence may be custom-entered by the interactor in the application download client in the first terminal. The target cross-end interaction sequence may be entered in any of the following ways: entering the target cross-end interaction sequence in a key position editing way, entering the target cross-end interaction sequence in an operation recording way, entering the target cross-end interaction sequence in a voice entry way, and entering the target cross-end interaction sequence in a natural language entry way.

[0161] Among them, using key editing to enter the target cross-end interaction sequence means: editing key instructions, using the edited key instructions as cross-end interaction instructions, and generating the target cross-end interaction sequence according to the edited multiple cross-end interaction instructions. Using operation recording to enter the target cross-end interaction sequence means: recording the interactive operations performed on the target application, and generating the target cross-end interaction sequence according to the cross-end interaction instructions mapped to the interactive operations. Using voice entry to enter the target cross-end interaction sequence means: entering the cross-end interaction instructions by voice, and generating the target cross-end interaction sequence according to the cross-end interaction instructions entered by voice. Using natural language entry to enter the target cross-end interaction sequence means: identifying the cross-end interaction instructions from the entered natural language, and generating the target cross-end interaction sequence according to the identified cross-end interaction instructions.

[0162] S702: Allocate a target shortcut command for the target cross-end interaction sequence.

[0163] In step S702, after the target cross-end interaction sequence is entered, a customized target shortcut instruction may be allocated to the target cross-end interaction sequence, and the allocated target shortcut instruction is used as a shortcut instruction mapped to the target cross-end interaction sequence.

[0164] S703: Map and store the target cross-end interaction sequence and the target shortcut command into a configuration file.

[0165] In step S703, mapping storage refers to storing the mapping relationship between the target cross-end interaction sequence and the target shortcut command in a configuration file.

[0166] In addition to storing the mapping relationship between the cross-end interaction sequence and the shortcut command, the configuration file can also store the sequence entries corresponding to the cross-end interaction sequence. For the target cross-end interaction sequence, a sequence entry corresponding to the target cross-end interaction sequence can be generated, and the sequence entry can be used to perform execution instructions on the target cross-end interaction sequence, and the sequence entry can be stored in the configuration file. Among them, the execution instruction means that the sequence entry can include the execution information of multiple application operations indicated in the target cross-end interaction sequence (for example, execution order, execution coordinates, and execution time, etc.), so that the multiple application operations indicated by the target cross-end interaction sequence can be accurately executed based on the execution information.

[0167] In detail, the method for generating sequence entries corresponding to the target cross-end interaction sequence may include: obtaining the execution time interval between any two adjacent cross-end interaction instructions in the target cross-end interaction sequence, and determining the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence according to the execution time interval; obtaining the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence; obtaining the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence; and generating the sequence entry corresponding to the target cross-end interaction sequence according to the execution time, execution coordinates and application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence.

[0168] In the above-mentioned method of generating sequence entries, the first point worth noting is that the execution time interval between any two adjacent cross-end interaction instructions in the target cross-end interaction sequence can be customized by the interactor when entering the target cross-end interaction sequence. In other words, the embodiment of the present application supports customizing the execution time interval between the application operations corresponding to any two adjacent cross-end interaction instructions in the target cross-end interaction sequence, and the execution time interval enables the multiple application operations indicated by the target cross-end interaction sequence to have a preset execution speed; thus, when the target cross-end interaction sequence is triggered, the multiple application operations indicated by the target cross-end interaction sequence can be automatically executed at the preset execution speed.

[0169] In the above-mentioned method of generating sequence entries, the second point worth noting is that the cross-end interaction instruction can include any one of a key instruction or a trigger instruction; a key instruction refers to an instruction input through a key input operation, and a trigger instruction refers to an instruction generated by a trigger operation. For a key instruction, there is a mapping relationship between the key instruction and the application operation, and the application operation corresponding to the key instruction can be obtained based on the mapping relationship between the key instruction and the application operation; and for a trigger instruction, the trigger instruction corresponds to the trigger operation.

[0170] In the above-mentioned method of generating sequence entries, it should also be noted that the sequence entries can also include the reference display window information of the target application, and the execution coordinates in the sequence entries are adapted to the reference display window information of the target application. Therefore, if the acquired coordinates are not adapted to the reference display window information of the target application, a coordinate conversion can be performed to convert the coordinates that are not adapted to the reference display window information of the target application into coordinates that are adapted to the reference display window information of the target application, and a sequence entry can be generated based on the coordinates that are adapted to the reference display window information of the target application.

[0171] Taking any cross-end interaction instruction (which can be expressed as a target interaction instruction) in the target cross-end interaction sequence as an example, the process of obtaining the execution coordinates corresponding to the target interaction instruction may include: obtaining the reference coordinates corresponding to the target interaction instruction; if the reference coordinates corresponding to the target interaction instruction are compatible with the reference display window information of the target application, the reference coordinates corresponding to the target interaction instruction may be determined as the execution coordinates corresponding to the target interaction instruction; if the reference coordinates corresponding to the target interaction instruction are not compatible with the reference display window information of the target application, the reference coordinates corresponding to the target interaction instruction may be converted to obtain a second converted coordinate that is compatible with the reference display window information of the target application, and the second converted coordinates may be determined as the execution coordinates corresponding to the target interaction instruction. Wherein, if the target interaction instruction is a key position instruction, there is a mapping relationship between the key position instruction and the reference coordinates, and the reference coordinates corresponding to the target interaction instruction may be obtained based on the mapping relationship between the target interaction instruction and the reference coordinates; if the target interaction instruction is a trigger instruction, the coordinates triggered by the trigger operation of the trigger instruction may be generated as the reference coordinates corresponding to the target interaction instruction.

[0172] For example, the reference display window information of the target application stored in the sequence entry is the default display window information. If the target interaction instruction is a key instruction, the reference coordinates corresponding to the target interaction instruction are adapted to the default display window information. Then it can be said that the reference coordinates corresponding to the target interaction instruction are adapted to the reference display window information of the target application. The reference coordinates corresponding to the target interaction instruction can be determined as the execution coordinates corresponding to the target interaction instruction and stored in the sequence entry.

[0173] For another example, if the reference display window information of the target application stored in the sequence entry is the default display window information, if the reference coordinates corresponding to the target interaction instruction are the click coordinates of the display screen of the target application, and the display window of the target application is enlarged relative to the default display window of the target application when the target interaction instruction is entered, it can be explained that the reference coordinates corresponding to the target interaction instruction do not match the reference display window information of the target application. In this case, the reference coordinates corresponding to the target interaction instruction can be converted according to the reference display window information of the target application and the display window information of the target application when the coordinates are clicked, and the reference coordinates that are not compatible with the reference display window information of the target application are converted into second converted coordinates that are compatible with the reference display window information of the target application, and the second converted coordinates are determined as the execution coordinates corresponding to the target interaction instruction.

[0174] It can be seen that the execution coordinates stored in the sequence entries are all coordinates that are compatible with the reference display window information of the target application. This makes the execution coordinates stored in the sequence entries have a unified coordinate standard, which is conducive to accurately executing multiple application operations indicated by the target cross-end interaction sequence.

[0175] S704, during the process of the first terminal running the target application across terminals, obtaining a target shortcut command input into the first terminal; the terminal type of the first terminal is a first terminal type, the target application is developed based on a second terminal type, and the first terminal type and the second terminal type are two different terminal types.

[0176] In the embodiment of the present application, the execution process of step S704 is the same as the above Figure 4 The execution process of step S401 in the embodiment shown is the same, and the details can be referred to above. Figure 4 The relevant description of step S401 in the illustrated embodiment will not be repeated here.

[0177] S705, obtaining a configuration file for cross-end operation, where the configuration file is used to record a mapping relationship between shortcut commands and cross-end interaction sequences.

[0178] In the embodiment of the present application, the execution process of step S705 is the same as that of the above Figure 4 The execution process of step S402 in the embodiment shown is the same, and the details can be referred to above. Figure 4 The relevant description of step S402 in the illustrated embodiment will not be repeated here.

[0179] S706: Determine a target cross-end interaction sequence mapped to the target shortcut instruction based on the mapping relationship, where the target cross-end interaction sequence is used to indicate multiple application operations executed in the target application being run across ends.

[0180] In the embodiment of the present application, the execution process of step S706 is the same as that of the above Figure 4 The execution process of step S403 in the embodiment shown is the same, and the details can be referred to above. Figure 4 The relevant description of step S403 in the illustrated embodiment will not be repeated here.

[0181] S707: Execute multiple application operations indicated by the target cross-end interaction sequence in the target application.

[0182] In the embodiment of the present application, the execution process of step S707 is the same as that of the above Figure 4 The execution process of step S404 in the embodiment shown is the same, and the details can be referred to above. Figure 4 The relevant description of step S404 in the illustrated embodiment will not be repeated here.

[0183] In the embodiment of the present application, the cross-end interaction sequence is custom-entered, and the shortcut instructions mapped to the cross-end interaction sequence are custom-assigned, which makes the configuration process of the cross-end interaction of the application flexible and diverse. Based on this custom configuration process, the interactor can customize the cross-end interaction sequence and the shortcut instructions mapped to the cross-end interaction sequence according to his own cross-end interaction habits, so that the cross-end interaction of the application can be performed based on the custom configuration, which improves the convenience of the cross-end interaction of the application, so that the cross-end interaction of the application can better meet the cross-end interaction habits of the interactor.

[0184] Above Figure 7 The illustrated embodiment introduces a unified configuration process of application start-end interaction under different entry modes of cross-end interaction sequences. The following introduces the detailed configuration process of application cross-end interaction under different entry modes in combination with different entry modes of cross-end interaction sequences:

[0185] 1. Use key editing to enter the target cross-end interaction sequence:

[0186] The key editing mode refers to: editing key instructions, using the edited key instructions as cross-end interaction instructions, and generating a target cross-end interaction sequence based on the edited multiple cross-end interaction instructions. In this mode, editing key instructions refers to inputting key instructions, and the edited key instructions are key instructions with existing interaction mappings; key instructions with existing interaction mappings refer to key instructions that can be mapped to application operations performed on applications during the cross-end interaction of applications.

[0187] 1. The interface of the first terminal is presented as follows:

[0188] (1) Start the application download client on the first terminal, open the target application running across terminals in the application download client, and then open the editing function of the cross-terminal interactive mapping document, such as Figure 8As shown, the editing control 801 of the document can be mapped through cross-end interaction (for example, Figure 8 The floating ball in the figure opens the editing function of the cross-end interactive mapping document, and displays the editing interface 80 of the cross-end interactive mapping document.

[0189] (2) After opening the editing function of the cross-end interaction mapping document, you can enter the target cross-end interaction sequence in the editing interface of the cross-end interaction mapping document, assign shortcut commands to the target cross-end interaction sequence, and customize the execution time interval between any two adjacent cross-end interaction commands in the target cross-end interaction sequence. Fig. 9 As shown, the target cross-end interaction sequence (for example, Fig. 9 DSAP in the ), you can assign a target shortcut command to the edited target cross-end interaction sequence (for example, Fig. 9 ”1” in the figure).

[0190] 2. Steps and module calls involved in the first terminal:

[0191] like Fig.10 As shown, in the process of entering the target cross-terminal interaction instruction in a key position editing mode, the steps and module calls involved in the first terminal may include:

[0192] (1) Create the target cross-end interaction sequence:

[0193] Sequence initialization module: This module can be used to start the editing function of the cross-end interaction mapping document and choose to create a new cross-end interaction sequence. This module is responsible for initializing a new configuration file.

[0194] User Interface Design (UI) module: This module can provide an interface for the interactor to input and configure the target cross-end interaction sequence. For example, the target cross-end interaction sequence can be input by inputting key commands, and the sequence description information of the target cross-end interaction sequence can be input. The sequence description information can include, for example, the name and function of the target cross-end interaction sequence.

[0195] (2) Find the key position:

[0196] Key scanning module: This module can scan the key mapping table, which can store the mapping relationship between the key instructions of the existing interactive mapping and the execution coordinates, as well as the mapping relationship between the key instructions and the application operations, and map the input key instructions to the specific coordinates on the simulator screen, and map them to the specific application operations.

[0197] Key mapping database module: This module can be used to store key mapping tables, that is, it can be used to store existing interactively mapped key instructions and their mapped execution coordinates and application operations.

[0198] (3) Edit the arrangement order of the cross-end interaction instructions in the target cross-end interaction sequence and the execution time interval between any two adjacent cross-end interaction instructions:

[0199] Sequence Editor Module: This module allows the user to customize the order of multiple cross-end interaction instructions in the target cross-end interaction sequence, ensuring that each cross-end interaction instruction is executed in the correct order and logic. In addition, this module also allows the user to customize the execution time interval between any two adjacent key instructions to simulate the operation rhythm of the user's real interaction.

[0200] Time manager module: This module can be used to manage the execution time interval between any two adjacent cross-end instructions.

[0201] (4) Generate sequence entries corresponding to the target cross-end interaction sequence:

[0202] Sequence storage module: This module can generate sequence entries corresponding to the target cross-end interaction sequence according to the application operation, execution time, and execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence; the sequence entries can be saved in the configuration file for subsequent use. This module can also store the mapping relationship between the target cross-end interaction sequence and the target shortcut instruction in the configuration file for subsequent use.

[0203] The specific operation is that after the interactor confirms that the target cross-end interaction sequence and the target shortcut command mapped thereto are correct, select "Save". The sequence storage module stores the generated sequence entries and the mapping relationship between the target cross-end interaction sequence and the target shortcut command in the configuration file, and saves the configuration file to the local database, server or cloud storage.

[0204] Through the above steps and module calls, a preset cross-end interaction sequence can be created through key editing. When the cross-end interaction sequence is triggered during the cross-end interaction of the application, it can be used in the simulator to automatically execute complex application operations across ends, thereby realizing the automation of cross-end interaction of complex applications, which is conducive to improving the efficiency and convenience of cross-end interaction of applications.

[0205] 2. Use operation recording to record the target cross-end interaction sequence:

[0206] The operation recording method refers to: recording the interactive operations performed on the target application, the interactive operations may include at least one of the key input operations and the trigger operations on the display screen of the target application; the interactive operations may be mapped to corresponding cross-end interactive instructions, and the cross-end interactive instructions may include at least one of the key instructions corresponding to the key input operations and the trigger instructions corresponding to the trigger operations; and the target cross-end interactive sequence may be generated according to the cross-end interactive instructions corresponding to the interactive operations.

[0207] 1. The interface of the first terminal is presented as follows:

[0208] like Fig.11 As shown, the editing function of the cross-end interactive mapping document is opened, and the editing interface 80 of the cross-end interactive mapping document can be displayed. Multiple interactive operations performed on the target application can be recorded. Each interactive operation (for example, Fig.11 The recorded interactive operation is the operation of inputting the key command "DSAP", and the cross-end interactive instructions corresponding to the interactive operation constitute the target cross-end interactive sequence; a target shortcut instruction can be assigned to the recorded target cross-end interactive sequence (for example, Fig.11 ”1” in the figure).

[0209] 2. Steps and module calls involved in the first terminal:

[0210] like Fig.12 As shown, in the process of recording the target cross-terminal interaction instruction in the operation recording mode, the steps and module calls involved in the first terminal may include:

[0211] (1) Create the target cross-end interaction sequence:

[0212] Sequence manager module (may include a sequence initialization module and an interface design module): This module may be responsible for initializing a new cross-end interaction sequence (i.e., starting the process of creating a new cross-end interaction sequence), and may provide an interface to configure a target cross-end interaction sequence, for example, inputting sequence description information of the target cross-end interaction sequence.

[0213] (2) Interactive operation recording:

[0214] Recorder module: This module can be used to capture the interactive operations of the interactor on the target application on the simulator. The interactive operations can include at least one of trigger operations (e.g., mouse click operations) and key input operations (e.g., key operations on the keyboard). The recorder module can record the order, coordinates, and operation types of these interactive operations.

[0215] (3) Find the key position:

[0216] Key mapper module: This module can be used to map key input operations in interactive operations to specific key commands. The keyboard input operations captured by the recorder module are sent to the key mapper module. The key mapper module can identify and record the key code (i.e. key command) of each keyboard key.

[0217] (4) Edit the arrangement order of the cross-end interaction instructions in the target cross-end interaction sequence and the execution time interval between any two adjacent cross-end interaction instructions:

[0218] Sequence Editor Module: This module allows the user to customize the order of multiple cross-end interaction instructions in the target cross-end interaction sequence, ensuring that each cross-end interaction instruction is executed in the correct order and logic. In addition, this module also allows the user to customize the execution time interval between any two adjacent key instructions to simulate the operation rhythm of the user's real interaction.

[0219] Time manager module: This module can be used to manage the execution time interval between any two adjacent cross-end instructions.

[0220] (5) Generate sequence entries corresponding to the target cross-end interaction sequence:

[0221] Coordinate converter module: This module can be used to convert the coordinates triggered by the trigger operation (which can be called trigger coordinates) into a coordinate system that the simulator can understand, that is, to convert them into precise coordinate positions on the simulator's screen canvas, and to coordinates that match the reference display window information of the target application, as the execution coordinates corresponding to the cross-end interaction instructions mapped by the trigger operation.

[0222] Operation sequence generator module: This module can generate sequence entries corresponding to the target cross-end interaction sequence according to the application operation, execution time, and execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence.

[0223] (6) Save the target cross-end interaction sequence:

[0224] Sequence storage module: This module can store the generated sequence entries and the mapping relationship between the target cross-end interaction sequence and the target shortcut instructions in a configuration file, and save the configuration file to a local database, server or cloud storage.

[0225] Through the above steps and module calls, a preset cross-end interaction sequence can be created by operation recording. When the cross-end interaction sequence is triggered during the cross-end interaction of the application, it can be used in the simulator to automatically execute complex application operations across ends, thereby realizing the automation of cross-end interaction of complex applications, which is conducive to improving the efficiency and convenience of cross-end interaction of applications.

[0226] 3. Use natural language input to enter the target cross-end interaction sequence:

[0227] The natural language input method refers to: identifying cross-end interaction instructions from the input natural language, and generating a target cross-end interaction sequence according to the identified cross-end interaction instructions.

[0228] 1. The interface of the first terminal is presented as follows:

[0229] (1) Input natural language:

[0230] Start the application download client on the first terminal, open the target application running across terminals in the application download client, and then open the editing function of the cross-terminal interaction mapping document. Fig.13 As shown, an editing interface 80 of a cross-end interactive mapping document may be displayed, and natural language may be entered in a natural language entry area 130. The manner of entering natural language may include any of the following:

[0231] ① Importing natural language documents: In this method, the document import function can be triggered through the document import control 1301, and the imported natural language document can be selected. The natural language in the natural language document can be used to describe one or more cross-end interaction sequences. In the embodiment of the present application, the document format of the natural language document supported for import is not limited. For example, the document format supported for import may include at least one of the following: PDF (Portable Document Format), and Word format (a document format), etc.

[0232] ② Input natural language text: Natural language text can be input in the natural language editing area 1302. The natural language text can be used to describe one or more cross-end interaction sequences.

[0233] (2) Identify cross-end interaction sequences based on AI (Artificial Intelligence) technology:

[0234] The first terminal may parse the input natural language (eg, natural language in a natural language document, or natural language in a natural language text) based on artificial intelligence technology, and extract one or more cross-terminal interaction sequences.

[0235] (3) Generate cross-end interaction guidance documents:

[0236] The first terminal may generate a cross-end interaction guidance document based on the mapping relationship between the cross-end interaction sequence and the shortcut command. The interactor may quickly view the mapping relationship between the cross-end interaction sequence and the shortcut command through the cross-end interaction guidance document, so as to facilitate cross-end interaction guidance for the interactor.

[0237] (4) Confirm the mapping relationship:

[0238] The interactor can review the mapping relationship between the refined cross-end interaction sequence and the shortcut commands, and make necessary adjustments to ensure that the mapping is accurate.

[0239] (5) Save the mapping configuration:

[0240] After the interactor confirms that it is correct, the mapping relationship between the cross-end interaction sequence and the shortcut command is stored in the configuration file. The configuration file can be used to quickly find the cross-end interaction sequence mapped to the input shortcut command during the cross-end interaction of the application.

[0241] 2. Steps and module calls involved in the first terminal:

[0242] The process of entering the target cross-end interaction sequence by natural language input may include: classifying and identifying the input natural language to obtain one or more interaction description sentences in the natural language, and any interaction description sentence in the natural language may be represented as the target sentence. For the target sentence, semantic recognition processing may be performed on the target sentence to obtain multiple operation description information; each operation description information may be mapped to a corresponding cross-end interaction instruction; and the target cross-end interaction sequence may be generated according to the cross-end interaction instruction corresponding to each operation description information.

[0243] In detail, Fig.14 As shown, in the process of inputting the target cross-terminal interaction instruction in a natural language input manner, the steps and module calls involved in the first terminal may include:

[0244] (1) Importing a natural language document or inputting a natural language text: This step may include inputting a natural language in the form of a document, text, etc. into the first terminal.

[0245] (2) Extracting and classifying interaction description sentences: This step involves using natural language processing technology to identify and classify interaction description sentences. An interaction description sentence can be used to describe the application operation indicated by a cross-end interaction sequence. An interaction description sentence can also be used to describe the function of a cross-end interaction sequence (for example, special game skills).

[0246] (3) The artificial intelligence analysis module performs semantic recognition processing: This module can use artificial intelligence technology (e.g., machine learning models) to analyze and understand the input natural language content, that is, to perform semantic recognition processing on the interaction description statement to obtain a semantic recognition processing result. The semantic recognition processing result may include multiple operation description information in the interaction description statement, as well as function description information; the operation description information may be used to describe the function of the application operation indicated by the cross-end interaction sequence, and the function description information may be used to describe the function of the cross-end interaction sequence.

[0247] (4) Generate entry template: Based on the semantic recognition processing results, an entry template corresponding to each interaction description sentence can be generated. The entry template corresponding to the interaction description sentence can include multiple operation description information in the interaction description sentence, as well as function description information.

[0248] (5) Generate a key-value (KV) format description based on the entry template: The storage format of the entry template corresponding to the interaction description statement can be converted and stored in a key-value pair format. The function description information can be used as the key and multiple operation description information can be used as the value. This key-value pair storage format facilitates subsequent processing and query.

[0249] (6) Mapping to cross-end interaction instructions: This step can map each operation description information in the entry template to a corresponding cross-end interaction instruction based on the mapping relationship between the operation description information and the cross-end interaction instruction. For example, if the operation description information is "up" and the mapped key instruction is "W", the mapped key instruction "W" can be used as the cross-end interaction instruction. The mapping between the operation description information and the cross-end interaction instruction can be based on a mapping table or a mapping algorithm to quickly map the operation description information to the cross-end interaction instruction.

[0250] (7) Creating a cross-end interaction sequence: For any entry template (which can be represented as a target entry template), this step can generate a target cross-end interaction sequence based on the cross-end interaction instructions corresponding to each operation description information in the target entry template.

[0251] (8) Deduplication module removes duplicate cross-end interaction sequences: This module can be used to remove duplicate cross-end interaction sequences to ensure the uniqueness and integrity of cross-end interaction sequences.

[0252] (9) Obtaining the execution coordinates corresponding to the cross-end interaction instruction in the target cross-end interaction sequence: Based on the mapping relationship between the cross-end interaction instruction in the target cross-end interaction sequence and the reference coordinates, the reference coordinates corresponding to the cross-end interaction instruction can be obtained; if the reference coordinates are compatible with the reference display window information, the reference coordinates corresponding to the cross-end interaction instruction can be determined as the execution coordinates corresponding to the cross-end interaction instruction; if the reference coordinates are not compatible with the reference display window information, the reference coordinates corresponding to the cross-end interaction instruction can be converted into coordinates compatible with the reference display window information to obtain the execution coordinates corresponding to the cross-end interaction instruction.

[0253] (10) Editing order and execution time interval: The interactor can customize the order of the cross-end interaction instructions in the target cross-end interaction sequence, as well as the execution time interval between the application operations corresponding to any two adjacent cross-end interaction sequences.

[0254] (11) Target cross-end interaction sequence after adjusting the order and time interval: After editing the order and execution time interval, the system may generate a new target cross-end interaction sequence, which has been adjusted according to the optimal order and execution time interval. The first terminal may determine the execution time of the application operation corresponding to each cross-end interaction instruction in the cross-end interaction sequence according to the execution time interval between the application operations corresponding to any two adjacent cross-end interaction sequences in the target cross-end interaction sequence.

[0255] (12) Generate sequence entries corresponding to the cross-end interaction sequence: Based on the creation of the target cross-end interaction sequence, a sequence entry corresponding to the target cross-end interaction sequence may also be generated. The sequence entry may include execution information corresponding to each cross-end interaction instruction in the cross-end interaction sequence, and the execution information may include at least one of the following: application operation, execution time of the application operation, execution coordinates of the application operation, and execution order of the application operation.

[0256] (13) Save the target cross-end interaction sequence:

[0257] Sequence storage module: This module can store the generated sequence entries and the mapping relationship between the target cross-end interaction sequence and the target shortcut instructions in a configuration file, and save the configuration file to a local database, server or cloud storage.

[0258] Through the above steps and module calls, a preset cross-end interaction sequence can be created by natural language input. When the cross-end interaction sequence is triggered during the cross-end interaction of the application, it can be used in the simulator to automatically execute complex application operations across ends, thereby realizing the automation of cross-end interaction of complex applications, which is conducive to improving the efficiency and convenience of cross-end interaction of applications.

[0259] 4. Use voice input to input the target cross-end interaction sequence:

[0260] The voice input method refers to: inputting cross-end interaction instructions through voice, and generating a target cross-end interaction sequence according to the cross-end interaction instructions input by voice.

[0261] 1. The interface of the first terminal is presented as follows:

[0262] Start the application download client on the first terminal, open the target application running across terminals in the application download client, and then enable the function of editing the cross-terminal interaction mapping document. Fig.15 As shown, an editing interface 80 of a cross-end interaction mapping document may be displayed, and a voice may be input through a voice input control 150, wherein the input voice refers to a voice including a cross-end interaction instruction, or the input voice refers to a voice including a natural language.

[0263] In the case where the input voice refers to voice including cross-end interaction instructions, the target cross-end interaction sequence composed of the cross-end interaction instructions can be directly displayed, the arrangement order between each cross-end interaction instruction in the target cross-end interaction sequence can be adjusted, and the execution time interval between the application operations corresponding to any two adjacent cross-end interaction instructions can be customized.

[0264] In the case where the input speech refers to speech including natural language, speech recognition processing can be performed on the natural language speech, and the recognized natural language text can be output, and the natural language text can be adjusted. The processing of the natural language text can be specifically referred to the above description of the natural language input method for entering the cross-end interaction sequence, which will not be repeated here.

[0265] 2. Steps and module calls involved in the first terminal:

[0266] In the case where the input voice refers to voice including cross-terminal interaction instructions, the steps and module calls involved in the first terminal can refer to the steps and module calls involved in the first terminal in the above-mentioned key editing method, which will not be repeated here.

[0267] When the input voice refers to a voice including natural language, the steps and module calls involved in the first terminal can refer to the steps and module calls involved in the first terminal in the above-mentioned natural language input method, which will not be repeated here.

[0268] Through the above steps and module calls, a preset cross-end interaction sequence can be created by voice input. When the cross-end interaction sequence is triggered during the cross-end interaction of the application, it can be used in the simulator to automatically execute complex application operations across ends, thereby realizing the automation of complex cross-end interaction of applications, which is conducive to improving the efficiency and convenience of cross-end interaction of applications.

[0269] In summary, if Fig.16 As shown, the entry process of the cross-end interaction sequence can be summarized as the following flow: an application download client can be started in the first terminal, a target application running across ends can be started in the application download client, and the editing function of the cross-end interaction mapping document can be opened; an entry method for the cross-end interaction sequence can be selected, and the target cross-end interaction sequence can be entered according to the selected entry method (for example, the selected entry method can be any one of a key editing method, an operation recording method, a natural language entry method, and a voice entry method); a target shortcut instruction can be assigned to the target cross-end interaction sequence, and the target cross-end interaction sequence and the target shortcut instruction mapping are stored in a configuration file; a sequence entry corresponding to the target cross-end interaction sequence can be generated, and the sequence entry corresponding to the target cross-end interaction sequence is stored in the configuration file.

[0270] In the process of entering the above cross-end interaction sequence, it can be seen that the embodiment of the present application supports the selection of the entry method, and the interactor can choose the entry method as needed to improve the entry experience of the cross-end interaction sequence. In addition, the interactor can also observe the execution of the cross-end interaction sequence during the cross-end interaction process. If the execution of the cross-end interaction sequence does not meet the target expectations, the interactor can reopen the editing function of the cross-end interaction mapping document and adjust the cross-end interaction sequence.

[0271] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0272] See also Fig.17 , Fig.17 It is a structural schematic diagram of a cross-end interaction device provided in an embodiment of the present application. The cross-end interaction device can be set in the computer device provided in the embodiment of the present application. The computer device can be a first terminal of a first terminal type. For example, the first terminal can be a PC terminal. Fig.17 The cross-end interactive device shown may be a computer program running in a computer device, and the cross-end interactive device may be used to execute Figure 4 or Figure 7 Some or all of the steps in the method embodiment shown. Fig.17 , the cross-end interaction device may include the following units:

[0273] An acquiring unit 1701 is configured to acquire a target shortcut command input into the first terminal during the process of the first terminal running the target application across terminals; the terminal type of the first terminal is a first terminal type, the target application is developed based on a second terminal type, and the first terminal type and the second terminal type are two different terminal types;

[0274] The acquisition unit 1701 is further used to acquire a configuration file for cross-end operation, where the configuration file is used to record a mapping relationship between shortcut commands and cross-end interaction sequences;

[0275] The processing unit 1702 is used to determine a target cross-end interaction sequence mapped to the target shortcut instruction based on the mapping relationship, where the target cross-end interaction sequence is used to indicate multiple application operations executed in the target application being run across the ends;

[0276] The processing unit 1702 is further configured to execute, in the target application, multiple application operations indicated by the target cross-end interaction sequence.

[0277] In one implementation, the target cross-end interaction sequence includes a plurality of cross-end interaction instructions arranged in sequence; the processing unit 1702 is configured to execute the following steps when executing the plurality of application operations indicated by the target cross-end interaction sequence in the target application:

[0278] Obtaining a sequence entry corresponding to a target cross-end interaction sequence in a configuration file; the sequence entry is used to instruct execution of the target cross-end interaction sequence, and the sequence entry includes application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence;

[0279] The application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence are executed in sequence in the target application.

[0280] In one implementation, the sequence entry also includes execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the display screen of the target application;

[0281] The processing unit 1702 is used to sequentially execute the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the target application, specifically to perform the following steps:

[0282] According to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, the application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence are sequentially executed in the display screen of the target application.

[0283] In one implementation, the sequence entry also includes the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence;

[0284] The processing unit 1702 is configured to sequentially execute the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the display screen of the target application according to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, and specifically to perform the following steps:

[0285] According to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence and according to the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence is executed sequentially in the display screen of the target application.

[0286] In one implementation, the sequence entry also includes reference display window information of the target application; any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; the processing unit 1702 is used to execute the following steps when executing the application operation corresponding to the target interaction instruction in the display screen of the target application:

[0287] If the current display window information of the target application matches the reference display window information, then according to the execution coordinates corresponding to the target interaction instruction, the application operation corresponding to the target interaction instruction is executed in the display screen of the target application;

[0288] If the current display window information of the target application does not match the reference display window information, coordinate transformation is performed on the execution coordinates corresponding to the target interaction instruction, and the application operation corresponding to the target interaction instruction is performed in the display screen of the target application according to the first transformed coordinates obtained by the coordinate transformation;

[0289] The reference display window information includes default display window information, or display window information when entering a target interaction instruction.

[0290] In one implementation, the execution coordinates corresponding to the target interaction instruction are adapted to the reference display window information; the processing unit 1702 is used to perform coordinate conversion on the execution coordinates corresponding to the target interaction instruction, specifically to perform the following steps:

[0291] Determine the coordinate conversion ratio according to the reference display window information and the current display window information;

[0292] According to the coordinate conversion ratio, the execution coordinates corresponding to the target interactive instruction are converted into coordinates adapted to the current display window information to obtain first converted coordinates.

[0293] In one implementation, any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; the simulator in the first terminal serves as a running container to run the target application across the terminals; the processing unit 1702 is used to execute the application operation corresponding to the target interaction instruction in the display screen of the target application, specifically to execute the following steps:

[0294] Execute the application operation corresponding to the target interaction instruction on the screen canvas of the simulator;

[0295] The execution of the application operation corresponding to the target interaction instruction is transparently transmitted to the display screen of the target application.

[0296] In one implementation, the obtaining unit 1701 is further configured to perform the following steps:

[0297] Obtain the entered target cross-end interaction sequence;

[0298] The processing unit 1702 is further configured to perform the following steps:

[0299] Assign target shortcuts to target cross-end interaction sequences;

[0300] Map the target cross-end interaction sequence and the target shortcut command and store them in the configuration file;

[0301] Among them, entering the target cross-end interaction sequence includes any of the following: entering the target cross-end interaction sequence by key editing, entering the target cross-end interaction sequence by operation recording, entering the target cross-end interaction sequence by voice entry, and entering the target cross-end interaction sequence by natural language entry.

[0302] In one implementation, the processing unit 1702 is configured to perform the following steps when recording the target cross-end interaction sequence in an operation recording manner:

[0303] Recording interactive operations performed on a target application; the interactive operations include at least one of a key input operation and a trigger operation on a display screen of the target application;

[0304] Mapping the interactive operation to a corresponding cross-end interactive instruction; the mapped cross-end interactive instruction includes at least one of a key instruction corresponding to the key input operation and a trigger instruction corresponding to the trigger operation;

[0305] Generate a target cross-end interaction sequence according to the cross-end interaction instructions mapped by the interaction operation.

[0306] In one implementation, the processing unit 1702 is used to perform the following steps when inputting the target cross-end interaction sequence in a natural language input manner:

[0307] Perform sentence classification and recognition on the input natural language to obtain one or more interaction description sentences in the natural language, wherein any interaction description sentence in the natural language is represented as a target sentence;

[0308] Perform semantic recognition processing on the target sentence to obtain multiple operation description information;

[0309] Map each operation description information to the corresponding cross-end interaction instruction;

[0310] According to the cross-end interaction instructions mapped to each operation description information, a target cross-end interaction sequence is generated.

[0311] In one implementation, the processing unit 1702 is further configured to perform the following steps:

[0312] Generate a sequence entry corresponding to the target cross-end interaction sequence, where the sequence entry is used to indicate the execution of the target cross-end interaction sequence;

[0313] Stores sequence entries into a configuration file.

[0314] In one implementation, the target cross-end interaction sequence includes a plurality of cross-end interaction instructions arranged in sequence; the processing unit 1702 is configured to generate a sequence entry corresponding to the target cross-end interaction sequence, specifically to perform the following steps:

[0315] Obtaining an execution time interval between any two adjacent cross-end interaction instructions in the target cross-end interaction sequence, and determining an execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence according to the execution time interval;

[0316] Obtain the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence;

[0317] Obtaining the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence;

[0318] A sequence entry corresponding to the target cross-end interaction sequence is generated according to the execution time, execution coordinates, and application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence.

[0319] In one implementation, any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; the processing unit 1702 is used to obtain the execution coordinates corresponding to the target interaction instruction, specifically to perform the following steps:

[0320] Obtain the reference coordinates corresponding to the target interaction instruction;

[0321] If the reference coordinates corresponding to the target interaction instruction match the reference display window information of the target application, the reference coordinates corresponding to the target interaction instruction are determined as the execution coordinates corresponding to the target interaction instruction;

[0322] If the reference coordinates corresponding to the target interaction instruction are not compatible with the reference display window information of the target application, a coordinate transformation is performed on the reference coordinates corresponding to the target interaction instruction to obtain second converted coordinates that are compatible with the reference display window information of the target application, and the second converted coordinates are determined as the execution coordinates corresponding to the target interaction instruction.

[0323] According to one embodiment of the present application, Fig.17 The various units in the cross-end interaction device shown can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the cross-end interaction device may also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0324] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 4 or Figure 7 A computer program for each step involved in part or all of the method shown in the figure is used to construct a computer program for each step involved in part or all of the method shown in the figure. Fig.17 The cross-end interaction device shown in and the cross-end interaction method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above-mentioned computing device through the computer-readable storage medium and run therein.

[0325] In the embodiment of the present application, the application is run across terminals by the first terminal, the first terminal is a terminal of the first terminal type, the application is developed based on the second terminal type, and the first terminal type and the second terminal type are two different terminal types; the mapping relationship between the shortcut command and the cross-terminal interaction sequence is recorded in the cross-terminal operation configuration file; based on the mapping relationship, in the process of the application being run across terminals by the first terminal, the target cross-terminal interaction sequence mapped to the target shortcut command input into the first terminal can be determined, and multiple application operations indicated by the target cross-terminal interaction sequence can be performed in the application run across terminals by the first terminal. It can be seen that in the embodiment of the present application, in the process of the application being run across terminals by the first terminal, based on the mapping relationship between the shortcut command and the cross-terminal interaction sequence, the input operation of a shortcut command in the first terminal can be quickly mapped to a series of application operations in the application run across terminals by the first terminal, which can improve the interaction efficiency between the application and the cross-terminal running in the first terminal, that is, the application cross-terminal interaction can be efficiently performed; and the input operation of the shortcut command is convenient, which can improve the convenience of the cross-terminal interaction with the application in the first terminal, that is, the application cross-terminal interaction can be conveniently performed.

[0326] Based on the above method and device embodiments, the present application embodiment provides a computer device. Fig.18 , Fig.18 It is a structural diagram of a computer device provided in an embodiment of the present application. Fig.18 The computer device shown includes at least a processor 1801, an input interface 1802, an output interface 1803, and a computer-readable storage medium 1804. The processor 1801, the input interface 1802, the output interface 1803, and the computer-readable storage medium 1804 may be connected via a bus or other means.

[0327] The computer-readable storage medium 1804 may be stored in a memory of a computer device. The computer-readable storage medium 1804 is used to store a computer program. The computer program includes computer instructions. The processor 1801 is used to execute the computer program stored in the computer-readable storage medium 1804. The processor 1801 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing a computer program, and is specifically suitable for loading and executing a computer program to implement a corresponding method flow or a corresponding function.

[0328] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the computer device. In addition, a computer program suitable for being loaded and executed by a processor is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0329] The computer device may be a first terminal of the first terminal type, and the first terminal may be, for example, a PC terminal. In a specific implementation, the processor 1801 may load and execute a computer program stored in the computer-readable storage medium 1804 to implement the above-mentioned Figure 4 or Figure 7 The corresponding steps in the method shown.

[0330] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned cross-end interaction method.

[0331] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0332] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0333] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0334] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A cross-terminal interaction method, characterized in that: include: In the process of running a target application across terminals on a first terminal, a target shortcut command input into the first terminal is obtained; the terminal type of the first terminal is a first terminal type, the target application is developed based on a second terminal type, and the first terminal type and the second terminal type are two different terminal types; Obtaining a configuration file for the cross-end operation, where the configuration file is used to record a mapping relationship between shortcut commands and cross-end interaction sequences; Determine, based on the mapping relationship, a target cross-end interaction sequence mapped to the target shortcut instruction, wherein the target cross-end interaction sequence is used to indicate a plurality of application operations executed in the target application being run across ends; A plurality of application operations indicated by the target cross-end interaction sequence are executed in the target application.

2. The method according to claim 1, characterized in that The target cross-end interaction sequence includes a plurality of cross-end interaction instructions arranged in sequence; and executing a plurality of application operations indicated by the target cross-end interaction sequence in the target application includes: Acquire a sequence entry corresponding to the target cross-end interaction sequence in the configuration file; the sequence entry is used to instruct execution of the target cross-end interaction sequence, and the sequence entry includes application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence; The application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence is executed in sequence in the target application.

3. The method according to claim 2, characterized in that The sequence entry also includes execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence in the display screen of the target application; The executing in the target application in sequence the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence includes: According to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, the application operations corresponding to each cross-end interaction instruction in the target cross-end interaction sequence are sequentially executed in the display screen of the target application.

4. The method according to claim 3, characterized in that The sequence entry also includes the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence; The executing, in sequence in the display screen of the target application, the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence according to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence includes: According to the execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence and according to the execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence, the application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence is executed sequentially in the display screen of the target application.

5. The method according to claim 3 or 4, characterized in that The sequence entry also includes reference display window information of the target application; any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; The process of executing the application operation corresponding to the target interaction instruction in the display screen of the target application includes: If the current display window information of the target application matches the reference display window information, then executing the application operation corresponding to the target interaction instruction in the display screen of the target application according to the execution coordinates corresponding to the target interaction instruction; If the current display window information of the target application does not match the reference display window information, coordinate transformation is performed on the execution coordinates corresponding to the target interaction instruction, and the application operation corresponding to the target interaction instruction is executed in the display screen of the target application according to the first transformed coordinates obtained by the coordinate transformation; The reference display window information includes default display window information, or display window information when the target interaction instruction is entered.

6. The method according to claim 5, characterized in that The execution coordinates corresponding to the target interaction instruction are adapted to the reference display window information; and the coordinate conversion of the execution coordinates corresponding to the target interaction instruction includes: Determining a coordinate conversion ratio according to the reference display window information and the current display window information; According to the coordinate conversion ratio, the execution coordinates corresponding to the target interaction instruction are converted into coordinates adapted to the current display window information to obtain the first converted coordinates.

7. The method according to claim 3 or 4, characterized in that Any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; the simulator in the first terminal serves as a running container to perform cross-end running on the target application; The process of executing the application operation corresponding to the target interaction instruction in the display screen of the target application includes: Executing the application operation corresponding to the target interaction instruction on the screen canvas of the simulator; The execution of the application operation corresponding to the target interaction instruction is transparently transmitted to the display screen of the target application.

8. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtain the entered target cross-end interaction sequence; Allocating a target shortcut command to the target cross-end interaction sequence; Mapping and storing the target cross-end interaction sequence and the target shortcut command into the configuration file; Among them, the entering of the target cross-end interaction sequence includes any one of the following: entering the target cross-end interaction sequence by key editing, entering the target cross-end interaction sequence by operation recording, entering the target cross-end interaction sequence by voice entry, and entering the target cross-end interaction sequence by natural language entry.

9. The method according to claim 8, characterized in that The step of recording the target cross-end interaction sequence in an operation recording manner includes: Recording an interactive operation performed on the target application; the interactive operation includes at least one of a key input operation and a trigger operation on a display screen of the target application; Mapping the interaction operation to a corresponding cross-end interaction instruction; the mapped cross-end interaction instruction includes at least one of a key instruction corresponding to the key input operation and a trigger instruction corresponding to the trigger operation; The target cross-end interaction sequence is generated according to the cross-end interaction instruction mapped by the interaction operation.

10. The method according to claim 8, characterized in that The step of entering the target cross-end interaction sequence in a natural language input manner includes: Performing sentence classification and recognition on the input natural language to obtain one or more interaction description sentences in the natural language, wherein any one of the interaction description sentences in the natural language is represented as a target sentence; Performing semantic recognition processing on the target sentence to obtain multiple operation description information; Map each operation description information to the corresponding cross-end interaction instruction; The target cross-end interaction sequence is generated according to the cross-end interaction instruction mapped to each operation description information.

11. The method according to claim 8, characterized in that The method further comprises: Generate a sequence entry corresponding to the target cross-end interaction sequence, where the sequence entry is used to indicate execution of the target cross-end interaction sequence; The sequence entries are stored in the configuration file.

12. The method according to claim 11, characterized in that The target cross-end interaction sequence includes a plurality of cross-end interaction instructions arranged in sequence; and the generating of a sequence entry corresponding to the target cross-end interaction sequence includes: Obtaining an execution time interval between any two adjacent cross-end interaction instructions in the target cross-end interaction sequence, and determining an execution time corresponding to each cross-end interaction instruction in the target cross-end interaction sequence according to the execution time interval; Obtaining execution coordinates corresponding to each cross-end interaction instruction in the target cross-end interaction sequence; Obtaining an application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence; A sequence entry corresponding to the target cross-end interaction sequence is generated according to the execution time, execution coordinates, and application operation corresponding to each cross-end interaction instruction in the target cross-end interaction sequence.

13. The method according to claim 12, characterized in that Any cross-end interaction instruction in the target cross-end interaction sequence is represented as a target interaction instruction; The process of obtaining the execution coordinates corresponding to the target interaction instruction includes: Obtaining reference coordinates corresponding to the target interaction instruction; If the reference coordinates corresponding to the target interaction instruction are compatible with the reference display window information of the target application, determining the reference coordinates corresponding to the target interaction instruction as the execution coordinates corresponding to the target interaction instruction; If the reference coordinates corresponding to the target interaction instruction are not compatible with the reference display window information of the target application, a coordinate transformation is performed on the reference coordinates corresponding to the target interaction instruction to obtain second converted coordinates that are compatible with the reference display window information of the target application, and the second converted coordinates are determined as the execution coordinates corresponding to the target interaction instruction.

14. A cross-terminal interaction device, characterized in that: include: an acquisition unit, configured to acquire a target shortcut command input into the first terminal during a process in which the first terminal runs a target application across terminals; the terminal type of the first terminal is a first terminal type, the target application is developed based on a second terminal type, and the first terminal type and the second terminal type are two different terminal types; The acquisition unit is further used to acquire a configuration file for the cross-end operation, where the configuration file is used to record a mapping relationship between shortcut commands and cross-end interaction sequences; A processing unit, configured to determine, based on the mapping relationship, a target cross-end interaction sequence mapped to the target shortcut instruction, wherein the target cross-end interaction sequence is used to indicate a plurality of application operations executed in the target application being run across ends; The processing unit is further configured to execute, in the target application, a plurality of application operations indicated by the target cross-end interaction sequence.

15. A computer device, characterized in that: The computer device comprises: a processor suitable for implementing a computer program; A computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by the processor and executing the cross-end interaction method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the cross-end interaction method according to any one of claims 1-13.

17. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the cross-end interaction method as described in any one of claims 1-13.

Images