Cross-terminal call control method and device, electronic equipment and readable storage medium

Abstract

The invention relates to a cross-terminal call control method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving a call control instruction from a second terminal; determining a display screen which can be used for receiving the call control instruction; obtaining an interaction mode corresponding to the call control instruction; and executing a call task corresponding to the call control instruction based on a simulation operation corresponding to the interaction mode. The method can be applied to various network telephone applications, so that the control universality of the cross-terminal call is higher.

Description

Technical Field

[0001] The present application relates to the control of cross-end calls, and in particular to a method, device, electronic device and computer-readable storage medium for controlling cross-end calls. Background Art

[0002] During a cross-terminal call, the call information of the first terminal (such as a mobile phone) is synchronized to the cloud, and then the second terminal (such as a car terminal) monitors the call information sent from the cloud, and draws the corresponding call interface on the interconnected second terminal according to its status and content.

[0003] Then the user's operations on the call interface on the second terminal (such as answering, hanging up, etc.) are synchronized to the cloud or directly transmitted to the first terminal. The first terminal and the second terminal need to install a specific Internet phone application with a cooperative relationship to realize cross-terminal call control, and its versatility is low. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device, and computer-readable storage medium for controlling cross-end calls, which can be applicable to a variety of Internet phone applications, thereby making the control of cross-end calls more versatile.

[0005] In a first aspect, the present application provides a method for controlling a cross-end call. The method comprises:

[0006] Receiving a call control instruction from the second terminal;

[0007] Determining a display screen that can be used to receive the call control instruction;

[0008] Obtaining the interaction mode corresponding to the call control instruction;

[0009] Based on the simulated operation corresponding to the interaction mode, the call task corresponding to the call control instruction is executed.

[0010] The present application also provides a device for controlling cross-end calls. The device comprises:

[0011] A parsing module, used for determining a call task based on the call control instruction sent by the second terminal;

[0012] A determination module, used to determine a display screen of a call interface; the call interface corresponds to the call task;

[0013] An interaction module, used to obtain an interaction mode corresponding to the call task;

[0014] The injection module is used to inject the interaction event corresponding to the interaction mode into the display screen.

[0015] The present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] Receiving a call control instruction from the second terminal;

[0017] Determining a display screen that can be used to receive the call control instruction;

[0018] Obtaining the interaction mode corresponding to the call control instruction;

[0019] Based on the simulated operation corresponding to the interaction mode, the call task corresponding to the call control instruction is executed.

[0020] The present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0021] Receiving a call control instruction from the second terminal;

[0022] Determining a display screen that can be used to receive the call control instruction;

[0023] Obtaining the interaction mode corresponding to the call control instruction;

[0024] Based on the simulated operation corresponding to the interaction mode, the call task corresponding to the call control instruction is executed.

[0025] The above-mentioned cross-terminal call control method, device, computer equipment and computer-readable storage medium receive call control instructions sent from the second terminal, determine the display screen that can be used to undertake the call control instructions, so that in the cross-terminal case, the display screen of the first terminal is controlled by the second terminal to determine the screen on which the call task acts; and obtain the interaction mode corresponding to the call control instruction, and execute the call task corresponding to the call control instruction based on the simulation operation corresponding to the interaction mode, so that the call control instruction can act on the display screen, which is suitable for a variety of Internet phone applications, thereby making the control of cross-terminal calls more versatile.

[0026] In a second aspect, the present application provides another method for controlling cross-end calls. The method comprises:

[0027] Receiving a call control instruction from the second terminal;

[0028] Determining a display screen that is currently available to receive the call control instruction;

[0029] Determining a control position of a control on the display screen that corresponds to the call control instruction;

[0030] A simulated operation is performed on the control position to execute the call task corresponding to the call control instruction.

[0031] The present application also provides a device for controlling cross-end calls. The device comprises:

[0032] A parsing module, used for receiving a call control instruction sent by the second terminal;

[0033] A determination module, used to determine a display screen that is currently available to receive the call control instruction;

[0034] A positioning module, used to determine the control position of the control corresponding to the call control instruction on the display screen;

[0035] The execution module is used to perform a simulation operation on the control position to execute the call task corresponding to the call control instruction.

[0036] The present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0037] Receiving a call control instruction from the second terminal;

[0038] Determining a display screen that is currently available to receive the call control instruction;

[0039] Determining a control position of a control on the display screen that corresponds to the call control instruction;

[0040] A simulated operation is performed on the control position to execute the call task corresponding to the call control instruction.

[0041] The present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0042] Receiving a call control instruction from the second terminal;

[0043] Determining a display screen that is currently available to receive the call control instruction;

[0044] Determining a control position of a control on the display screen that corresponds to the call control instruction;

[0045] A simulated operation is performed on the control position to execute the call task corresponding to the call control instruction.

[0046] The above-mentioned cross-terminal call control method, device, computer equipment and computer-readable storage medium receive call control instructions sent from the second terminal, determine the display screen that can be used to undertake the call control instructions, so that in the cross-terminal case, the display screen of the first terminal is controlled by the second terminal to determine the screen on which the call task acts; and determine the control position of the control corresponding to the call control instruction on the display screen; simulate the operation on the control position to execute the call task corresponding to the call control instruction, so that the call control instruction can act on the display screen by simulating the character operation, and is suitable for a variety of Internet phone applications, thereby making the control of cross-terminal calls more versatile. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a diagram of an application environment of a method for controlling a cross-end call in an embodiment;

[0049] Figure 2 is a flow chart of a method for controlling a cross-end call in one embodiment;

[0050] Figure 3 This is a diagram of an application environment of a method for controlling a cross-end call in an embodiment;

[0051] Figure 4 A flowchart for determining a display screen in one embodiment;

[0052] Figure 5 A flowchart of performing button position parsing on a display screen in one embodiment;

[0053] Figure 6 A flowchart of injecting interactive events into different display screens in one embodiment;

[0054] Figure 7 is a flow chart of a method for controlling a cross-end call in one embodiment;

[0055] Figure 8 is a structural block diagram of a control device for cross-end calls in one embodiment;

[0056] Fig. 9 is a structural block diagram of a control device for cross-end calls in one embodiment;

[0057] Fig.10FIG. 4 is a structural block diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The cross-end call control method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The first terminal 102 communicates with the second terminal 104 through the network. The first terminal 102 and the second terminal 104 may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.; head-mounted devices include augmented reality glasses. The first terminal 102 and the second terminal 104 may be electronic devices of the same type or different types, for example, the first terminal 102 is a mobile phone terminal, and the second terminal 104 is a car terminal.

[0060] The cross-end call control method is applied in scenarios including a product that uses the phone call function on a device connected to a smartphone; the cross-end call can be an Internet phone based on an Internet phone application, and the Internet phone is a voice transmission technology based on IP (Voice over Internet Protocol, VOIP).

[0061] In one embodiment, Figure 2 As shown, a cross-end call control method is provided, which is applied to Figure 1 The method is applied to the first terminal 102, and includes the following steps:

[0062] Step 202: Receive a call control instruction from the second terminal.

[0063] The call control instruction is a command sent by the second terminal to the first terminal to control the first terminal, and is used to instruct the first terminal to assist the second terminal in executing certain steps related to the call. By sending the call control instruction from the second terminal to the first terminal, the control of the first terminal can be initiated by the second terminal without modifying the phone application, enriching the application ecology in the terminal interconnection scenario.

[0064] Optionally, the call control instruction may act on the Internet phone application installed on the first terminal. The types of Internet phone applications include, but are not limited to, applications specifically designed for voice calls, applications for team collaboration such as video conferencing, applications used as media, and applications for instant messaging. It is understandable that the types of these Internet phone applications are divided according to their functions, and some applications may belong to multiple categories.

[0065] Optionally, after the call control instruction is sent to the first terminal, the first terminal first parses the call task from the call control instruction. A call task is an event related to a call, which is implemented based on a certain application. A call task can be an answering event or a closing event during a call; and according to the application to which the call task belongs, a hands-free event, a contact selection event, a dialing event, etc. during a call can be implemented. Optionally, a call task can be implemented through a call event of a certain type of Internet phone application.

[0066] Optionally, the second terminal can be used to control the first terminal to perform a call task, and during the process of the first terminal performing the call task, the first terminal can feed back information about the call task execution process to the second terminal. Optionally, during the process of performing the call task, the first terminal performs logical processing such as judgment and selection on the relevant information of the call task to obtain a processing result, and then feeds back the logical processing result to the second terminal. The relevant information of the call task can be information related to the call interface. The call control instruction issued by the second terminal instructs the first terminal to perform certain steps, and the data processing process can be moved to the first terminal, simplifying the data processing requirements of the second terminal during the call.

[0067] In one embodiment, when a third terminal initiates a call request to a second terminal through an Internet phone application, the second terminal generates a call control instruction belonging to the call task according to the call task selection instruction; the second terminal then sends the call control instruction to the first terminal, and then the first terminal executes the call task; and the process of parsing and executing the call control instruction includes steps 204-208.

[0068] Step 204, determining a display screen that can be used to receive call control instructions.

[0069] The display screen refers to a screen that can be used to accept call control instructions. Optionally, the display screen is a display screen that can be used to accept call control instructions at present, and is set for the current time point or time period. Optionally, if the display screen accepts the call control instruction, the call interface corresponding to the call control instruction is displayed, and the call interface is the application interface involved in the call task execution process, and the call task can be executed based on the call interface. Optionally, the call interface is the interface of the application installed on the first terminal, and the call interface can display call information related to the first terminal and the second terminal. Optionally, the call interface may include contact information for the call with the second terminal, and may also include certain buttons for executing the call task. The call interface corresponds to the call task, which means that the call interface can convey certain information of the call task, or execute the call task based on the call interface. Optionally, the call interface corresponding to the call task can be determined based on the display screen of the call interface, so as to determine the call interface corresponding to the call task.

[0070] Optionally, the display screen is determined based on the display information of the screen, and the display information is used to characterize the information displayed on the screen of the first terminal. Optionally, the display screen is one of the main screen of the first terminal and the virtual screen of the first terminal; the main screen refers to the preset screen of the first terminal, and the virtual screen is created by a system program or an application program, and both the main screen and the virtual screen can display the call interface. The main screen is different from the virtual screen. The main screen is a visible screen on the physical structure, and the virtual screen is one or more screens in the application program, and the virtual screen can be set to be visible or invisible. It is understandable that the virtual screen can display certain information of the first terminal to the second terminal or other terminals by means of screen projection.

[0071] In one embodiment, determining a display screen that can be used to receive a call control instruction includes: collecting display information of a preset type of screen for a first terminal; and selecting a screen from various types of screens of the first terminal as a display screen that can be used to receive a call control instruction based on the display information of the preset type of screen. Optionally, the preset type of screen is the main screen of the first terminal; and the various types of screens include a virtual screen that can be projected to a second terminal for interface display.

[0072] Step 206, obtaining the interaction mode corresponding to the call control instruction.

[0073] The interaction mode is based on certain rules of the first terminal, so as to trigger the first terminal to perform the call task. Optionally, the interaction mode corresponding to the call task can be an interaction mode implemented based on the call interface, or an interaction mode implemented based on gesture recognition or voice recognition.

[0074] In an optional embodiment, obtaining the interaction mode corresponding to the call control instruction includes: determining the Internet phone application installed on the first terminal; determining the preset interaction mode of the Internet phone application; and determining, among each preset interaction mode, the interaction mode used to trigger the first terminal to perform the call task.

[0075] Thus, through steps 202-206, the interaction mode is parsed from the call control instruction, and the first terminal is controlled to execute the corresponding steps based on the simulation operation corresponding to the interaction mode. Optionally, after the simulation operation, the first terminal detects the interaction event and injects the interaction event into the display screen to execute the call task corresponding to the call control instruction.

[0076] Step 208: Execute the call task corresponding to the call control instruction based on the simulated operation corresponding to the interaction mode.

[0077] The simulated operation is generated by the first terminal in an interactive manner triggered by the call control instruction at the first terminal. Optionally, the display screen of the first terminal can execute a certain scheme based on the interactive event, and the interactive event is used to simulate the interactive information performed by the user at the first terminal, which is the result of the simulated operation executed at the first terminal.

[0078] In one embodiment, based on the simulated operation corresponding to the interaction method, a call task corresponding to the call control instruction is executed, including: generating an interaction event according to the simulated operation corresponding to the interaction method; inputting the interaction event into the display screen so that the display screen receives the interaction event, thereby executing the call task corresponding to the call control instruction through the display screen.

[0079] In an exemplary embodiment, the mobile phone is the first terminal; this technical solution uses electronic terminal devices such as tablets, computers, cars, smart TVs, watches, etc. that can be interconnected with the mobile phone as the second terminal. Based on the existing interconnection channel between the first terminal and the second terminal, the interconnected terminal device controls the VOIP call on the mobile phone by parsing and executing the call control instructions sent by the interconnected terminal device.

[0080] Taking the vehicle-machine interconnection call scenario as an example, the control application scenario diagram is as follows Figure 3 This technical solution focuses on parsing and executing call control instructions. Parsing means parsing the call task in the received call control instruction, and then parsing or intelligently identifying the button position of the call control instruction on the display screen; generating simulated operations according to the triggering method of the button position, and injecting interactive events into the display screen by means of simulated operations to realize control.

[0081] The above-mentioned cross-terminal call control method receives a call control instruction sent from the second terminal, determines a display screen that can be used to receive the call control instruction, so that in the cross-terminal case, the display screen of the first terminal is controlled by the second terminal to determine the screen on which the call task acts; and obtains the interaction mode corresponding to the call control instruction, executes the call task corresponding to the call control instruction based on the simulation operation corresponding to the interaction mode, so that the call control instruction can act on the display screen, is suitable for a variety of Internet phone applications, and thus makes the control of cross-terminal calls more universal.

[0082] In one embodiment, receiving the call control instruction from the second terminal includes: receiving the call control instruction sent by the second terminal through an Internet phone application. The method also includes: parsing the call task in the call control instruction.

[0083] The second terminal can directly send call control instructions to the first terminal through the Internet phone application installed on it, so that the system program and application of the second terminal do not need to be significantly modified, thereby ensuring high versatility; in this case, the call task is parsed based on the system program or application of the first terminal, so that the first terminal can dynamically determine the interaction method and the corresponding simulation operation, ensuring that this solution can be applied to the telephone application installed on the first terminal.

[0084] In one embodiment, determining a display screen that can be used to receive call control instructions includes: determining whether the current call interface is displayed on the main screen of the first terminal; if so, the main screen is the display screen that can currently be used to receive call control instructions; if not, the virtual screen of the first terminal is the display screen that can currently be used to receive call control instructions.

[0085] The current call interface is the call interface currently displayed by the first terminal. Optionally, when the current call interface is displayed on the main screen, a user is currently using the main screen, and the screen used to receive call control instructions is determined as the main screen, so that the user can continue to use the main screen; and when the current call interface is not displayed on the main screen, the first terminal is in an inconvenient state to use, and the virtual screen of the first terminal is determined as the display screen used to receive call control instructions, and the virtual screen can be in a visible or invisible state.

[0086] Take the application scenario where the first terminal is a mobile phone and the second terminal is a vehicle computer as an example. If the current call interface is displayed on the main screen of the mobile phone, it at least means that someone in the vehicle is ready to make a call. At this time, there is no need to accept the call control instructions through the virtual screen; if the current call interface is not displayed on the main screen of the mobile phone, it means that the physical screen of the main screen of the mobile phone is not in use. If the vehicle only carries one driver, the call control instructions are accepted through the virtual screen of the mobile phone.

[0087] If the current call interface is displayed on the main screen of the first terminal, the main screen is the display screen for receiving call control instructions, so as to avoid the current call interface jumping due to the virtual screen, so that the information transmitted by the current call interface can be continuously displayed; if the current call interface is not displayed on the main screen of the first terminal, it means that it is not appropriate to use the physical screen of the main screen of the first terminal, so the virtual screen of the first terminal is determined as the virtual screen of the first terminal, so as to avoid the information transmitted by the main screen affecting the user's use of other terminals. In this way, in the control scenario of cross-terminal calls, the user experience is greatly improved.

[0088] In one embodiment, the method further includes: when the virtual screen of the first terminal is a display screen, displaying a call interface corresponding to the call control instruction on the virtual screen of the first terminal.

[0089] In one embodiment, displaying the call interface corresponding to the call control instruction on the virtual screen of the first terminal includes: starting the call interface corresponding to the call control instruction on the virtual screen to display the call interface; wherein the call interface corresponding to the call control instruction can be started through an Internet phone application or directly based on the call control instruction. Optionally, when the call interface is started on the virtual screen of the first terminal, a prompt message indicating that the call interface is started is transmitted.

[0090] In this embodiment, the virtual screen is used as a display screen as a condition for the continued execution of this method. When the condition is met, the call interface corresponding to the call control instruction is displayed on the virtual screen of the first terminal, so that the call interface is displayed based on the virtual screen, so that the call interface follows the virtual screen to display the call status, call information and other contents.

[0091] In one embodiment, the method further includes: when the current call interface is not displayed on the main screen of the first terminal, determining whether a virtual screen of the first terminal has been created; if not, creating a virtual screen of the first terminal; if created, the virtual screen of the first terminal is the display screen of the call interface.

[0092] Optionally, determining whether the virtual screen of the first terminal has been created includes: determining whether the virtual screen of the first terminal has been created based on log information of the first terminal; or detecting the virtual screen of the first terminal to obtain a detection result; and determining whether the virtual screen of the first terminal has been created based on the detection result.

[0093] In one embodiment, creating a virtual screen of the first terminal includes: calling an API interface through a system program of the first terminal, and generating the virtual screen of the first terminal according to a calling result of the API interface.

[0094] In this embodiment, when a virtual screen does not exist, a virtual screen is created, and the virtual screen of the first terminal is used as the display screen of the call interface; the virtual screen of the first terminal can exist continuously, so that the first terminal does not occupy too many resources due to the creation of the virtual screen, thereby maintaining the fluency of the first terminal.

[0095] Optionally, the virtual screen is not visible on the first terminal. During the execution of steps 202-208, the virtual screen of the second terminal is not visible on the first terminal, which can be reflected in: during the call, the main screen of the first terminal is always in screen silence; screen silence means that the main screen does not display the call interface corresponding to the call control command, or even in a black screen state.

[0096] Therefore, the virtual screen is invisible, and the virtual screen of the first terminal does not transmit information, so as to avoid disturbing the user due to the information transmitted by the first terminal. More importantly, the invisible virtual screen provides a new way of realizing other intelligent scenarios in the future by intelligently identifying the button position and other materials in the call interface.

[0097] Optionally, the virtual screen is projected to the second terminal, and the second terminal displays a call interface corresponding to the call control instruction through the virtual screen. The call interface corresponding to the call control instruction includes controls corresponding to the call control instruction.

[0098] Thus, the call interface corresponding to the call control instruction is projected from the first terminal to the second terminal through the virtual screen, so that the information of the call control instruction during the execution process is fed back to the second terminal, thereby displaying the call interface corresponding to the call control instruction based on the virtual screen received by the second terminal. Even if the second terminal only has the function of generating call control instructions and the function of displaying the virtual screen, calls between the second terminal and other terminals can be realized based on the first terminal.

[0099] Based on this, according to whether the current call interface is displayed on the main screen of the first terminal, the display screen for receiving the call control command is determined, and the call interface corresponding to the call control command is displayed on the display screen. On this basis, the corresponding implementation methods and corresponding effects of the display screen in the determination process are respectively explained from different angles.

[0100] In one embodiment, obtaining an interaction mode corresponding to a call control instruction includes: obtaining a control position indicated by the call control instruction on a display screen; and determining an interaction mode indicated by the call control instruction based on a triggering mode of the control position.

[0101] The control position is a certain position of the control in the display screen, and the position is used to indicate that a certain control of the Internet phone application triggers a certain operation. Each control position is respectively related to at least one call control instruction, and various call control instructions can be triggered by the control position. Optionally, the control position includes the position of a button, or the position of other elements that can trigger the instruction, and the position can be a coordinate position; for example, the button position is the coordinate position of the button in the call interface.

[0102] Optionally, the control position is selected and acted on by the call control instruction on the control position in the call interface of the Internet phone application. The control position is located in the call interface corresponding to the call control instruction, and the call interface is located in the display screen, so the control position indicated by the call control instruction can be obtained in the display screen.

[0103] Trigger mode, used to trigger the interactive event of the control position. Optionally, when the control position is related to a call task, the interactive mode corresponding to a call control instruction can be directly determined from the trigger mode of the control position; when the control position is related to at least two call tasks, the interactive mode corresponding to a call task can be determined from the different trigger modes of the control position.

[0104] Since the control position and the triggering method of the control position are obtained for the call control command, this solution can be applied to various call interfaces corresponding to different call control commands. Each call interface includes but is not limited to: the preset call interface and the changed call interface of the Internet phone application installed on the first terminal; wherein the changed call interface can be obtained because the phone application installed on the first terminal is changed, or because the phone application installed on the first terminal adjusts the call interface.

[0105] Take the call task of answering and hanging up events as an example: the Internet phone application A triggers the answering event by clicking or sliding the answering button; the Internet phone application A triggers the hang up event by clicking or sliding the hang up button.

[0106] The button position of the call task is obtained in the call interface, and the corresponding button position and the triggering method of the button position can be more specifically extracted. It is suitable for different call interfaces of different Internet phone applications and different call interfaces of the same Internet phone application, ensuring the versatility of this solution.

[0107] In one embodiment, obtaining the control position indicated by the call control instruction on the display screen includes: parsing application page element information on the display screen; and determining the parsed control position as the control position indicated by the call control instruction.

[0108] The application page element information includes layout information of application page elements for the call interface on the display screen, and the layout information can be used to determine at least part of the control positions on the display screen and the functions of each control position. Optionally, the application page element information can be information stored in an application page element list.

[0109] Optionally, the application page element information of the display screen is parsed, including: determining a call interface corresponding to the call control instruction in the display screen; and directly obtaining a control position corresponding to the call task from a list of application page elements of the call interface according to an identifier indicated by the call task.

[0110] Parsing the application page element information on the display screen helps to obtain the control position more quickly and accurately when the application page element information is successfully parsed. The user is unaware of the parsing of the application page element information on the display screen and does not disturb the user.

[0111] Optionally, in the process of parsing the application page element information on the display screen, the first terminal can parse the Internet phone application installed by itself based on its own system program. If the application page element information contains a control position, and the control position of the application page element information can be parsed by the first terminal, then the application page element information contains the control position; if the application page element information does not contain a control position, or the control position of the application page element information cannot be parsed by the first terminal, then the application page element information does not contain the control position.

[0112] In one embodiment, after parsing the application page element information of the display screen, it also includes: if the application page element information does not contain the control position, performing image classification and recognition on the control in the display screen to obtain a control recognition result; according to the control position indicated by the control recognition result, determining the control position indicated by the call control instruction.

[0113] The application page element information does not contain a control position, which means that the application page element information does not contain a control position corresponding to the call control instruction, or the application page element information does not exist. For example, when the application page element information is stored through an application page element list and the application page element list is empty, the application page element information does not exist.

[0114] Optionally, during the process of image classification and recognition of controls in the call interface, the image of the control is input into a composite control recognition model including a deep learning classification sub-model and a machine learning classification sub-model, so that the machine learning classification module can classify the controls to be recognized based on the feature vectors output by the deep learning classification module, thereby improving the accuracy of control recognition.

[0115] Among them, the deep learning sub-model is used for high-level abstract features of the control image, and the machine learning sub-model is used to explore the classification boundaries between categories. As a method, the deep learning classification sub-model can be any image classification network, including but not limited to YOLO, MobileNet and other models; the model suitable for deployment on the mobile phone can be selected according to the operating efficiency. Through the deep learning classification sub-model in the control recognition model, the high-dimensional feature vector of the control image is extracted, and the high-dimensional feature vector contains high-dimensional abstract nonlinear classification features. The machine learning classification sub-model is a machine learning network with strong nonlinear fitting ability. As a method, the machine learning classification sub-model can be implemented based on integrated learning algorithms such as LightGBM and XGBoost of the gradient boosting tree, thereby, through the deep learning classification sub-model and the machine learning classification sub-model. Therefore, through the accurately identified control image, the control position can be obtained in various general interfaces.

[0116] In an optional embodiment, the control position indicated by the call control instruction is determined according to the control position indicated by the control recognition result, including: searching for a control position with a matching identifier with the call control instruction in the control position indicated by the control recognition result, and using the control position with a matching identifier with the call control instruction as the control position indicated by the call control instruction. For example: if the control recognition result indicates an answer button position and a hang up button position, and the call control instruction is an answer instruction, then the control position with a matching identifier with the call control instruction is the answer button position. The matching identifier includes, but is not limited to, matching in terms of semantics or indexes.

[0117] First, the display screen is parsed for application page element information to determine that the display screen does not contain a control location. In this case, image classification and recognition of the controls in the display screen can enable this solution to still obtain the control location, thereby ensuring the versatility of the solution and unaware and non-disturbing to the user.

[0118] In one embodiment, based on the simulated operation corresponding to the interaction mode, a call task corresponding to the call control instruction is executed, including: constructing a simulated operation of the interaction mode according to the interaction mode; and performing a simulated operation on the display screen to execute the call task corresponding to the call control instruction.

[0119] The simulation operation is set for the Internet phone application of the first terminal, and is an interactive operation between a certain program of the first terminal and the Internet phone application installed on the first terminal. The interactive operation is the result of simulating user operation. The simulation operation is used to control the first terminal to perform the corresponding logical judgment process of the call task. For example: the simulation operation is used to simulate the user's click or slide operation on the first terminal to input the corresponding instruction into the first terminal.

[0120] The construction method of the simulated operation can be dynamically adjusted based on the interactive method of the Internet phone application installed on the first terminal; and the first terminal can be set with multiple interactive methods of Internet phone applications. For example: the Internet phone application A triggers the answering event by clicking or sliding the answering button position, and then the click event or sliding event of the call task is constructed according to the click or sliding method; the Internet phone application A triggers the hang-up event by clicking or sliding the hang-up button position, and then the click event or sliding event of the call task is constructed according to the click or sliding method.

[0121] Optionally, according to the interaction mode, a simulated operation of the interaction mode is constructed, including: if the interaction mode of the call control instruction at the first terminal is implemented based on the control position, then the triggering mode of the control position is simulated to generate a simulated operation; if the interaction mode of the call control instruction at the first terminal is implemented based on the gesture recognition result, then the gesture recognition result is simulated to generate a simulated operation.

[0122] By constructing simulated operations in an interactive manner, the control method of the first terminal on the display screen can be flexibly adjusted according to the input method of the first terminal. Therefore, when the Internet phone application used by the first terminal has different interactive logic, the simulated operation can still be transmitted to the display screen, thereby ensuring universality.

[0123] In one embodiment, according to the interaction mode, constructing a simulation operation of the interaction mode includes, if the interaction mode is a triggering mode of a control position of a call task, constructing a simulation operation of the interaction mode according to the triggering mode of the control position;

[0124] Correspondingly, a simulated operation is performed on the display screen to execute the call task corresponding to the call control instruction, including: when the control position is located on the main screen of the first terminal, a simulated operation is performed on the main screen to enable the main screen to realize the call task; when the control position is located on the virtual screen of the first terminal, a simulated operation is performed on the virtual screen to enable the virtual screen to realize the call task.

[0125] Through the triggering method of the control position, a simulated operation of the interactive method is constructed, and the display screen where the control position is located is determined. A screen is selected from the main screen and the virtual screen as the display screen, and the simulated operation is applied to the display screen to implement the call task. Other tasks implemented by the control position in the display screen can also be triggered in sequence based on the triggering method of the control position, which helps the versatility of this implementation method combined with other solutions.

[0126] In one embodiment, the first terminal is a mobile phone terminal, and the second terminal is a vehicle terminal. Figure 4 As shown:

[0127] Determine a display screen that can be used to receive call control instructions, including: determining whether the current call interface is being displayed on the main screen of the mobile phone; if the call interface is displayed on the main screen of the mobile phone, the main screen is the display screen that can currently be used to receive call control instructions; if the current call interface is not displayed on the main screen of the mobile phone, determine whether a virtual screen of the mobile phone has been created; if not, create a virtual screen of the mobile phone; if created, the virtual screen of the mobile phone is the display screen that can currently be used to receive call control instructions; wherein, the display screen is a screen that displays the call interface.

[0128] Correspondingly, obtaining the interaction method corresponding to the call control instruction includes: displaying a call interface corresponding to the call control instruction on the display screen; obtaining the button position indicated by the call control instruction in the call interface; and determining the interaction method corresponding to the call control instruction according to the triggering method of the button position.

[0129] In one embodiment, if Figure 5 As shown, in the display screen, the control position indicated by the call control instruction is obtained, including: through the accessibility service of the system program, for the call interface in the display screen, an application page element list is obtained; if the application page element list is not empty, the button position corresponding to the control instruction is directly obtained from the application page element list; otherwise, image classification and recognition are performed on the controls in the display screen to obtain control recognition results; according to the control position indicated by the control recognition result, the control position corresponding to the control instruction is determined.

[0130] In one embodiment, if Figure 6 As shown, the button position obtained by the above method is the button position, the interaction method corresponding to the call control instruction is the triggering method of the button position, and the input event of the call task is constructed according to the triggering method of the button position.

[0131] The input event is injected into the display screen so that the display screen can realize the call task, including: when the button position is located on the main screen of the mobile phone, the input event is injected into the main screen so that the main screen can realize the call task; when the button position is located on the virtual screen of the mobile phone, the input event is injected into the virtual screen so that the virtual screen can realize the call task.

[0132] Based on this, this embodiment provides a general solution for controlling Internet phone applications on terminal devices that are interconnected with mobile phones. Through this solution, it is possible to ensure that the interconnected terminal devices can control the Internet phone applications on the mobile phone side without intruding on the third-party Internet phone applications, enrich the application ecology in the mobile phone interconnection scenario, and be applicable to almost all Internet phone applications on the market, so that this solution does not need to be specifically used for a certain Internet phone application, and can be adapted to a variety of Internet phone applications, with strong versatility and low access cost. This solution creates a virtual screen that is invisible to the user, obtains the button position, identifies the controls of the call interface, and identifies the call interface on the mobile phone side. The user is unaware and does not disturb the user, and it provides a new implementation idea for realizing other intelligent scenarios in the future.

[0133] In addition, the user can draw a call interface on the second terminal itself, so that the call interface complies with the design specifications of the interconnected terminal device, and control the Internet phone application on the mobile phone through the call interface, greatly improving the user experience of the Internet phone application on the interconnected terminal device.

[0134] In one embodiment, Figure 7 As shown, a cross-end call control method is provided, which is applied to Figure 1 The method is applied to the first terminal 102, and includes the following steps:

[0135] Step 702: Receive a call control instruction from the second terminal.

[0136] Optionally, after the call control instruction is sent to the first terminal, the first terminal first parses the call task from the call control instruction.

[0137] Optionally, the second terminal can be used to control the first terminal to perform a call task, and during the process of the first terminal performing the call task, the first terminal can feed back information about the call task execution process to the second terminal. Optionally, during the process of performing the call task, the first terminal performs logical processing such as judgment and selection on the relevant information of the call task to obtain a processing result, and then feeds back the logical processing result to the second terminal. The relevant information of the call task can be information related to the call interface. The call control instruction issued by the second terminal instructs the first terminal to perform certain steps, and the data processing process can be moved to the first terminal, simplifying the data processing requirements of the second terminal during the call.

[0138] In one embodiment, when a third terminal initiates a call request to a second terminal through an Internet phone application, the second terminal generates a call control instruction belonging to the call task according to the call task selection instruction; the second terminal then sends the call control instruction to the first terminal, and then the first terminal executes the call task; and the process of parsing and executing the call control instruction includes steps 704-708.

[0139] Step 704, determining the display screen that is currently available to receive call control instructions.

[0140] Optionally, the display screen is a screen of a main screen of the first terminal and a virtual screen of the first terminal.

[0141] In one embodiment, determining the display screen that can currently be used to receive call control instructions includes: collecting display information of preset types of screens for the first terminal; and selecting a screen from various types of screens of the first terminal as the display screen that can currently be used to receive call control instructions based on the display information of the preset types of screens. Optionally, the preset type of screen is the main screen of the first terminal; the various types of screens include a virtual screen that can be projected to the second terminal for interface display.

[0142] Step 706: Determine the control position of the control corresponding to the call control instruction on the display screen.

[0143] A control is an interface element that can operate the input data of the first terminal, and the control is determined based on the call control instruction. Optionally, the call control instruction sent by the second terminal cannot directly act on the first terminal, so the corresponding control is determined based on the information transmitted by the call control instruction, and then the control position of the control is determined to simulate the simulated operation on the first terminal, so that the call control instruction sent by the second terminal can act on the first terminal.

[0144] In an optional embodiment, determining a control position of a control on a display screen that corresponds to a call control instruction includes: determining an Internet phone application installed on a first terminal according to the call control instruction; for the Internet phone application, determining a call interface that corresponds to the call control instruction; and in the call interface, determining a control position of a control that corresponds to the call control instruction.

[0145] Therefore, through steps 702-706, the control position of the control is parsed from the call control instruction, and a simulated operation is performed based on the control position, so as to control the first terminal to execute the corresponding steps.

[0146] Step 708, performing a simulated operation on the control position to execute the call task corresponding to the call control instruction.

[0147] In one embodiment, a simulated operation is performed on the control position to execute a call task corresponding to the call control instruction, including: generating an interaction event according to the simulated operation of the control position; inputting the interaction event into the display screen so that the display screen receives the interaction event, thereby executing the call task corresponding to the call control instruction through the display screen.

[0148] The above-mentioned cross-terminal call control method receives a call control instruction sent from the second terminal, determines a display screen that can be used to receive the call control instruction, so that in the cross-terminal case, the display screen of the first terminal is controlled by the second terminal to determine the screen on which the call task acts; and determines the control position of the control corresponding to the call control instruction on the display screen; simulates the operation on the control position to execute the call task corresponding to the call control instruction, so that the call control instruction can act on the display screen by simulating the human operation, and is suitable for a variety of Internet phone applications, thereby making the control of cross-terminal calls more versatile.

[0149] In one embodiment, determining the display screen that can currently be used to receive call control instructions includes: determining whether the current call interface is displayed on the main screen of the first terminal; if so, the main screen is the display screen for receiving call control instructions; if not, the virtual screen of the first terminal is the display screen for receiving call control instructions.

[0150] If the current call interface is displayed on the main screen of the first terminal, the main screen is the display screen for receiving call control instructions, so as to avoid the current call interface jumping due to the virtual screen, so that the information transmitted by the current call interface can be continuously displayed; if the current call interface is not displayed on the main screen of the first terminal, it means that it is not appropriate to use the physical screen of the main screen of the first terminal, so the virtual screen of the first terminal is determined as the virtual screen of the first terminal, so as to avoid the information transmitted by the main screen affecting the user's use of other terminals. In this way, in the control scenario of cross-terminal calls, the user experience is greatly improved.

[0151] In one embodiment, the method further includes: when the virtual screen of the first terminal is a display screen, displaying a call interface corresponding to the call control instruction on the virtual screen of the first terminal.

[0152] In this embodiment, the virtual screen is used as a display screen as a condition for the continued execution of this method. When the condition is met, the call interface corresponding to the call control instruction is displayed on the virtual screen of the first terminal, so that the call interface is displayed based on the virtual screen, so that the call interface follows the virtual screen to display the call status, call information and other contents.

[0153] Optionally, the method also includes: when the current call interface is not displayed on the main screen of the first terminal, determining whether the virtual screen of the first terminal has been created; if not, creating the virtual screen of the first terminal; if created, the virtual screen of the first terminal is the display screen.

[0154] Optionally, determining whether the virtual screen of the first terminal has been created includes: determining whether the virtual screen of the first terminal has been created based on log information of the first terminal; or detecting the virtual screen of the first terminal to obtain a detection result; and determining whether the virtual screen of the first terminal has been created based on the detection result.

[0155] In one embodiment, creating a virtual screen of the first terminal includes: calling an API interface through a system program of the first terminal, and generating the virtual screen of the first terminal according to a calling result of the API interface.

[0156] In this embodiment, when a virtual screen does not exist, a virtual screen is created, and the virtual screen of the first terminal is used as the display screen of the call interface; the virtual screen of the first terminal can exist continuously, so that the first terminal does not occupy too many resources due to the creation of the virtual screen, thereby maintaining the fluency of the first terminal.

[0157] Optionally, the virtual screen is not visible on the first terminal. During the execution of steps 702-208, the virtual screen of the second terminal is not visible on the first terminal, which can be reflected in: during the call, the main screen of the first terminal is always in screen silence; screen silence means that the main screen does not display the call interface corresponding to the call control command, or even in a black screen state.

[0158] Therefore, the virtual screen is invisible, and the virtual screen of the first terminal does not transmit information, so as to avoid disturbing the user due to the information transmitted by the first terminal. More importantly, the invisible virtual screen provides a new way of realizing other intelligent scenarios in the future by intelligently identifying the button position and other materials in the call interface.

[0159] Optionally, the virtual screen is projected to the second terminal, and the second terminal displays a call interface corresponding to the call control instruction through the virtual screen.

[0160] Thus, the call interface corresponding to the call control instruction is projected from the first terminal to the second terminal through the virtual screen, so that the information of the call control instruction during the execution process is fed back to the second terminal, thereby displaying the call interface corresponding to the call control instruction based on the virtual screen received by the second terminal. Even if the second terminal only has the function of generating call control instructions and the function of displaying the virtual screen, calls between the second terminal and other terminals can be realized based on the first terminal.

[0161] In one embodiment, determining a control position of a control on a display screen corresponding to a call control instruction includes: parsing application page element information on the display screen; and determining the control position of the parsed control as the control position of the control indicated by the call control instruction.

[0162] Optionally, the application page element information of the display screen is parsed, including: determining a call interface corresponding to the call control instruction in the display screen; and directly obtaining a control position corresponding to the call task from a list of application page elements of the call interface according to an identifier indicated by the call task.

[0163] Parsing the application page element information on the display screen helps to more quickly and accurately obtain the control position of the control when the application page element information is successfully parsed. The user is not aware of the parsing of the application page element information on the display screen, and the user is not disturbed.

[0164] Optionally, in the process of parsing the application page element information on the display screen, the first terminal can parse the Internet phone application installed by itself based on its own system program. If the application page element information contains the control position of the control, and the control position is parseable by the first terminal, the application page element information contains the control position; if the application page element information does not contain the control position of the control, or the control position of the control in the application page element information is not parseable by the first terminal, the application page element information does not contain the control position of the control.

[0165] In one embodiment, after parsing the application page element information of the display screen, it also includes: if the application page element information does not contain the control position, performing image classification and recognition on the control in the display screen to obtain a control recognition result; according to the control position indicated by the control recognition result, determining the control position indicated by the call control instruction.

[0166] Optionally, during the process of image classification and recognition of controls in the call interface, the image of the control is input into a composite control recognition model including a deep learning classification sub-model and a machine learning classification sub-model, so that the machine learning classification module can classify the controls to be recognized based on the feature vectors output by the deep learning classification module, thereby improving the accuracy of control recognition.

[0167] Among them, the deep learning sub-model is used for high-level abstract features of the control image, and the machine learning sub-model is used to explore the classification boundaries between categories. As a method, the deep learning classification sub-model can be any image classification network, including but not limited to YOLO, MobileNet and other models; the model suitable for deployment on the mobile phone can be selected according to the operating efficiency. Through the deep learning classification sub-model in the control recognition model, the high-dimensional feature vector of the control image is extracted, and the high-dimensional feature vector contains high-dimensional abstract nonlinear classification features. The machine learning classification sub-model is a machine learning network with strong nonlinear fitting ability. As a method, the machine learning classification sub-model can be implemented based on integrated learning algorithms such as LightGBM and XGBoost of the gradient boosting tree, thereby, through the deep learning classification sub-model and the machine learning classification sub-model. Therefore, through the accurately identified control image, the control position can be obtained in various general interfaces.

[0168] In an optional embodiment, the control position indicated by the call control instruction is determined according to the control position indicated by the control recognition result, including: searching for a control position with a matching identifier with the call control instruction in the control position indicated by the control recognition result, and using the control position with a matching identifier with the call control instruction as the control position indicated by the call control instruction. For example: if the control recognition result indicates an answer button position and a hang up button position, and the call control instruction is an answer instruction, then the control position with a matching identifier with the call control instruction is the answer button position. The matching identifier includes, but is not limited to, matching in terms of semantics or indexes.

[0169] First, the display screen is parsed for application page element information to determine that the display screen does not contain a control location. In this case, image classification and recognition of the controls in the display screen can enable this solution to still obtain the control location, thereby ensuring the versatility of the solution and unaware and non-disturbing to the user.

[0170] In one embodiment, a simulated operation is performed on the control position to execute the call task corresponding to the call control instruction, including: constructing a simulated operation of the trigger mode for the control position; and performing a simulated operation on the display screen to execute the call task corresponding to the call control instruction.

[0171] The construction method of the simulation operation can be dynamically adjusted based on the interaction method of the Internet phone application installed on the first terminal; and the first terminal can be provided with a plurality of interaction methods of the Internet phone application. Optionally, for the triggering method of the control position, the simulation operation of the triggering method is constructed, including: simulating the triggering method of the control position to generate the simulation operation.

[0172] Therefore, by constructing simulated operations through control positions, simulated operations acceptable to the first terminal can be generated for the first terminal, and call control instructions can be converted into simulated operations recognizable to the first terminal. In this way, when the Internet phone application used by the first terminal has different interaction logic, the simulated operations can still be transmitted to the display screen, thereby ensuring universality.

[0173] In one embodiment, a simulated operation is performed on the control position to execute a call task corresponding to the call control instruction, including: when the control position is located on the main screen of the first terminal, a simulated operation is performed on the main screen to enable the main screen to implement the call task; when the control position is located on the virtual screen of the first terminal, a simulated operation is performed on the virtual screen to enable the virtual screen to implement the call task.

[0174] The simulated operation is generated by the control position, and a screen is selected from the main screen and the virtual screen as the display screen, so that the simulated operation can act on the display screen, and the call task is implemented with the display screen. For other tasks implemented by the control position in the display screen, other simulated operations based on the control position can also be performed, which helps the versatility of this implementation method combined with other solutions.

[0175] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0176] Based on the same inventive concept, the embodiment of the present application also provides a cross-end call control device for implementing the cross-end call control method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more cross-end call control devices provided below can refer to the limitations of the cross-end call control method above, and will not be repeated here.

[0177] In one embodiment, Figure 8 As shown, a control device for cross-end calls is provided, comprising:

[0178] The parsing module 802 is used to receive a call control instruction from the second terminal;

[0179] A determination module 804, configured to determine a display screen that can be used to receive the call control instruction;

[0180] An interaction module 806 is used to obtain an interaction mode corresponding to the call control instruction;

[0181] The execution module 808 is used to execute the call task corresponding to the call control instruction based on the simulation operation corresponding to the interaction mode.

[0182] The parsing module 802 is used to receive a call control instruction from the second terminal;

[0183] A determination module 804, configured to determine a display screen that can be used to receive the call control instruction;

[0184] An interaction module 806 is used to obtain an interaction mode corresponding to the call control instruction;

[0185] The execution module 808 is used to execute the call task corresponding to the call control instruction based on the simulation operation corresponding to the interaction mode.

[0186] In one embodiment, the determining module 804 is used to:

[0187] Determining whether the current call interface is displayed on the main screen of the first terminal;

[0188] If yes, the main screen is the display screen currently available for receiving the call control command;

[0189] If not, the virtual screen of the first terminal is the display screen that can currently be used to receive the call control instruction.

[0190] In one embodiment, the determining module 804 is used to:

[0191] In a case where the virtual screen of the first terminal is the display screen, a call interface corresponding to the call control instruction is displayed on the virtual screen of the first terminal.

[0192] In one embodiment, the determining module 804 is used to:

[0193] In a case where the current call interface is not displayed on the main screen of the first terminal, determining whether a virtual screen of the first terminal has been created;

[0194] If not created, creating a virtual screen of the first terminal;

[0195] If it has been created, the virtual screen of the first terminal is the display screen.

[0196] In one of the embodiments, the virtual screen is not visible on the first terminal.

[0197] In one of the embodiments, the virtual screen is projected to the second terminal, and the second terminal displays a call interface corresponding to the call control instruction through the virtual screen, and the call interface corresponding to the call control instruction includes a control corresponding to the call control instruction.

[0198] In one embodiment, the interaction module 806 is used to:

[0199] In the display screen, obtaining a control position indicated by the call control instruction;

[0200] The interaction mode indicated by the call control instruction is determined according to the triggering mode of the control position.

[0201] In one embodiment, the interaction module 806 is used to:

[0202] Parsing the application page element information of the display screen;

[0203] The parsed control position is determined as the control position indicated by the call control instruction.

[0204] In one embodiment, the interaction module 806 is used to:

[0205] If the application page element information does not contain the control position, performing image classification and recognition on the control in the display screen to obtain a control recognition result;

[0206] The control position indicated by the call control instruction is determined according to the control position indicated by the control identification result.

[0207] In one embodiment, the execution module 808 is configured to:

[0208] According to the interaction mode, construct a simulation operation of the interaction mode;

[0209] The simulation operation is performed on the display screen to execute the call task corresponding to the call control instruction.

[0210] In one embodiment, the interaction module 806 is used to:

[0211] If the interaction mode is a triggering mode of a control position of a call task, constructing a simulation operation of the interaction mode according to the triggering mode of the control position;

[0212] Correspondingly, the execution module 808 is used to:

[0213] When the control is located on the main screen of the first terminal, performing the simulation operation on the main screen so that the main screen realizes the call task;

[0214] When the control is located on the virtual screen of the first terminal, the simulation operation is performed on the virtual screen so that the virtual screen realizes the call task.

[0215] In one embodiment, Fig. 9 As shown, a control device for cross-end calls is provided, comprising:

[0216] The parsing module 802 is used to receive the call control instruction sent by the second terminal;

[0217] A determination module 904, configured to determine a display screen that is currently available for receiving the call control instruction;

[0218] A positioning module 906, configured to determine a position of a control on the display screen corresponding to the call control instruction;

[0219] The execution module 808 is used to perform a simulation operation on the control position to execute the call task corresponding to the call control instruction.

[0220] In one embodiment, the determining module 904 is used to:

[0221] Determining whether the current call interface is displayed on the main screen of the first terminal;

[0222] If yes, the main screen is a display screen for receiving the call control instructions;

[0223] If not, the virtual screen of the first terminal is the display screen for receiving the call control instructions.

[0224] In one embodiment, the determining module 904 is used to:

[0225] In a case where the virtual screen of the first terminal is the display screen, a call interface corresponding to the call control instruction is displayed on the virtual screen of the first terminal.

[0226] In one embodiment, the determining module 904 is used to:

[0227] In a case where the current call interface is not displayed on the main screen of the first terminal, determining whether a virtual screen of the first terminal has been created;

[0228] If not created, creating a virtual screen of the first terminal;

[0229] If it has been created, the virtual screen of the first terminal is the display screen.

[0230] In one embodiment, the virtual screen is not visible on the first terminal.

[0231] In one embodiment, the virtual screen is projected to the second terminal, and the second terminal displays a call interface corresponding to the call control instruction through the virtual screen.

[0232] In one embodiment, the positioning module 906 is used to:

[0233] Parsing the application page element information of the display screen;

[0234] The control position of the parsed control is determined as the control position of the control indicated by the call control instruction.

[0235] In one embodiment, the positioning module 906 is used to:

[0236] If the application page element information does not contain the control position, performing image classification and recognition on the control in the display screen to obtain a control recognition result;

[0237] The control position indicated by the call control instruction is determined according to the control position indicated by the control identification result.

[0238] In one embodiment, the execution module 808 is used to:

[0239] According to the triggering mode of the control position, construct a simulation operation of the triggering mode;

[0240] The simulation operation is performed on the display screen to execute the call task corresponding to the call control instruction.

[0241] In one embodiment, the execution module 808 is used to:

[0242] When the control is located on the main screen of the first terminal, performing the simulation operation on the main screen so that the main screen realizes the call task;

[0243] When the control is located on the virtual screen of the first terminal, the simulation operation is performed on the virtual screen so that the virtual screen realizes the call task.

[0244] Each module in the above-mentioned cross-end call control device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0245] In one embodiment, a computer device is provided. The computer device may be a first terminal or a second terminal. The internal structure diagram of the computer device may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a control method for cross-end calls is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0246] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0247] The embodiment of the present application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the cross-end call control method.

[0248] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0249] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0250] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0251] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for controlling a cross-end call, characterized in that: Applied to the first terminal, comprising: Receiving a call control instruction from the second terminal; Determining a display screen that can be used to receive the call control instruction; Obtaining the interaction mode corresponding to the call control instruction; Based on the simulated operation corresponding to the interaction mode, the call task corresponding to the call control instruction is executed.

2. The method according to claim 1, characterized in that The determining of a display screen that can be used to receive the call control instruction includes: Determining whether the current call interface is displayed on the main screen of the first terminal; If yes, the main screen is the display screen currently available for receiving the call control command; If not, the virtual screen of the first terminal is the display screen that can currently be used to receive the call control instruction.

3. The method according to claim 2, characterized in that The method further comprises: In a case where the virtual screen of the first terminal is the display screen, a call interface corresponding to the call control instruction is displayed on the virtual screen of the first terminal.

4. The method according to claim 2, characterized in that: The method further comprises: In a case where the current call interface is not displayed on the main screen of the first terminal, determining whether a virtual screen of the first terminal has been created; If not created, creating a virtual screen of the first terminal; If it has been created, the virtual screen of the first terminal is the display screen.

5. The method according to claim 2, characterized in that: The virtual screen is not visible on the first terminal.

6. The method according to claim 2, characterized in that The virtual screen is projected to the second terminal, and the second terminal displays a call interface corresponding to the call control instruction through the virtual screen. The call interface corresponding to the call control instruction includes a control corresponding to the call control instruction.

7. The method according to claim 1, characterized in that The obtaining of the interaction mode corresponding to the call control instruction includes: In the display screen, obtaining a control position indicated by the call control instruction; The interaction mode indicated by the call control instruction is determined according to the triggering mode of the control position.

8. The method according to claim 7, characterized in that The obtaining, on the display screen, a control position indicated by the call control instruction includes: Parsing the application page element information of the display screen; The parsed control position is determined as the control position indicated by the call control instruction.

9. The method according to claim 8, characterized in that After parsing the application page element information of the display screen, the method further includes: If the application page element information does not contain the control position, performing image classification and recognition on the control in the display screen to obtain a control recognition result; The control position indicated by the call control instruction is determined according to the control position indicated by the control identification result.

10. The method according to claim 1, characterized in that The performing of the call task corresponding to the call control instruction based on the simulated operation corresponding to the interaction mode includes: According to the interaction mode, construct a simulation operation of the interaction mode; The simulation operation is performed on the display screen to execute the call task corresponding to the call control instruction.

11. The method according to claim 10, characterized in that The step of constructing a simulation operation of the interaction mode according to the interaction mode includes: If the interaction mode is a triggering mode of a control position of a call task, constructing a simulation operation of the interaction mode according to the triggering mode of the control position; The performing the simulation operation on the display screen to execute the call task corresponding to the call control instruction includes: When the control is located on the main screen of the first terminal, performing the simulation operation on the main screen so that the main screen realizes the call task; When the control is located on the virtual screen of the first terminal, the simulation operation is performed on the virtual screen so that the virtual screen realizes the call task.

12. A method for controlling a cross-end call, characterized in that: Applied to the first terminal, comprising: Receiving a call control instruction from the second terminal; Determining a display screen that is currently available to receive the call control instruction; Determining a control position of a control on the display screen that corresponds to the call control instruction; A simulated operation is performed on the control position to execute the call task corresponding to the call control instruction.

13. The method according to claim 12, characterized in that The determining of a display screen that is currently available for receiving the call control instruction includes: Determining whether the current call interface is displayed on the main screen of the first terminal; If yes, the main screen is the display screen currently available for receiving the call control command; If not, the virtual screen of the first terminal is the display screen that can currently be used to receive the call control instruction.

14. The method according to claim 13, characterized in that The method further comprises: In a case where the virtual screen of the first terminal is the display screen, a call interface corresponding to the call control instruction is displayed on the virtual screen of the first terminal.

15. The method according to claim 13, characterized in that The method further comprises: In a case where the current call interface is not displayed on the main screen of the first terminal, determining whether a virtual screen of the first terminal has been created; If not created, creating a virtual screen of the first terminal; If it has been created, the virtual screen of the first terminal is the display screen.

16. The method according to claim 13, characterized in that The virtual screen is not visible on the first terminal.

17. The method according to claim 13, characterized in that The virtual screen is projected to the second terminal, and the second terminal displays a call interface corresponding to the call control instruction through the virtual screen.

18. The method according to claim 12, characterized in that The determining a control position of a control corresponding to the call control instruction on the display screen includes: Parsing the application page element information of the display screen; The control position of the parsed control is determined as the control position of the control indicated by the call control instruction.

19. The method according to claim 12, characterized in that After parsing the application page element information of the display screen, the method further includes: If the application page element information does not contain the control position, performing image classification and recognition on the control in the display screen to obtain a control recognition result; The control position indicated by the call control instruction is determined according to the control position indicated by the control identification result.

20. The method according to claim 12, characterized in that The simulating operation on the control position to execute the call task corresponding to the call control instruction includes: According to the triggering mode of the control position, construct a simulation operation of the triggering mode; The simulation operation is performed on the display screen to execute the call task corresponding to the call control instruction.

21. The method according to claim 12, characterized in that The simulating operation on the control position to execute the call task corresponding to the call control instruction includes: When the control is located on the main screen of the first terminal, performing the simulation operation on the main screen so that the main screen realizes the call task; When the control is located on the virtual screen of the first terminal, the simulation operation is performed on the virtual screen so that the virtual screen realizes the call task.

22. A control device for cross-end calls, characterized in that: include: A parsing module, used for receiving a call control instruction from the second terminal; A determination module, used to determine a display screen that can be used to receive the call control instruction; An interaction module, used to obtain an interaction mode corresponding to the call control instruction; An execution module is used to execute a call task corresponding to the call control instruction based on the simulation operation corresponding to the interaction mode.

23. A control device for cross-end calls, characterized in that: include: A parsing module, used for receiving a call control instruction sent by the second terminal; A determination module, used to determine a display screen that is currently available to receive the call control instruction; A positioning module, used to determine the control position of the control corresponding to the call control instruction on the display screen; The execution module is used to perform a simulation operation on the control position to execute the call task corresponding to the call control instruction.

24. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor executes the steps of the method for controlling a cross-end call according to any one of claims 1 to 21.

25. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 21 are implemented.

Images