Screen sharing processing method and device

Abstract

The invention provides a screen sharing processing method and device, and the method comprises the steps: entering a service application program according to an access instruction of the service application program after a first terminal builds a call with a second terminal through a call application program, obtaining a screen sharing instruction submitted in the service application program, and responding to the screen sharing instruction, screen recording is carried out through the service application program, the screen video stream for screen sharing is obtained, and screen sharing in the call process is achieved.

Description

Technical Field

[0001] This document relates to the field of communication technology, and in particular to a screen sharing processing method and device. Background Art

[0002] With the continuous development of Internet technology and communication technology, screen sharing has been widely used. Screen sharing refers to the remote terminal screen reproducing the interface display content of the local terminal screen within a tolerable time delay, providing remote users with a virtual interface environment as if they were sitting next to the local terminal.

[0003] At present, screen sharing achieved through conference software and social software mainly relies on the Internet for data transmission. The two parties performing screen sharing download the corresponding software in advance and create a screen sharing channel, such as a conference ID (Identifier). The sharing user shares the conference ID with the user to be shared, so that the user to be shared joins the conference, and then shares the screen through the Internet channel corresponding to the conference. The screen sharing process is complicated and cumbersome. How to improve the convenience of screen sharing is a focus of increasing concern for users. Summary of the invention

[0004] An objective of an embodiment of the present specification is to provide a screen sharing processing method and device to improve the convenience of implementing screen sharing.

[0005] To solve the above technical problems, an embodiment of this specification is implemented as follows: In a first aspect, an embodiment of the present specification provides a first screen sharing processing method, including: Establishing a call with the second terminal through the call application; Entering the service application according to the access instruction of the service application, and obtaining the screen sharing instruction submitted in the service application; In response to the screen sharing instruction, screen recording is performed through the service application to obtain a screen video stream for screen sharing.

[0006] In a second aspect, another embodiment of the present specification provides a second screen sharing processing method, including: Establishing a call with the first terminal through a call application; Acquire a screen video stream received by the call application and obtained by the first terminal through screen recording by the service application; The screen video stream is displayed on a call interface of the call application.

[0007] In a third aspect, another embodiment of the present specification provides a third screen sharing processing method, including: Acquire a screen sharing instruction of the first terminal forwarded by the application server; the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by recording the screen of the first terminal through the service application; The screen video stream is sent to a second terminal corresponding to the call.

[0008] In a fourth aspect, another embodiment of the present specification provides a fourth screen sharing processing method, including: Entering the service application according to an access instruction of the service application; the access instruction is obtained after the call is established based on the call application; Get screen sharing instructions; In response to the screen sharing instruction, screen recording is performed to obtain a screen video stream for screen sharing.

[0009] In a fifth aspect, another embodiment of the present specification provides a fifth screen sharing processing method, including: Obtaining a screen sharing instruction forwarded by the application server and submitted by the service application after establishing a call; the call is established based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by screen recording through the service application; The screen video stream is sent to the calling application.

[0010] In a sixth aspect, another embodiment of the present specification provides a first screen sharing processing device, including: A call establishment module, used to establish a call with the second terminal through a call application; An instruction acquisition module, used to enter the service application according to the access instruction of the service application, and acquire the screen sharing instruction submitted in the service application; The screen recording module is used to respond to the screen sharing instruction, perform screen recording through the service application, and obtain a screen video stream for screen sharing.

[0011] In a seventh aspect, another embodiment of the present specification provides a second screen sharing processing device, including: A call establishment module, used to establish a call with the first terminal through a call application; A video stream acquisition module, used to acquire a screen video stream received by the call application and obtained by the first terminal through screen recording by the service application; The video stream display module is used to display the screen video stream on the call interface of the call application.

[0012] In an eighth aspect, another embodiment of the present specification provides a third screen sharing processing device, including: An instruction acquisition module, used for acquiring a screen sharing instruction of the first terminal forwarded by the application server; the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; A video stream acquisition module, configured to acquire, according to the screen sharing instruction, a screen video stream obtained by recording the screen of the first terminal through the service application; The video stream sending module is used to send the screen video stream to the second terminal corresponding to the call.

[0013] In a ninth aspect, another embodiment of the present specification provides a fourth screen sharing processing device, including: An application entry module, used to enter the service application according to an access instruction of the service application; the access instruction is obtained after the call is established based on the call application; An instruction acquisition module is used to acquire screen sharing instructions; The screen recording module is used to perform screen recording in response to the screen sharing instruction and obtain a screen video stream for screen sharing.

[0014] In a tenth aspect, another embodiment of the present specification provides a fifth screen sharing processing device, including: An instruction acquisition module, used to acquire a screen sharing instruction forwarded by an application server and submitted by a service application after a call is established; the call is established based on the call application; A video stream acquisition module, used to acquire a screen video stream obtained by screen recording through the service application according to the screen sharing instruction; The video stream sending module is used to send the screen video stream to the call application.

[0015] In the eleventh aspect, another embodiment of the present specification provides a first screen sharing processing device, comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the screen sharing processing method as described in the first aspect above.

[0016] In the twelfth aspect, another embodiment of the present specification provides a second screen sharing processing device, comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the screen sharing processing method as described in the second aspect above.

[0017] In the thirteenth aspect, another embodiment of the present specification provides a third screen sharing processing device, comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the screen sharing processing method as described in the third aspect above.

[0018] In the fourteenth aspect, another embodiment of the present specification provides a fourth screen sharing processing device, comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the screen sharing processing method as described in the fourth aspect above.

[0019] In the fifteenth aspect, another embodiment of the present specification provides a fifth screen sharing processing device, comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the screen sharing processing method as described in the fifth aspect above.

[0020] In the sixteenth aspect, another embodiment of the present specification provides a first computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the screen sharing processing method described in the first aspect above are implemented.

[0021] In the seventeenth aspect, another embodiment of the present specification provides a second computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the screen sharing processing method as described in the second aspect above are implemented.

[0022] In the eighteenth aspect, another embodiment of the present specification provides a third computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the screen sharing processing method described in the third aspect above are implemented.

[0023] In the nineteenth aspect, another embodiment of the present specification provides a fourth computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the screen sharing processing method described in the fourth aspect above are implemented.

[0024] In the twentieth aspect, another embodiment of the present specification provides a fifth computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the screen sharing processing method as described in the fifth aspect above are implemented.

[0025] In the twenty-first aspect, another embodiment of the present specification provides a first computer program product, wherein the computer program product includes a screen sharing processing program, and the screen sharing processing program is executed by a processor to implement the steps of the screen sharing processing method as described in the first aspect above.

[0026] In the twenty-second aspect, another embodiment of the present specification provides a second computer program product, wherein the computer program product includes a screen sharing processing program, and the screen sharing processing program is executed by a processor to implement the steps of the screen sharing processing method as described in the second aspect above.

[0027] In the twenty-third aspect, another embodiment of the present specification provides a third computer program product, wherein the computer program product includes a screen sharing processing program, and the screen sharing processing program is executed by a processor to implement the steps of the screen sharing processing method described in the third aspect above.

[0028] In the twenty-fourth aspect, another embodiment of the present specification provides a fourth computer program product, wherein the computer program product includes a screen sharing processing program, and the screen sharing processing program is executed by a processor to implement the steps of the screen sharing processing method as described in the fourth aspect above.

[0029] In the twenty-fifth aspect, another embodiment of the present specification provides a fifth computer program product, wherein the computer program product includes a screen sharing processing program, and the screen sharing processing program is executed by a processor to implement the steps of the screen sharing processing method as described in the fifth aspect above.

[0030] The screen sharing processing method provided in this embodiment is that after the first terminal establishes a call with the second terminal based on the call application, the first terminal enters the service application according to the access instruction of the service application, and obtains the screen sharing instruction submitted in the service application, thereby realizing screen sharing during the call. Specifically, the first terminal can submit the screen sharing instruction based on the call by entering the service application. Furthermore, the first terminal responds to the screen sharing instruction, performs screen recording through the service application, and obtains the screen video stream for screen sharing. In this way, not only the screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in one or more embodiments of the present specification, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A first screen sharing process flow chart provided for one embodiment of this specification; Figure 2 A schematic diagram of call record data transmission provided by an embodiment of this specification; Figure 3 A schematic diagram of a call interface provided by an embodiment of this specification; Figure 4 A schematic diagram of a terminal interface provided by an embodiment of this specification; Figure 5 A schematic diagram of a service interface provided for one embodiment of this specification; Figure 6 A schematic diagram of a screen interface provided for one embodiment of this specification; Figure 7 A timing diagram of a unilateral video negotiation process provided by one embodiment of this specification; Figure 8 A schematic diagram of a screen sharing processing method for a call-based screen sharing scenario provided by an embodiment of the present specification; Fig. 9 A processing flow chart of a screen sharing processing method applied to a call-based screen sharing scenario provided by one embodiment of this specification; Fig.10 A second screen sharing process flow chart provided for one embodiment of this specification; Fig.11 A third screen sharing process flow chart provided for one embodiment of this specification; Fig.12 A processing flow chart of another screen sharing processing method applied to a call-based screen sharing scenario provided by an embodiment of the present specification; Fig.13 A fourth screen sharing process flow chart provided in one embodiment of this specification; Fig.14 A fifth screen sharing process flow chart provided for one embodiment of this specification; Fig.15 A schematic diagram of a first screen sharing processing device provided in one embodiment of this specification; Fig.16A schematic diagram of a second screen sharing processing device provided in one embodiment of this specification; Fig.17 A schematic diagram of a third screen sharing processing device provided in one embodiment of this specification; Fig.18 A schematic diagram of a fourth screen sharing processing device provided by an embodiment of this specification; Fig.19 A schematic diagram of a fifth screen sharing processing device provided by an embodiment of this specification; Fig. 20 A schematic diagram of the structure of a first screen sharing processing device provided in one embodiment of this specification; Fig.21 A schematic diagram of the structure of a second screen sharing processing device provided in one embodiment of this specification; Fig. 22 A schematic diagram of the structure of a third screen sharing processing device provided in one embodiment of this specification; Fig.23 A schematic diagram of the structure of a fourth screen sharing processing device provided in one embodiment of this specification; Fig.24 This is a schematic diagram of the structure of a fifth screen sharing processing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0034] The first screen sharing processing method embodiment provided in this specification is as follows: Reference Figure 1 , which shows a processing flow chart of a screen sharing processing method provided in this embodiment, which can be applied to the first terminal. The screen sharing processing method provided in this embodiment specifically includes the following steps S102 to S106.

[0035] Step S102: Establishing a call with the second terminal through a call application.

[0036] In this embodiment, the first terminal and the second terminal establish a call through a call application; the first terminal includes the terminal of either party participating in the call; the second terminal is the terminal of the other party participating in the call. The call application may be an application that provides the ability to make a call based on a phone number; taking the first user and the second user as an example, the first user inputs the phone number of the second user through the call application on the first terminal to initiate a call with the second user, and the second user can answer the call through the call application on the second terminal and perform voice interaction with the first user; wherein the call application may be an application pre-installed on the terminal when the terminal leaves the factory, and the call application may provide functions such as a dial pad, contact access, and call records; the user may initiate and establish a call in the call application based on the phone number. In this embodiment, in addition to establishing a call based on the phone number itself, a call may also be established based on a virtual number. The screen sharing processing method provided in this embodiment may be applied to a call server, and the call server may be a server corresponding to the call application, and the call server may implement audio and video transmission between terminals based on a communication protocol.

[0037] During the specific execution process, the first terminal establishes a call with the second terminal, and the first terminal may also perform the following operations: In response to the establishment of the call, audio is recorded through the call application to obtain a first audio stream; and / or, a second audio stream of the second terminal is received through the call application, and a broadcast component is called to broadcast the second audio stream.

[0038] The second terminal can also perform the following operations: In response to the establishment of the call, audio is recorded through the call application to obtain a second video stream; and / or, the first video stream of the first terminal is received through the call application, and the broadcast component is called to broadcast the first audio stream.

[0039] Correspondingly, the call server also performs the following operations: Receiving a first audio stream transmitted by the first terminal based on a call protocol; Sending the first audio stream to the second terminal based on the call protocol; and / or, Receiving a second audio stream transmitted by the second terminal based on the call protocol; The second audio stream is sent to the first terminal based on the call protocol.

[0040] Optionally, the call protocol includes RTP / RTCP protocol.

[0041] Specifically, the audio stream of the call established between the first terminal and the second terminal based on the call application is transmitted through the call server based on the call protocol.

[0042] Corresponding to the call server receiving the first audio stream transmitted by the first terminal based on the call protocol, in an optional implementation manner provided by this embodiment, if the first terminal detects that the call application calls the recording component, the first terminal performs audio recording based on the recording component to obtain the first audio stream; the first audio stream is transmitted to the call server through the call application, so as to send the first audio stream to the second terminal based on the call protocol.

[0043] Corresponding to the call server sending the second video stream to the first terminal, in an optional implementation provided by this embodiment, the first terminal may first receive the second audio stream sent by the call server through the call application; and then perform broadcast processing on the second audio stream based on the broadcast component; optionally, the second audio stream is transmitted to the call server by the second terminal through the call application.

[0044] It should also be noted that, in the process of establishing a call, call negotiation between the first terminal and the second terminal is required. In an optional implementation manner provided in this embodiment, the signaling server implements the call negotiation in the following manner: querying a service subscription record for a screen sharing service based on a terminal number of the first terminal and / or a terminal number of the second terminal; If the service contract record is found, the service contract record is sent to the call record server, so that the call record server sends the call record information to the contract server according to the service contract record.

[0045] In addition, in addition to querying and sending service contract records to the call bill server, the signaling server also sends relevant media data of the interaction between the first terminal and the second terminal to the call server during the call. The specific relevant media data is determined according to the actual configuration and is not limited in this embodiment.

[0046] Step S104: entering the service application according to the access instruction of the service application, and obtaining the screen sharing instruction submitted in the service application.

[0047] The screen sharing instruction in this embodiment is submitted by the first terminal participating in the call, including a screen sharing instruction for sharing the terminal screen of the first terminal to the second terminal participating in the call. Optionally, the screen sharing instruction enters the service application after the first terminal establishes a call based on the call application and triggers the screen sharing control in the service application to submit. The service application can be an application provided by an operator that assigns a phone number to a user, such as a China Mobile application; in addition, the service application may not exist in the form of an application, but may be a terminal native capability that can send video streams or other forms of data based on the Internet or a terminal integrated function that can send video streams or other forms of data based on the Internet; in this case, step S104 can also be replaced by obtaining a screen sharing instruction, and correspondingly, step S106 can also be replaced by performing screen recording in response to the screen sharing instruction, obtaining a screen video stream for screen sharing, and forming a new implementation method with other processing steps provided in this embodiment.

[0048] In a specific implementation, the first terminal and the second terminal establish a call through a call application. During the call between the first terminal and the second terminal, if the first terminal wants to share the screen with the second terminal, the service application running on the first terminal can be entered to obtain the screen sharing instruction submitted in the service application. In the process of obtaining the screen sharing instruction submitted in the service application, the screen sharing instruction can be submitted through the screen sharing control in the service application. Specifically, the first terminal enters the service application according to the access instruction to the service application to submit the screen sharing instruction to the application server through the screen sharing control in the service application. In addition, the screen sharing instruction can also be submitted through the call log data of the call. In this embodiment, after obtaining the screen sharing instruction, the screen sharing instruction can also be submitted to the application server. Optionally, the application server can be the server corresponding to the service application. Specifically, in the process of submitting a screen sharing instruction through the screen sharing control in the service application, after submitting the screen sharing instruction through the screen sharing control in the service application of the first terminal, the screen sharing instruction is submitted to the application server, and the application server can query the call list data of the call based on the phone number or virtual number of the first terminal according to the screen sharing instruction, and after querying the call list data, send the screen sharing instruction carrying the call identifier in the call list data to the signaling server; in this way, the user of the first terminal can realize "one-click screen sharing" through the service application during the call; wherein, the application server can send the screen sharing instruction to the signaling server through the HTTPS protocol; In addition, it is also possible to interact with the user to submit call record data. In an optional implementation provided by this embodiment, in the process of obtaining a screen sharing instruction submitted in a service application, the first terminal first obtains a service access instruction for a screen sharing service submitted in the service application, and in response to the service access instruction, displays the call record data of the call through the program interface of the service application, and finally obtains the screen sharing instruction submitted in the service application through the call record data. Specifically, a user may submit a service access instruction for a screen sharing service through a service application program running on a first terminal. The application server receives the service access instruction for a screen sharing service submitted by the first terminal after establishing a call based on a call application program, and queries the call list data for a call based on the terminal number according to the terminal number (telephone number) of the first terminal carried in the service access instruction and returns the query to the service application program. The first terminal receives the call list data through the service application program and displays it. If it is detected that the call list data is triggered, a screen sharing instruction carrying at least a call identifier in the call list data is submitted to the application server. The application server receives the screen sharing instruction for the call submitted based on the call list data and sends it to a signaling server. It should be noted that a user may also establish multiple calls simultaneously based on the call application program of the first terminal. When multiple calls are established, the application server queries the call list data of at least one call according to the terminal number and returns it to the service application program. The first terminal receives and displays at least one call list data through the service application program. If it is detected that the target call list data is triggered, a screen sharing instruction carrying at least a call identifier in the target call list data is submitted to the application server.

[0049] It should be noted that the user logs in to the service application through the phone number and then obtains the screen sharing instruction submitted in the service application through the first terminal; that is, the service application is logged in based on the phone number; optionally, the phone number for logging in to the service application can be the phone number for making calls.

[0050] This embodiment is described by taking a call record data as an example. Optionally, the call record data is sent from the call record server to the application server after the call is established. Figure 2 As shown, after the first terminal and the second terminal establish a call connection, the signaling server sends the IP, port and media information of the first terminal and the second terminal to the call server based on the http protocol, so that the call server forwards the audio and video data of the first terminal and the second terminal. After the first terminal and the second terminal establish a call connection, the signaling server also sends the call record data of the call to the call record server based on MQ (Message Queue), and the call record server sends the call record data to the application server of the service application based on the http protocol.

[0051] In addition to the above-mentioned submission of screen sharing instructions based on call record data, screen sharing instructions can also be submitted only by entering the screen sharing service in the service application. In an optional implementation provided by this embodiment, in the process of obtaining the screen sharing instruction submitted in the service application, the service access instruction for the screen sharing service submitted in the service application is first obtained. In response to the service access instruction, the screen sharing control is displayed through the program interface of the service application. If it is detected that the screen sharing control is triggered, the screen sharing instruction submitted in the service application is obtained; in this way, the call-based screen sharing instruction can be submitted without displaying the call record data, thereby improving the convenience and simplicity of submitting the screen sharing instruction; in order to further improve the perception of the program for starting screen recording and the perception of the recording end control, in an optional implementation provided by this embodiment, after obtaining the screen sharing instruction submitted in the service application, the program interface for displaying the screen sharing control is jumped to the program interface for displaying the recording end control in the service application, so that the recording end instruction can be submitted based on the recording end control displayed in the program interface for displaying the recording end control.

[0052] It should also be noted that after obtaining the screen sharing instruction submitted through the call record data in the service application, it is also possible to jump from the program interface displaying the screen sharing control to the program interface displaying the recording end control in the service application to enhance the perception of the submission of the end of the recording.

[0053] That is, the various implementation methods provided in this embodiment can be combined or replaced according to actual needs, and this embodiment is not limited here.

[0054] Step S106, in response to the screen sharing instruction, performing screen recording through the service application to obtain a screen video stream for screen sharing.

[0055] In specific implementation, in order to realize screen sharing based on calls, on the one hand, the first terminal needs to perform screen recording, and on the other hand, a video stream needs to be sent to the second terminal; in this embodiment, the screen is recorded by the service application at the first terminal, and the screen video stream is stored through the Internet. In the process of screen sharing with the second terminal, the screen video stream is sent to the call application of the second terminal through the call network to perform screen sharing; the screen video stream is sent to the call application of the second terminal through the call network, which needs to be sent through the call server. Therefore, after receiving the screen sharing instruction, the signaling server forwards the screen sharing instruction to the call server; Furthermore, after receiving the screen sharing instruction, the call server will read the screen video stream according to the call identifier carried by the screen sharing instruction. Therefore, the service application of the first terminal needs to record the screen of the first terminal and store it in the video stream storage area, so that the call server can obtain the screen video stream corresponding to the call from the video stream storage area to ensure the implementation of screen sharing; based on this, in addition to forwarding the screen sharing instruction to the call server, the signaling server also returns a screen sharing response to the application server; optionally, the screen sharing instruction is sent by the signaling server to the call server, and the signaling server forwards the screen sharing instruction to the call server after receiving the screen sharing instruction sent by the application server of the service application, and returns a screen sharing response to the application server; the application server receives the screen sharing response and returns it to the service application, so that the service application calls the screen recording component of the first terminal to perform screen recording. In this way, the signaling server forwards the screen sharing instruction to the call server, so that the call server reads the screen video stream and sends it to the second terminal. At the same time, the signaling server returns a screen sharing response to the application server, so that the application server returns the screen sharing response to the service application. The service application calls the screen recording component of the first terminal based on the screen sharing response to perform screen recording and stores the screen video stream obtained by the screen recording to the video stream storage area. In this way, by establishing a data link of "the service application stores the screen video stream to the video stream storage area - the call server reads the screen video stream from the video stream storage area", real-time and synchronous screen sharing is achieved. That is, in the process of the first terminal performing screen recording through the service application in response to the screen sharing instruction and obtaining the screen video stream for screen sharing, the screen can be recorded through the service application according to the screen sharing response returned by the application server to obtain the screen video stream, and further, the screen video stream can be sent to the video storage area for storage.

[0056] In the specific execution process, in response to the screen sharing instruction, during the process of recording the screen through the service application, the screen recording can be performed by calling the screen recording component through the service application. Optionally, if the first terminal detects the call of the screen recording component by the service application, the screen recording is performed based on the screen recording component to obtain the screen video stream. In this embodiment, the screen recording component can be the ReplayKit Extension of the IOS client, or the MediaProjection of the Android client; it can also be other components with screen recording function configured by the first terminal, which will not be described in detail in this embodiment.

[0057] In this embodiment, the first terminal and the second terminal establish a call through a call application. During the call between the first terminal and the second terminal, if the first terminal wants to share the screen with the second terminal, it can enter the service application running on the first terminal and submit the screen sharing instruction through the screen sharing control in the service application. Specifically, the first terminal enters the service application according to the access instruction to the service application to submit the screen sharing instruction to the application server through the screen sharing control in the service application. It should be noted that after the first terminal enters the service application running on the first terminal, the call application does not stop running. For example, after the first terminal and the second terminal establish a call through the call application, the first terminal displays the following Figure 3 The call interface shown; the user can exit the call interface to access the terminal interface. In the process of exiting the call interface to access the terminal interface, in order to enhance the user's awareness of the call, a call reminder can be displayed on the terminal interface. Optionally, after the first terminal enters the service application according to the access instruction to the service application, the program interface of the service application displays a call icon of the call; if it is detected that the call icon is triggered, the call application is entered and the call interface is displayed: Figure 4 As shown in 401 of the terminal interface shown in the figure, the call icon and call duration are displayed; the user can enter the service application through the access entrance 402 of the service application configured in the terminal interface, and the service interface of the service application can be configured with a screen sharing control, and the user can trigger the screen sharing control as shown in FIG. Figure 5 The screen sharing control 501 in the service interface shown submits the screen sharing instruction. In addition, in an optional implementation provided by this embodiment, after entering the service application according to the access instruction of the service application, the program interface of the service application and the call icon of the call can also be displayed. If it is detected that the call icon is triggered, the program interface jumps to the call interface of the call application. For example, the user triggers Figure 4 If the 401 in Figure 4 The terminal interface shown jumps to Figure 3 The call interface shown.

[0058] In order to enhance the user's perception of screen recording or screen sharing, a screen recording mark may be displayed through a screen interface (any interface operated by the user) during the screen recording process; wherein the screen recording mark may be the screen recording time. Optionally, during the screen recording process through the service application, a screen recording mark is displayed based on the screen interface. Figure 6The screen interface in the process of screen recording is shown, and 601 shows the sharing time T. It should be noted that during the screen recording process, the user may enter other applications. Although the call application and the service application cannot be seen from the screen interface, the call application runs to support the call between the first terminal and the second terminal, and the service application runs to support screen recording and the storage of screen video streams. Further, the user can also end the screen sharing through the screen recording mark. Optionally, if the screen recording mark is detected to be triggered, the screen recording is terminated and the screen sharing end reminder is displayed. In addition to terminating the screen recording through the screen recording mark, after the screen sharing instruction submitted in the service application is obtained, in the case of jumping from the program interface of the screen sharing control to the program interface of the recording end control in the service application, the recording end instruction can also be submitted based on the recording end control displayed in the program interface of the recording end control; Optionally, during the process of screen recording through the service application, if the recording end control is detected to be triggered, the recording end instruction submitted in the service application is obtained, and the screen recording is terminated after the recording end instruction is obtained and the screen sharing end reminder is displayed.

[0059] In this embodiment, the screen sharing instruction obtained by the call server is forwarded by the application server to the call server through the signaling server; after receiving the screen sharing instruction sent by the application server, the signaling server forwards the screen sharing instruction to the call server and returns a screen sharing response to the application server; the application server receives the screen sharing response and returns it to the service application, so that the service application performs screen recording of the first terminal.

[0060] In order to ensure that the second terminal can receive and display the screen video stream sent by the call server, and to avoid the situation where the second terminal cannot receive the screen video stream but the first terminal performs screen recording and video stream storage, resulting in the occupation of the storage area, in an optional implementation manner provided by this embodiment, before the signaling server sends the screen sharing instruction to the call server, the following operations are further performed: Performing unilateral video negotiation on the first terminal and the second terminal; If the negotiation is successful, the screen sharing instruction is forwarded to the call server.

[0061] Specifically, in order to ensure that the first terminal and the second terminal can successfully share the screen, the signaling server first performs unilateral video negotiation on the first terminal and the second terminal after receiving the screen sharing instruction sent by the application server. If the negotiation is successful, it forwards the screen sharing instruction to the call server and returns a screen sharing response to the application server.

[0062] In an optional implementation manner provided by this embodiment, the process of performing unilateral video negotiation on the first terminal and the second terminal is implemented in the following manner: Sending a request message with an empty session description protocol to the first terminal, and receiving a response message carrying a session description proposal returned by the first terminal; Performing encoding filtering on the session description proposal based on the access encoding format of the call server, and performing address modification on the session description proposal based on the address information of the call server to obtain a modified session description proposal; Sending a response request message carrying the proposal for modifying the session description to the second terminal, and receiving a session description response returned by the second terminal; A response detection is performed on the session description response, and a detection result is used as a negotiation result of the unilateral video negotiation.

[0063] Specifically, the signaling server first sends a request message with an empty session protocol to the first terminal, and receives a response message carrying a session description proposal returned by the first terminal, sends a response request message carrying a session description proposal to the second terminal, and receives a session description response returned by the second terminal, performs a response detection on the session description response, and uses the detection result as the negotiation result of the unilateral video negotiation.

[0064] In actual applications, data of the first terminal and the second terminal may pass through multiple network elements. In this embodiment, screen sharing is performed by forwarding the video stream through the call server. Therefore, after receiving the response information returned by the first terminal, the signaling server can encode and filter the session description proposal based on the access coding format of the call server, so that only the media format supported by the call server is retained, and the address information in the session description proposal is modified to the address information of the call server, to obtain the corrected session description proposal, and then send a response request message carrying the corrected session description proposal to the second terminal, and when the response detection of the session description response is successful, the session description response is encoded and filtered and the address is modified to obtain the target session description response, and a confirmation response containing the target session description response is sent to the first terminal.

[0065] In an optional implementation provided by this embodiment, during the process of detecting the response to the session description response, it is detected whether the session description response contains abnormal response information. If so, it is determined that the detection fails, that is, the negotiation fails. If not, it is determined that the detection succeeds, that is, the negotiation succeeds. Optionally, the abnormal response information includes at least one of the following: The session description response is a normal response and the video port in the session description response is 0, the session description response is a normal response and the media channel in the session description response is in a closed state, and the session description response is an abnormal response.

[0066] like Figure 7 As shown, the signaling server sends an invite message without SDP (Session Description Protocol) to the first terminal, and the first terminal returns a 200 response message carrying an SDP offer (Session Description Protocol Offer). The signaling server filters and encodes the SDP offer according to the media format supported by the call server, and modifies the media IP (Internet Protocol) and port information in the SDP offer to the media IP and port information of the call server, obtains the modified SDP offer, and sends a reinvite message carrying the modified SDP offer to the second terminal. After receiving the reinvite message, the second terminal returns a 200 response carrying the SDP answer. After receiving the 200 response returned by the second terminal, the signaling server first returns an ACK (Acknowledgment) confirmation to the second terminal, and then filters and encodes the SDP answer according to the media format supported by the call server, and modifies the media IP and port information in the SDP answer to the media IP and port information of the call server, obtains the modified SDP answer, and returns the modified SDP answer to the first terminal. answer; on this basis, the signaling server determines that the negotiation fails if it detects that the 200 response returned by the second terminal is a 2xx response, the video port is 0, or the 200 response returned is a 2xx response, the media stream direction is inactive, or the 200 response returned is not a 2xx response, otherwise, the negotiation is successful. After determining that the negotiation is successful, the signaling server sends a screen sharing instruction to the call server.

[0067] In a specific implementation, the call server obtains a screen sharing instruction sent by the signaling server after the unilateral video negotiation is successful. Optionally, the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; the call is established based on a virtual number.

[0068] After obtaining the screen sharing instruction of the first terminal forwarded by the application, the call server obtains the screen video stream obtained by screen recording of the first terminal through the service application according to the screen sharing instruction and sends the screen video stream to the second terminal corresponding to the call. In the process of obtaining the screen video stream obtained by screen recording of the first terminal through the service application, the screen video stream obtained by screen recording of the first terminal through the service application can be obtained according to the call instruction carried by the screen sharing instruction.

[0069] During the specific implementation process, after receiving the screen sharing response sent by the signaling server and issued by the application server, the service application of the first terminal calls the recording component to perform screen recording; in order to store the video stream, this embodiment can configure a video stream storage area for storing the screen video stream stored by the service application of the first terminal, and the communication server can also obtain the video stream obtained by performing screen recording of the first terminal through the service application from the video stream storage area based on the screen sharing instruction.

[0070] In this embodiment, the service application sends the screen video stream to the video stream storage area based on the RTMP protocol, and the call server needs to send the video stream to the second terminal through the call application, and needs to send the video stream based on the RTP / RTCP protocol, that is, it needs to transmit the screen video stream based on the Internet to the second terminal through the call network. In order to ensure the effectiveness of the transmission, the screen video stream needs to be transcoded so that the screen video stream based on the Internet can be transmitted based on the call network, and a transcoded video stream that can be transmitted based on the call network is obtained. In an optional implementation provided by this embodiment, after obtaining the screen video stream obtained by the screen recording of the first terminal by the service application, the call server transcodes the screen video stream according to the call network format to obtain a transcoded video stream; accordingly, sending the screen video stream to the second terminal corresponding to the call is sending the transcoded video stream to the second terminal.

[0071] In order to store the video stream, this embodiment can configure two video stream storage areas, one deployed at the sending end for receiving and storing the screen video stream sent by the service application; the other deployed at the receiving end for storing the transcoded video stream obtained after transcoding the screen video stream; specifically, after the service application performs screen recording, the screen video stream obtained by the screen recording and the call identifier are pushed to the first video stream storage area deployed at the sending end using the RTMP protocol; optionally, the video stream storage area can be a streaming media server; the streaming media server deployed at the sending end can be a first streaming media server, and the streaming media server deployed at the receiving end can be a second streaming media server. Specifically, the call server obtains the screen video stream obtained by the screen recording of the first terminal through the service application according to the screen sharing instruction, which can be the screen video stream obtained by the screen recording of the first terminal through the service application in the first streaming media server according to the screen sharing instruction.

[0072] In order to ensure the real-time and synchronization of screen sharing, in an optional implementation manner provided in this embodiment, in the process of transcoding the screen video stream according to the call network format to obtain the transcoded video stream, the following operations are performed: Inserting key frames into the screen video stream according to a preset video frame interval, and deleting the bidirectional prediction frames in the screen video stream to obtain an intermediate video stream; The intermediate video stream is format-converted according to the call network format to obtain the transcoded video stream.

[0073] Specifically, in order to reduce latency, the video is synchronized and detected, unsynchronized video frames are discarded, key frames are inserted into the screen video stream according to a preset video frame interval, and the bidirectional prediction frames in the screen video stream are deleted to obtain an intermediate video stream. The intermediate video stream is then transcoded according to a pre-stored call network format (the format defined by the call server) to obtain a transcoded video stream, and the transcoded video stream is stored in a second streaming media server.

[0074] In addition, the call server obtains the screen video stream obtained by screen recording of the first terminal through the service application according to the screen sharing instruction and transcodes the screen video stream. The process of obtaining the transcoded video stream can also be replaced by calling the transcoding service to read the screen video stream based on the call identifier carried by the screen sharing instruction, and transcoding the screen video stream according to the call network format to obtain the transcoded video stream.

[0075] Specifically, the transcoding service can read the screen video stream from the first streaming media server through RTMP based on the call identifier, and use ffmpeg to perform real-time transcoding processing on the screen video stream according to the format defined by the call media (for example, encoding format: H264, frame rate: 25-30 frames / second, frame format: AnnexB, specification: BaseLine), and then send the transcoded video stream obtained by the transcoding process to the second streaming media server through the RTMP protocol. In order to reduce the delay, RTMP reading can be read in a non-cached low-delay (-flags low_delay) mode, and the video frame is discarded (-framedrop) if the video is not synchronized. During transcoding, a key frame is inserted every ten frames (keyint=10) to reduce the video pull-up delay of the second terminal. The encoding preset and tune respectively use the fast and low-delay modes, and remove the B frames that increase the delay.

[0076] In the specific implementation process, in order to improve the efficiency of screen sharing, the call server can call the transcoding service to perform transcoding processing, obtain the transcoded video stream and store it in the second streaming media server, while reading the transcoded video stream from the second streaming media server. It should be noted that in this embodiment, the video streams in the first streaming media server and the second streaming media server are uniquely marked based on the call identifier. It should also be noted that the first streaming media server and the second streaming media server in this embodiment can be a streaming media server for storing screen video streams and transcoded video streams, which is not limited in this embodiment.

[0077] The call server sends the screen video stream to the second terminal corresponding to the call, which may be sent to the second terminal through the call network; the second terminal obtains the screen video stream sent by the call server and obtained by the first terminal through screen recording of the service application, and then displays the screen video stream on the call interface of the call application. In specific implementation, the call server sends the screen video stream or transcoded video stream to the second terminal through the RTP / RTCP protocol.

[0078] In addition, there is also a screen sharing from the second terminal to the first terminal. The relevant content of the process of the second terminal sharing the screen to the first terminal is similar to the above corresponding content, which will not be repeated here; in an optional implementation provided by this embodiment, in the process of the second terminal sharing the screen to the first terminal, the second terminal first establishes a call with the first terminal through the call application, and then enters the service application according to the access instruction of the service application to obtain the screen sharing instruction submitted in the service application; in response to the screen sharing instruction, the screen is recorded through the service application to obtain a shared screen video stream for screen sharing; correspondingly, in an optional implementation provided by this embodiment, the first terminal first obtains the shared screen video stream obtained by the second terminal through the screen recording of the service application through the call application, and then displays the shared screen video stream through the call interface of the call application. Optionally, the shared screen video stream is the screen video stream of the second terminal for screen sharing. Specifically, the first terminal receives the target transcoded video stream sent by the call server; then displays the target transcoded video stream on the call interface of the first terminal; optionally, the target transcoded video stream is obtained by transcoding the screen video stream obtained by the service application of the second terminal for screen recording of the second terminal.

[0079] It should also be noted that if the first terminal detects that the screen recording mark is triggered, in addition to terminating the screen recording and displaying a screen sharing end reminder, it also submits a screen sharing end instruction to the application server, and the application server forwards the screen sharing end instruction to the call server. The call server obtains the screen recording end instruction for the first terminal to end the screen recording, and ends the acquisition and sending of the screen video stream to the second terminal according to the screen recording end instruction, and sends the screen recording end instruction to the second terminal. The second terminal obtains the screen recording end instruction for the first terminal to end the screen recording sent by the call server, and based on the screen recording end instruction, displays the call interface of the call through the call application, and the call interface does not display the screen video stream.

[0080] In summary, the screen sharing processing method provided in this embodiment is that after the first terminal establishes a call with the second terminal through the call application, it can enter the service application and trigger the screen sharing control in the service application to submit a screen sharing instruction to the application server. The application server searches for the call of the first terminal according to the screen sharing instruction, adds the call identifier of the call to the screen sharing instruction, and sends the screen sharing instruction with the call identifier added to the signaling server; in this way, after the first terminal logs in to the service application based on the phone number of the call during the call, "one-click screen sharing" is implemented based on the service application, and screen sharing is implemented without relying on other tools, which also improves the activity of the service application; After receiving the screen sharing instruction, the signaling server performs unilateral video negotiation, i.e., signaling negotiation, with the first terminal and the second terminal based on the screen sharing instruction. If the negotiation is successful, the screen sharing instruction is forwarded to the call server and a screen sharing response is returned to the application server. After receiving the screen sharing response, the application server sends the screen sharing response to the service application. The service application calls the screen recording component of the first terminal to perform screen recording based on the screen sharing response, and pushes the screen video stream obtained by the screen recording to the video stream storage area through the Internet. In this way, the screen sharing instruction is forwarded to the call server and the screen sharing response is returned to the application server at the same time, so that screen recording and video stream reading are realized at the same time, thereby ensuring the real-time and synchronization of screen sharing; After obtaining the screen sharing instruction sent by the signaling server, the call server, on the one hand, calls the transcoding service to read the screen video stream in the video stream storage area according to the call identifier carried by the screen sharing instruction and transcodes the screen video stream according to the call network format, obtains the transcoded video stream and stores the transcoded video stream in the video stream storage area; on the other hand, reads the transcoded video stream in the video stream storage area based on the call identifier, and sends the transcoded video stream to the second terminal through the call network for display on the call interface of the second terminal, thereby realizing screen sharing from the first terminal participating in the call to the second terminal. In this way, screen sharing can be achieved only through a call without the need for the second terminal to download any service application, thereby improving the convenience of screen sharing.

[0081] It should be noted that the screen sharing processing method provided in the above embodiment can be applied to the first terminal, the second screen sharing processing method provided in the following embodiment can be applied to the second terminal, and the third screen sharing processing method provided can be applied to the call server. The screen sharing processing method applicable to the first terminal provided above has a coordinated relationship with the second screen sharing processing method applicable to the second terminal and the third screen sharing processing method applicable to the call server provided below during the execution process. When reading the above embodiment, you can refer to the relevant content of the following embodiment, and when reading the following embodiment, you can also refer to the relevant content of the above embodiment.

[0082] The following combination Figure 8 and attached Fig. 9 , taking the application of the screen sharing processing method provided by this embodiment in the screen sharing scenario based on a call as an example, the screen sharing processing method provided by this embodiment is further described. Fig. 9 The screen sharing processing method applied to the call-based screen sharing scenario specifically includes steps S902 to S914.

[0083] like Figure 8 As shown, during a call between a first terminal and a second terminal through a call application, the first terminal sends an audio stream to a call server based on the RTP / RTCP protocol, and the call server sends an audio stream to the second terminal based on the RTP / RTCP protocol. Similarly, the second terminal sends an audio stream to the call server based on the RTP / RTCP protocol, and the call server sends an audio stream to the first terminal based on the RTP / RTCP protocol; The first terminal enters the service application, and triggers the screen sharing control in the service application to submit a screen sharing instruction to the application server based on the https protocol. After receiving the screen sharing instruction, the application server adds the call identifier of the call of the first terminal to the screen sharing instruction, and sends the screen sharing instruction to the signaling server through the https protocol. After receiving the screen sharing instruction, the signaling server performs unilateral video negotiation on the first terminal and the second terminal, and if the negotiation is successful, sends the screen sharing instruction to the call server based on the https protocol, and returns a screen sharing response to the application server. The application server sends the screen sharing response to the service application, and the service application calls the screen recording component of the first terminal to perform screen recording based on the screen sharing response, obtains the screen video stream, and sends it to the first streaming media server based on the RTMP protocol; After the call server obtains the screen sharing instruction sent by the signaling server, on the one hand, it calls the transcoding service to first read the screen video stream from the first streaming media server through the RTMP protocol according to the call identifier carried by the screen sharing instruction and transcode the screen video stream, obtain the transcoded video stream and store it on the second streaming media server based on RTMP; on the other hand, it reads the transcoded video stream from the second streaming media server through the RTMP protocol according to the call identifier; finally, based on the RTP / RTCP protocol, it sends the transcoded video stream to the second terminal corresponding to the call for display on the call interface of the second terminal.

[0084] Step S902: Establish a call with the second terminal through the phone number in the call application.

[0085] Step S904: Enter the service application according to the access instruction of the service application.

[0086] Step S906: Obtain the service access instruction for the screen sharing service submitted in the service application and submit it to the application server.

[0087] Optionally, the application server queries the call record data of calls based on the terminal number based on the terminal number carried in the service access instruction.

[0088] Step S908, receiving the call record data through the service application, and displaying the call record data through the program interface of the service application.

[0089] Optionally, the program interface also displays a call icon for the call.

[0090] Step S910: Obtain the screen sharing instruction submitted through the call record data and submit it to the application server.

[0091] Optionally, the application server receives the screen sharing instruction and forwards it to the signaling server. After receiving the screen sharing instruction, the signaling server performs unilateral video negotiation on the first terminal and the second terminal. If the negotiation is successful, it forwards the screen sharing instruction to the call server and sends a screen sharing response to the application server.

[0092] Step S912: receiving a screen sharing response returned by the application server through the service application.

[0093] Step S914: the service application performs screen recording according to the screen sharing response returned by the application server, obtains the screen video stream for screen sharing and stores it in the video stream storage area.

[0094] The second screen sharing processing method embodiment provided in this specification is as follows: Reference Fig.10, which shows a processing flow chart of a screen sharing processing method provided in this embodiment, which can be applied to the second terminal. The screen sharing processing method provided in this embodiment specifically includes the following steps S1002 to S1006.

[0095] Step S1002: Establishing a call with the first terminal through a call application.

[0096] In this embodiment, the first terminal and the second terminal establish a call through a call application; the first terminal includes the terminal of either party participating in the call; the second terminal is the terminal of the other party participating in the call. The call application may be an application that provides the ability to make a call based on a phone number; taking the first user and the second user as an example, the first user inputs the phone number of the second user through the call application on the first terminal to initiate a call with the second user, and the second user can answer the call and perform voice interaction with the first user through the call application on the second terminal; wherein the call application may be an application pre-installed on the terminal when the terminal leaves the factory, and the call application may provide functions such as a dial pad, contact access, and call log; the user may initiate and establish a call in the call application based on the phone number. In this embodiment, in addition to establishing a call based on the phone number itself, a call may also be established based on a virtual number.

[0097] In an optional implementation manner provided by this embodiment, in response to the establishment of the call, audio is recorded through the call application to obtain a second video stream; and / or, the first video stream of the first terminal is received through the call application, and the broadcast component is called to broadcast the first audio stream.

[0098] In addition, after the second terminal establishes a call with the first terminal through the call application, it can initiate screen sharing to the first terminal. In an optional implementation provided by this embodiment, during the process of the second terminal sharing the screen with the first terminal, the second terminal first establishes a call with the first terminal through the call application, and then enters the service application according to the access instruction of the service application to obtain the screen sharing instruction submitted in the service application; in response to the screen sharing instruction, the screen is recorded through the service application to obtain a shared screen video stream for screen sharing; correspondingly, in an optional implementation provided by this embodiment, the first terminal first obtains the shared screen video stream obtained by the second terminal through the screen recording of the service application through the call application, and then displays the shared screen video stream through the call interface of the call application. Optionally, the shared screen video stream is the screen video stream of the second terminal for screen sharing. Specifically, the first terminal receives the target transcoded video stream sent by the call server; then displays the target transcoded video stream on the call interface of the first terminal; optionally, the target transcoded video stream is obtained by transcoding the screen video stream obtained by the service application of the second terminal for screen recording of the second terminal.

[0099] Step S1004: acquiring a screen video stream received by the call application and obtained by the first terminal performing screen recording through the service application.

[0100] Step S1006: display the screen video stream on the call interface of the call application.

[0101] In an optional implementation manner provided by this embodiment, if a screen recording end instruction for ending screen recording of the first terminal is obtained; based on the screen recording end instruction, a call interface of the call is displayed through the call application.

[0102] During the screen sharing process, the second terminal can achieve shared access to the screen of the first terminal through the call application without installing the service application. That is, it is sufficient to install the service application on the terminal that initiates the screen sharing to initiate the screen sharing, and the terminal that receives the screen sharing does not need to install the service application, and can access the screen video stream for screen sharing obtained by the terminal that initiates the screen sharing based on the screen recording of the service application through the call application.

[0103] The third screen sharing processing method embodiment provided in this specification is as follows: Reference Fig.11, which shows a processing flow chart of a screen sharing processing method provided in this embodiment, which can be applied to a call server. The screen sharing processing method provided in this embodiment specifically includes the following steps S1102 to S1106.

[0104] Step S1102: obtaining a screen sharing instruction of the first terminal forwarded by the application server.

[0105] Optionally, the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application.

[0106] In this embodiment, the first terminal and the second terminal establish a call through a call application; the first terminal includes the terminal of either party participating in the call; the second terminal is the terminal of the other party participating in the call. The call application may be an application that provides the ability to make a call based on a phone number; taking the first user and the second user as an example, the first user inputs the phone number of the second user through the call application on the first terminal to initiate a call with the second user, and the second user can answer the call through the call application on the second terminal and perform voice interaction with the first user; wherein the call application may be an application pre-installed on the terminal when the terminal leaves the factory, and the call application may provide functions such as a dial pad, contact access, and call records; the user may initiate and establish a call in the call application based on the phone number. In this embodiment, in addition to establishing a call based on the phone number itself, a call may also be established based on a virtual number. The screen sharing processing method provided in this embodiment may be applied to a call server, and the call server may be a server corresponding to the call application, and the call server may implement audio and video transmission between terminals based on a communication protocol.

[0107] During the specific execution process, the first terminal establishes a call with the second terminal, and the first terminal may also perform the following operations: In response to the establishment of the call, audio is recorded through the call application to obtain a first audio stream; and / or, a second audio stream of the second terminal is received through the call application, and a broadcast component is called to broadcast the second audio stream.

[0108] The second terminal can also perform the following operations: In response to the establishment of the call, audio is recorded through the call application to obtain a second video stream; and / or, the first video stream of the first terminal is received through the call application, and the broadcast component is called to broadcast the first audio stream.

[0109] Correspondingly, the call server also performs the following operations: Receiving a first audio stream transmitted by the first terminal based on a call protocol; Sending the first audio stream to the second terminal based on the call protocol; and / or, Receiving a second audio stream transmitted by the second terminal based on the call protocol; The second audio stream is sent to the first terminal based on the call protocol.

[0110] Optionally, the call protocol includes RTP / RTCP protocol.

[0111] Specifically, the audio stream of the call established between the first terminal and the second terminal based on the call application is transmitted through the call server based on the call protocol.

[0112] Corresponding to the call server receiving the first audio stream transmitted by the first terminal based on the call protocol, in an optional implementation manner provided by this embodiment, if the first terminal detects that the call application calls the recording component, the first terminal performs audio recording based on the recording component to obtain the first audio stream; the first audio stream is transmitted to the call server through the call application, so as to send the first audio stream to the second terminal based on the call protocol.

[0113] Corresponding to the call server sending the second video stream to the first terminal, in an optional implementation provided by this embodiment, the first terminal may first receive the second audio stream sent by the call server through the call application; and then perform broadcast processing on the second audio stream based on the broadcast component; optionally, the second audio stream is transmitted to the call server by the second terminal through the call application.

[0114] It should also be noted that, in the process of establishing a call, call negotiation between the first terminal and the second terminal is required. In an optional implementation manner provided in this embodiment, the signaling server implements the call negotiation in the following manner: querying a service subscription record for a screen sharing service based on a terminal number of the first terminal and / or a terminal number of the second terminal; If the service contract record is found, the service contract record is sent to the call record server, so that the call record server sends the call record information to the contract server according to the service contract record.

[0115] In addition, in addition to querying and sending service contract records to the call bill server, the signaling server also sends relevant media data of the interaction between the first terminal and the second terminal to the call server during the call. The specific relevant media data is determined according to the actual configuration and is not limited in this embodiment.

[0116] After the first terminal establishes a call with the second terminal through the call application, it enters the service application according to the access instruction of the service application, obtains the screen sharing instruction submitted in the service application, responds to the screen sharing instruction, performs screen recording through the service application, and obtains the screen video stream for screen sharing.

[0117] The screen sharing instruction in this embodiment is submitted by the first terminal participating in the call, including a screen sharing instruction for sharing the terminal screen of the first terminal to the second terminal participating in the call. Optionally, the screen sharing instruction enters the service application after the first terminal establishes a call based on the call application and triggers the screen sharing control in the service application to be submitted. The service application may be an application provided by an operator that assigns a phone number to a user, such as a China Mobile application.

[0118] In a specific implementation, the first terminal and the second terminal establish a call through a call application. During the call between the first terminal and the second terminal, if the first terminal wants to share the screen with the second terminal, the service application running on the first terminal can be entered to obtain the screen sharing instruction submitted in the service application. In the process of obtaining the screen sharing instruction submitted in the service application, the screen sharing instruction can be submitted through the screen sharing control in the service application. Specifically, the first terminal enters the service application according to the access instruction to the service application to submit the screen sharing instruction to the application server through the screen sharing control in the service application. In addition, the screen sharing instruction can also be submitted through the call log data of the call. In this embodiment, after obtaining the screen sharing instruction, the screen sharing instruction can also be submitted to the application server. Optionally, the application server can be the server corresponding to the service application. Specifically, in the process of submitting a screen sharing instruction through the screen sharing control in the service application, after submitting the screen sharing instruction through the screen sharing control in the service application of the first terminal, the screen sharing instruction is submitted to the application server, and the application server can query the call list data of the call based on the phone number or virtual number of the first terminal according to the screen sharing instruction, and after querying the call list data, send the screen sharing instruction carrying the call identifier in the call list data to the signaling server; in this way, the user of the first terminal can realize "one-click screen sharing" through the service application during the call; wherein, the application server can send the screen sharing instruction to the signaling server through the HTTPS protocol; In addition, it is also possible to interact with the user to submit call record data. In an optional implementation provided by this embodiment, in the process of obtaining a screen sharing instruction submitted in a service application, the first terminal first obtains a service access instruction for a screen sharing service submitted in the service application, and in response to the service access instruction, displays the call record data of the call through the program interface of the service application, and finally obtains the screen sharing instruction submitted in the service application through the call record data. Specifically, a user may submit a service access instruction for a screen sharing service through a service application program running on a first terminal. The application server receives the service access instruction for a screen sharing service submitted by the first terminal after establishing a call based on a call application program, and queries the call list data for a call based on the terminal number according to the terminal number (telephone number) of the first terminal carried in the service access instruction and returns the query to the service application program. The first terminal receives the call list data through the service application program and displays it. If it is detected that the call list data is triggered, a screen sharing instruction carrying at least a call identifier in the call list data is submitted to the application server. The application server receives the screen sharing instruction for the call submitted based on the call list data and sends it to a signaling server. It should be noted that a user may also establish multiple calls simultaneously based on the call application program of the first terminal. When multiple calls are established, the application server queries the call list data of at least one call according to the terminal number and returns it to the service application program. The first terminal receives and displays at least one call list data through the service application program. If it is detected that the target call list data is triggered, a screen sharing instruction carrying at least a call identifier in the target call list data is submitted to the application server.

[0119] It should be noted that the user logs in to the service application through the phone number and then obtains the screen sharing instruction submitted in the service application through the first terminal; that is, the service application is logged in based on the phone number; optionally, the phone number for logging in to the service application can be the phone number for making calls.

[0120] This embodiment is described by taking a call record data as an example. Optionally, the call record data is sent from the call record server to the application server after the call is established. Figure 2 As shown, after the first terminal and the second terminal establish a call connection, the signaling server sends the IP, port and media information of the first terminal and the second terminal to the call server based on the http protocol, so that the call server forwards the audio and video data of the first terminal and the second terminal. After the first terminal and the second terminal establish a call connection, the signaling server also sends the call record data of the call to the call record server based on MQ (Message Queue), and the call record server sends the call record data to the application server of the service application based on the http protocol.

[0121] In addition to the above-mentioned submission of screen sharing instructions based on call record data, screen sharing instructions can also be submitted only by entering the screen sharing service in the service application. In an optional implementation provided by this embodiment, in the process of obtaining the screen sharing instruction submitted in the service application, the service access instruction for the screen sharing service submitted in the service application is first obtained. In response to the service access instruction, the screen sharing control is displayed through the program interface of the service application. If it is detected that the screen sharing control is triggered, the screen sharing instruction submitted in the service application is obtained; in this way, the call-based screen sharing instruction can be submitted without displaying the call record data, thereby improving the convenience and simplicity of submitting the screen sharing instruction; in order to further improve the perception of the program for starting screen recording and the perception of the recording end control, in an optional implementation provided by this embodiment, after obtaining the screen sharing instruction submitted in the service application, the program interface for displaying the screen sharing control is jumped to the program interface for displaying the recording end control in the service application, so that the recording end instruction can be submitted based on the recording end control displayed in the program interface for displaying the recording end control.

[0122] It should also be noted that after obtaining the screen sharing instruction submitted through the call record data in the service application, it is also possible to jump from the program interface displaying the screen sharing control to the program interface displaying the recording end control in the service application to enhance the perception of the submission of the end of the recording.

[0123] That is, the various implementation methods provided in this embodiment can be combined or replaced according to actual needs, and this embodiment is not limited here.

[0124] In specific implementation, in order to realize screen sharing based on calls, on the one hand, the first terminal needs to perform screen recording, and on the other hand, a video stream needs to be sent to the second terminal; in this embodiment, the screen is recorded by the service application at the first terminal, and the screen video stream is stored through the Internet. In the process of screen sharing with the second terminal, a transcoded video stream is sent to the call application of the second terminal through the call network to perform screen sharing; the transcoded video stream is sent to the call application of the second terminal through the call network, and it needs to be sent through the call server. Therefore, after receiving the screen sharing instruction, the signaling server forwards the screen sharing instruction to the call server; Furthermore, after receiving the screen sharing instruction, the call server will read the screen video stream according to the call identifier carried by the screen sharing instruction. Therefore, the service application of the first terminal needs to record the screen of the first terminal and store it in the video stream storage area, so that the call server can obtain the screen video stream corresponding to the call from the video stream storage area to ensure the implementation of screen sharing; based on this, in addition to forwarding the screen sharing instruction to the call server, the signaling server also returns a screen sharing response to the application server; optionally, the screen sharing instruction is sent by the signaling server to the call server, and the signaling server forwards the screen sharing instruction to the call server after receiving the screen sharing instruction sent by the application server of the service application, and returns a screen sharing response to the application server; the application server receives the screen sharing response and returns it to the service application, so that the service application calls the screen recording component of the first terminal to perform screen recording. In this way, the signaling server forwards the screen sharing instruction to the call server, so that the call server reads the screen video stream and sends it to the second terminal. At the same time, the signaling server returns a screen sharing response to the application server, so that the application server returns the screen sharing response to the service application. The service application calls the screen recording component of the first terminal based on the screen sharing response to perform screen recording and stores the screen video stream obtained by the screen recording to the video stream storage area. In this way, by establishing a data link of "the service application stores the screen video stream to the video stream storage area - the call server reads the screen video stream from the video stream storage area", real-time and synchronous screen sharing is achieved. That is, in the process of the first terminal performing screen recording through the service application in response to the screen sharing instruction and obtaining the screen video stream for screen sharing, the screen can be recorded through the service application according to the screen sharing response returned by the application server to obtain the screen video stream, and further, the screen video stream can be sent to the video storage area for storage.

[0125] In the specific execution process, in response to the screen sharing instruction, during the process of recording the screen through the service application, the screen recording can be performed by calling the screen recording component through the service application. Optionally, if the first terminal detects the call of the screen recording component by the service application, the screen recording is performed based on the screen recording component to obtain the screen video stream. In this embodiment, the screen recording component can be the ReplayKit Extension of the IOS client, or the MediaProjection of the Android client; it can also be other components with screen recording function configured by the first terminal, which will not be described in detail in this embodiment.

[0126] In this embodiment, the first terminal and the second terminal establish a call through a call application. During the call between the first terminal and the second terminal, if the first terminal wants to share the screen with the second terminal, it can enter the service application running on the first terminal and submit the screen sharing instruction through the screen sharing control in the service application. Specifically, the first terminal enters the service application according to the access instruction to the service application to submit the screen sharing instruction to the application server through the screen sharing control in the service application. It should be noted that after the first terminal enters the service application running on the first terminal, the call application does not stop running. For example, after the first terminal and the second terminal establish a call through the call application, the first terminal displays the following Figure 3 The call interface shown; the user can exit the call interface to access the terminal interface. In the process of exiting the call interface to access the terminal interface, in order to enhance the user's awareness of the call, a call reminder can be displayed on the terminal interface. Optionally, after the first terminal enters the service application according to the access instruction to the service application, the program interface of the service application displays a call icon of the call; if it is detected that the call icon is triggered, the call application is entered and the call interface is displayed: Figure 4 As shown in 401 of the terminal interface shown in the figure, the call icon and call duration are displayed; the user can enter the service application through the access entrance 402 of the service application configured in the terminal interface, and the service interface of the service application can be configured with a screen sharing control, and the user can trigger the screen sharing control as shown in FIG. Figure 5 The screen sharing control 501 in the service interface shown submits the screen sharing instruction. In addition, in an optional implementation provided by this embodiment, after entering the service application according to the access instruction of the service application, the program interface of the service application and the call icon of the call can also be displayed. If it is detected that the call icon is triggered, the program interface jumps to the call interface of the call application. For example, the user triggers Figure 4 If the 401 in Figure 4 The terminal interface shown jumps to Figure 3 The call interface shown.

[0127] In order to enhance the user's perception of screen recording or screen sharing, a screen recording mark may be displayed through a screen interface (any interface operated by the user) during the screen recording process; wherein the screen recording mark may be the screen recording time. Optionally, during the screen recording process through the service application, a screen recording mark is displayed based on the screen interface. Figure 6The screen interface in the process of screen recording is shown, and 601 shows the sharing time T. It should be noted that during the screen recording process, the user may enter other applications. Although the call application and the service application cannot be seen from the terminal interface, the call application runs to support the call between the first terminal and the second terminal, and the service application runs to support screen recording and the storage of screen video streams. Further, the user can also end the screen sharing through the screen recording mark. Optionally, if the screen recording mark is detected to be triggered, the screen recording is terminated and the screen sharing end reminder is displayed. In addition to terminating the screen recording through the screen recording mark, after the screen sharing instruction submitted in the service application is obtained, in the case of jumping from the program interface of the screen sharing control to the program interface of the recording end control in the service application, the recording end instruction can also be submitted based on the recording end control displayed in the program interface of the recording end control; Optionally, during the process of screen recording through the service application, if the recording end control is detected to be triggered, the recording end instruction submitted in the service application is obtained, and the screen recording is terminated after the recording end instruction is obtained and the screen sharing end reminder is displayed.

[0128] Step S1104: acquiring, according to the screen sharing instruction, a screen video stream obtained by recording the screen of the first terminal through the service application.

[0129] In this embodiment, the screen sharing instruction obtained by the call server is forwarded by the application server to the call server through the signaling server; after receiving the screen sharing instruction sent by the application server, the signaling server forwards the screen sharing instruction to the call server and returns a screen sharing response to the application server; the application server receives the screen sharing response and returns it to the service application, so that the service application performs screen recording of the first terminal.

[0130] In order to ensure that the second terminal can receive and display the screen video stream sent by the call server, and to avoid the situation where the second terminal cannot receive the screen video stream but the first terminal performs screen recording and video stream storage, resulting in the occupation of the storage area, in an optional implementation manner provided by this embodiment, before the signaling server sends the screen sharing instruction to the call server, the following operations are further performed: Performing unilateral video negotiation on the first terminal and the second terminal; If the negotiation is successful, the screen sharing instruction is forwarded to the call server.

[0131] Specifically, in order to ensure that the first terminal and the second terminal can successfully share the screen, the signaling server first performs unilateral video negotiation on the first terminal and the second terminal after receiving the screen sharing instruction sent by the application server. If the negotiation is successful, it forwards the screen sharing instruction to the call server and returns a screen sharing response to the application server.

[0132] In an optional implementation manner provided by this embodiment, the process of performing unilateral video negotiation on the first terminal and the second terminal is implemented in the following manner: Sending a request message with an empty session description protocol to the first terminal, and receiving a response message carrying a session description proposal returned by the first terminal; Performing encoding filtering on the session description proposal based on the access encoding format of the call server, and performing address modification on the session description proposal based on the address information of the call server to obtain a modified session description proposal; Sending a response request message carrying the proposal for modifying the session description to the second terminal, and receiving a session description response returned by the second terminal; A response detection is performed on the session description response, and a detection result is used as a negotiation result of the unilateral video negotiation.

[0133] Specifically, the signaling server first sends a request message with an empty session protocol to the first terminal, and receives a response message carrying a session description proposal returned by the first terminal, sends a response request message carrying a session description proposal to the second terminal, and receives a session description response returned by the second terminal, performs a response detection on the session description response, and uses the detection result as the negotiation result of the unilateral video negotiation.

[0134] In actual applications, data of the first terminal and the second terminal may pass through multiple network elements. In this embodiment, screen sharing is performed by forwarding the video stream through the call server. Therefore, after receiving the response information returned by the first terminal, the signaling server can encode and filter the session description proposal based on the access coding format of the call server, so that only the media format supported by the call server is retained, and the address information in the session description proposal is modified to the address information of the call server, to obtain the corrected session description proposal, and then send a response request message carrying the corrected session description proposal to the second terminal, and when the response detection of the session description response is successful, the session description response is encoded and filtered and the address is modified to obtain the target session description response, and a confirmation response containing the target session description response is sent to the first terminal.

[0135] In an optional implementation provided by this embodiment, during the process of detecting the response to the session description response, it is detected whether the session description response contains abnormal response information. If so, it is determined that the detection fails, that is, the negotiation fails. If not, it is determined that the detection succeeds, that is, the negotiation succeeds. Optionally, the abnormal response information includes at least one of the following: The session description response is a normal response and the video port in the session description response is 0, the session description response is a normal response and the media channel in the session description response is in a closed state, and the session description response is an abnormal response.

[0136] like Figure 7 As shown, the signaling server sends an invite message without SDP (Session Description Protocol) to the first terminal, and the first terminal returns a 200 response message carrying an SDP offer (Session Description Protocol Offer). The signaling server filters and encodes the SDP offer according to the media format supported by the call server, and modifies the media IP (Internet Protocol) and port information in the SDP offer to the media IP and port information of the call server, obtains the modified SDP offer, and sends a reinvite message carrying the modified SDP offer to the second terminal. After receiving the reinvite message, the second terminal returns a 200 response carrying the SDP answer. After receiving the 200 response returned by the second terminal, the signaling server first returns an ACK (Acknowledgment) confirmation to the second terminal, and then filters and encodes the SDP answer according to the media format supported by the call server, and modifies the media IP and port information in the SDP answer to the media IP and port information of the call server, obtains the modified SDP answer, and returns the modified SDP answer to the first terminal. answer; on this basis, the signaling server determines that the negotiation has failed if it detects that the 200 response returned by the second terminal is a 2xx response, the video port is 0, or the returned 200 response is a 2xx response, the media stream direction is inactive, or the returned 200 response is not a 2xx response, otherwise, it determines that the negotiation is successful.

[0137] In a specific implementation, the call server obtains a screen sharing instruction sent by the signaling server after the unilateral video negotiation is successful. Optionally, the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; the call is established based on a virtual number.

[0138] After obtaining the screen sharing instruction of the first terminal forwarded by the application, the call server obtains the screen video stream obtained by screen recording of the first terminal through the service application according to the screen sharing instruction and sends the screen video stream to the second terminal corresponding to the call. In the process of obtaining the screen video stream obtained by screen recording of the first terminal through the service application, the screen video stream obtained by screen recording of the first terminal through the service application can be obtained according to the call instruction carried by the screen sharing instruction.

[0139] During the specific implementation process, after receiving the screen sharing response sent by the signaling server and issued by the application server, the service application of the first terminal calls the recording component to perform screen recording; in order to store the video stream, this embodiment can configure a video stream storage area for storing the screen video stream stored by the service application of the first terminal, and the communication server can also obtain the screen from the video stream storage area based on the screen sharing instruction. The video stream obtained by performing screen recording of the first terminal through the service application.

[0140] In this embodiment, the service application sends the screen video stream to the video stream storage area based on the RTMP protocol, and the call server needs to send the video stream to the second terminal through the call application, and needs to send the video stream based on the RTP / RTCP protocol, that is, it needs to transmit the screen video stream based on the Internet to the second terminal through the call network. In order to ensure the effectiveness of the transmission, the screen video stream needs to be transcoded so that the screen video stream based on the Internet can be transmitted based on the call network, and a transcoded video stream that can be transmitted based on the call network is obtained. In an optional implementation provided by this embodiment, after obtaining the screen video stream obtained by the screen recording of the first terminal by the service application, the screen video stream is transcoded according to the call network format to obtain a transcoded video stream; accordingly, sending the screen video stream to the second terminal corresponding to the call is sending the transcoded video stream to the second terminal.

[0141] In order to store the video stream, this embodiment can configure two video stream storage areas, one deployed at the sending end for receiving and storing the screen video stream sent by the service application; the other deployed at the receiving end for storing the transcoded video stream obtained after transcoding the screen video stream; specifically, after the service application performs screen recording, the screen video stream obtained by the screen recording and the call identifier are pushed to the first video stream storage area deployed at the sending end using the RTMP protocol; optionally, the video stream storage area can be a streaming media server; the streaming media server deployed at the sending end can be a first streaming media server, and the streaming media server deployed at the receiving end can be a second streaming media server. Specifically, the call server obtains the screen video stream obtained by the screen recording of the first terminal through the service application according to the screen sharing instruction, which can be the screen video stream obtained by the screen recording of the first terminal through the service application in the first streaming media server according to the screen sharing instruction.

[0142] In order to ensure the real-time and synchronization of screen sharing, in an optional implementation manner provided in this embodiment, in the process of transcoding the screen video stream according to the call network format to obtain the transcoded video stream, the following operations are performed: Inserting key frames into the screen video stream according to a preset video frame interval, and deleting the bidirectional prediction frames in the screen video stream to obtain an intermediate video stream; The intermediate video stream is format-converted according to the call network format to obtain the transcoded video stream.

[0143] Specifically, in order to reduce latency, the video is synchronized and detected, unsynchronized video frames are discarded, key frames are inserted into the screen video stream according to a preset video frame interval, and the bidirectional prediction frames in the screen video stream are deleted to obtain an intermediate video stream. The intermediate video stream is then transcoded according to a pre-stored call network format (the format defined by the call server) to obtain a transcoded video stream, and the transcoded video stream is stored in a second streaming media server.

[0144] In addition, the call server obtains the screen video stream obtained by screen recording of the first terminal through the service application according to the screen sharing instruction and transcodes the screen video stream. The process of obtaining the transcoded video stream can also be replaced by calling the transcoding service to read the screen video stream based on the call identifier carried by the screen sharing instruction, and transcoding the screen video stream according to the call network format to obtain the transcoded video stream.

[0145] Specifically, the transcoding service can read the screen video stream from the first streaming media server through RTMP based on the call identifier, and use ffmpeg to perform real-time transcoding processing on the screen video stream according to the format defined by the call media (for example, encoding format: H264, frame rate: 25-30 frames / second, frame format: AnnexB, specification: BaseLine), and then send the transcoded video stream obtained by the transcoding process to the second streaming media server through the RTMP protocol. In order to reduce the delay, RTMP reading can be read in a non-cached low-delay (-flags low_delay) mode, and the video frame is discarded (-framedrop) if the video is not synchronized. During transcoding, a key frame is inserted every ten frames (keyint=10) to reduce the video pull-up delay of the second terminal. The encoding preset and tune respectively use the fast and low-delay modes, and remove the B frames that increase the delay.

[0146] In the specific implementation process, in order to improve the efficiency of screen sharing, the call server can call the transcoding service to perform transcoding processing, obtain the transcoded video stream and store it in the second streaming media server, while reading the transcoded video stream from the second streaming media server. It should be noted that in this embodiment, the video streams in the first streaming media server and the second streaming media server are uniquely marked based on the call identifier. It should also be noted that the first streaming media server and the second streaming media server in this embodiment can be a streaming media server for storing screen video streams and transcoded video streams, which is not limited in this embodiment.

[0147] Step S1106: Send the screen video stream to the second terminal corresponding to the call.

[0148] The call server sends the screen video stream to the second terminal corresponding to the call, which may be sent to the second terminal through the call network; the second terminal obtains the screen video stream sent by the call server and obtained by the first terminal through screen recording of the service application, and then displays the screen video stream on the call interface of the call application. In specific implementation, the call server sends the screen video stream or transcoded video stream to the second terminal through the RTP / RTCP protocol.

[0149] In addition, there is also a screen sharing from the second terminal to the first terminal. The relevant content of the process of the second terminal sharing the screen to the first terminal is similar to the above corresponding content, which will not be repeated here; in an optional implementation provided by this embodiment, in the process of the second terminal sharing the screen to the first terminal, the second terminal first establishes a call with the first terminal through the call application, and then enters the service application according to the access instruction of the service application to obtain the screen sharing instruction submitted in the service application; in response to the screen sharing instruction, the screen is recorded through the service application to obtain a shared screen video stream for screen sharing; correspondingly, in an optional implementation provided by this embodiment, the first terminal first obtains the shared screen video stream obtained by the second terminal through the screen recording of the service application through the call application, and then displays the shared screen video stream through the call interface of the call application. Optionally, the shared screen video stream is the screen video stream of the second terminal for screen sharing. Specifically, the first terminal receives the target transcoded video stream sent by the call server; then displays the target transcoded video stream on the call interface of the first terminal; optionally, the target transcoded video stream is obtained by transcoding the screen video stream obtained by the service application of the second terminal for screen recording of the second terminal. It should also be noted that if the first terminal detects that the screen recording mark is triggered, in addition to terminating the screen recording and displaying a screen sharing end reminder, it also submits a screen sharing end instruction to the application server, and the application server forwards the screen sharing end instruction to the call server. The call server obtains the screen recording end instruction for the first terminal to end the screen recording, and ends the acquisition and sending of the screen video stream to the second terminal according to the screen recording end instruction, and sends the screen recording end instruction to the second terminal. The second terminal obtains the screen recording end instruction for the first terminal to end the screen recording sent by the call server, and based on the screen recording end instruction, displays the call interface of the call through the call application, and the call interface does not display the screen video stream.

[0150] In summary, the screen sharing processing method provided in this embodiment is that after the first terminal establishes a call with the second terminal through the call application, it can enter the service application and trigger the screen sharing control in the service application to submit a screen sharing instruction to the application server. The application server searches for the call of the first terminal according to the screen sharing instruction, adds the call identifier of the call to the screen sharing instruction, and sends the screen sharing instruction with the call identifier added to the signaling server; in this way, after the first terminal logs in to the service application based on the phone number of the call during the call, "one-click screen sharing" is implemented based on the service application, and screen sharing is implemented without relying on other tools, which also improves the activity of the service application; After receiving the screen sharing instruction, the signaling server performs unilateral video negotiation, i.e., signaling negotiation, with the first terminal and the second terminal based on the screen sharing instruction. If the negotiation is successful, the screen sharing instruction is forwarded to the call server and a screen sharing response is returned to the application server. After receiving the screen sharing response, the application server sends the screen sharing response to the service application. The service application calls the screen recording component of the first terminal to perform screen recording based on the screen sharing response, and pushes the screen video stream obtained by the screen recording to the video stream storage area through the Internet. In this way, the screen sharing instruction is forwarded to the call server and the screen sharing response is returned to the application server at the same time, so that screen recording and video stream reading are realized at the same time, thereby ensuring the real-time and synchronization of screen sharing; After obtaining the screen sharing instruction sent by the signaling server, the call server, on the one hand, calls the transcoding service to read the screen video stream in the video stream storage area according to the call identifier carried by the screen sharing instruction and transcodes the screen video stream according to the call network format, obtains the transcoded video stream and stores the transcoded video stream in the video stream storage area; on the other hand, reads the transcoded video stream in the video stream storage area based on the call identifier, and sends the transcoded video stream to the second terminal through the call network for display on the call interface of the second terminal, thereby realizing screen sharing from the first terminal participating in the call to the second terminal. In this way, screen sharing can be achieved only through a call without the need for the second terminal to download any service application, thereby improving the convenience of screen sharing.

[0151] The following combination Figure 8 and attached Fig.12 , taking the application of the screen sharing processing method provided by this embodiment in the screen sharing scenario based on a call as an example, the screen sharing processing method provided by this embodiment is further described. Fig.12 The screen sharing processing method applied to the call-based screen sharing scenario specifically includes steps S1202 to S1210.

[0152] Step S1202 , obtaining a screen sharing instruction sent by a signaling server, which is submitted after the first terminal enters a processing application and triggers a screen sharing control in the processing application to establish a call with the second terminal based on the call application.

[0153] Step S1204 , calling a transcoding service to read from the first streaming media server according to the call identifier in the screen sharing instruction a screen video stream obtained by processing the screen recording of the first terminal by the application program.

[0154] Step S1206: The transcoding service inserts key frames into the screen video stream according to a preset video frame interval, and deletes the bidirectional prediction frames in the screen video stream to obtain an intermediate video stream.

[0155] Step S1208: The transcoding service converts the format of the intermediate video stream according to the call network format, obtains the transcoded video stream and stores the transcoded video stream to the second streaming media server.

[0156] Step S1210, reading the transcoded video stream from the second streaming media server according to the screen sharing instruction and sending it to the second terminal, so as to be displayed on the call interface of the second terminal.

[0157] The fourth screen sharing processing method embodiment provided in this specification is as follows: The screen sharing processing method applicable to the client of the service application provided in this embodiment and the screen sharing processing method applicable to the server of the call application provided in the following embodiment can cooperate with each other during the execution process. The relevant contents of the following embodiment can be referred to when reading this embodiment, and the relevant contents of this embodiment can be referred to when reading the following embodiment. It should also be noted that the screen sharing processing method applicable to the client of the service application provided in this embodiment and the screen sharing processing method applicable to the server of the call application provided in the following embodiment are similar to the relevant contents of the screen sharing processing method applicable to the first terminal provided in the first embodiment, the screen sharing processing method applicable to the second terminal provided in the second embodiment, and the screen sharing processing method applicable to the call server provided in the third embodiment, and can be referenced to each other or referenced after being adaptively modified to each other.

[0158] Reference Fig.13 , which shows a processing flow chart of a screen sharing processing method provided in this embodiment, which can be applied to the client of the service application. The screen sharing processing method provided in this embodiment specifically includes the following steps S1302 to S1306.

[0159] Step S1302: Enter the service application according to the access instruction of the service application.

[0160] Optionally, the access instruction is obtained after a call is established based on a call application.

[0161] This embodiment can be applied to a client of a service application, which is also a service application. The client of the service application can be run based on a first terminal; the first terminal can also establish a call with a second terminal through a call application.

[0162] In specific implementation, before the client of the service application running on the first terminal enters the service application according to the access instruction of the service application, the first terminal also establishes a call with the second terminal through the running call application or the client of the call application; the call application may be an application that provides the ability to make calls based on a phone number; taking the first user and the second user as an example, the first user inputs the phone number of the second user through the call application on the first terminal to initiate a call with the second user, and the second user can answer the call through the call application on the second terminal and perform voice interaction with the first user; wherein the call application may be an application pre-installed on the terminal when the terminal leaves the factory, and the call application may provide functions such as a dial pad, contact access and call log; the user may initiate and establish a call in the call application based on the phone number. In this embodiment, in addition to establishing a call based on the phone number itself, a call may also be established based on a virtual number. Optionally, the call is established based on a phone number. Among them, the phone number may be a real phone number or a virtual phone number, which is not limited in this embodiment.

[0163] In a specific implementation, the first terminal and the second terminal establish a call through a call application. During the call between the first terminal and the second terminal, if the first terminal wants to share the screen with the second terminal, it can enter the service application running on the first terminal, and the client of the service application enters the service application according to the access instruction of the service application.

[0164] Step S1304, obtaining a screen sharing instruction.

[0165] In the process of obtaining the screen sharing instruction, the screen sharing instruction submitted by triggering the screen sharing control in the service application can be obtained. Specifically, the service application is entered according to the access instruction of the service application. If it is detected that the screen sharing control displayed by the service application is triggered, the screen sharing instruction is obtained; or, in an optional implementation provided by this embodiment, the service access instruction of the screen sharing service is first obtained, and then based on the service access instruction, the first program interface containing the screen sharing control is displayed. If it is detected that the screen sharing control is triggered, it is determined that the screen sharing instruction is obtained. In this way, the convenience and simplicity of submitting the screen sharing instruction are improved; in order to further improve the perception of the program for starting screen recording and to improve the perception of the recording end control, in an optional implementation provided by this embodiment, after obtaining the screen sharing instruction, the display of the first program interface jumps to the display of the second program interface containing the recording end control, so that the recording end instruction can be submitted based on the recording end control displayed in the second program interface.

[0166] In addition, the screen sharing instruction may be obtained through call record data. In this embodiment, after the screen sharing instruction is obtained, the screen sharing instruction may be submitted to the server of the service application (hereinafter referred to as the application server).

[0167] Specifically, in the process of obtaining the screen sharing instruction, after obtaining the screen sharing instruction through the screen sharing control, the screen sharing instruction is submitted to the server of the service application. The server of the service application can query the call record data of the call based on the phone number or virtual number of the service application according to the screen sharing instruction, and after querying the call record data, send the screen sharing instruction carrying the call identifier in the call record data to the signaling server, so as to realize "one-click screen sharing" through the service application during the call; wherein, the server of the service application can send the screen sharing instruction to the signaling server through the HTTPS protocol; In addition, the call record data can also be submitted by interacting with the user. In an optional implementation provided by this embodiment, in the process of obtaining the screen sharing instruction, it can be implemented in the following manner: Get the service access instruction of the screen sharing service; Perform a call record data query based on the service access instruction, obtain the call record data of the call and display it; Obtain a screen sharing instruction submitted based on the call record data.

[0168] Specifically, the service application obtains a service access instruction for the screen sharing service and submits it to the server of the service application. The application server receives the service access instruction for the screen sharing service submitted by the service application based on the service access instruction, queries the call bill data of the call based on the phone number carried in the service access instruction and returns it to the service application. The service application receives the call bill data and displays it. If it is detected that the call bill data is triggered, the screen sharing instruction carrying at least the call identifier in the call bill data is submitted to the application server. The application server receives the screen sharing instruction for the call submitted based on the call bill data and sends it to the signaling server. It should be noted that the user can also establish multiple calls at the same time based on the call application of the first terminal. When multiple calls are established, the application server queries the call bill data of at least one call according to the terminal number and returns it to the service application. The service application receives at least one call bill data and displays it. If it is detected that the target call bill data is triggered, the screen sharing instruction carrying at least the call identifier in the target call bill data is submitted to the application server.

[0169] It should be noted that the user logs in to the service application through the phone number and then obtains the screen sharing instruction submitted in the service application through the first terminal; that is, the service application is logged in based on the phone number; optionally, the phone number for logging in to the service application can be the phone number for making calls.

[0170] This embodiment is described by taking a call record data as an example. Optionally, the call record data is sent from the call record server to the application server after the call is established. Specifically, after the first terminal and the second terminal establish a call connection, the signaling server sends the IP, port and media information of the first terminal and the second terminal to the call server based on the http protocol, so that the call server forwards the audio and video data of the first terminal and the second terminal. After the first terminal and the second terminal establish a call connection, the signaling server also sends the call record data of the call to the call record server based on MQ (Message Queue), and the call record server sends the call record data to the application server that serves the application based on the http protocol.

[0171] Step S1306: Perform screen recording in response to the screen sharing instruction to obtain a screen video stream for screen sharing.

[0172] In specific implementation, in order to realize screen sharing based on a call, on the one hand, the first terminal needs to perform screen recording, and on the other hand, a video stream needs to be sent to the second terminal; in this embodiment, the screen is recorded in the service application of the first terminal, and the screen video stream is stored through the Internet. In the process of screen sharing with the second terminal, the screen video stream is sent to the call application of the second terminal through the call network to perform screen sharing; the screen video stream is sent to the call application of the second terminal through the call network, which needs to be sent through the call service end (the service end of the call application). Therefore, after receiving the screen sharing instruction, the signaling server forwards the screen sharing instruction to the call service end; Furthermore, after receiving the screen sharing instruction, the call server will read the screen video stream according to the call identifier carried by the screen sharing instruction. Therefore, the service application is required to record the screen of the first terminal and store it in the video stream storage area, so that the call server can obtain the screen video stream corresponding to the call from the video stream storage area to ensure the implementation of screen sharing; based on this, in addition to forwarding the screen sharing instruction to the call server, the signaling server also returns a screen sharing response to the application server; optionally, the screen sharing instruction is sent by the signaling server to the call server, and the signaling server forwards the screen sharing instruction to the call server after receiving the screen sharing instruction sent by the application server, and returns a screen sharing response to the application server; the application server receives the screen sharing response and returns it to the service application, so that the service application calls the screen recording component of the first terminal to perform screen recording. In this way, the signaling server forwards the screen sharing instruction to the call server, so that the call server obtains the screen video stream and sends it to the second terminal. At the same time, the signaling server returns a screen sharing response to the application server, so that the application server returns the screen sharing response to the service application. The service application calls the screen recording component of the first terminal based on the screen sharing response to perform screen recording and stores the screen video stream obtained by the screen recording to the video stream storage area. In this way, by establishing a data link of "the service application stores the screen video stream to the video stream storage area - the call server obtains the screen video stream from the video stream storage area", real-time and synchronous screen sharing is achieved. That is, in the process of the service application performing screen recording in response to the screen sharing instruction and obtaining the screen video stream for screen sharing, the service application can perform screen recording according to the screen sharing response returned by the application server to obtain the screen video stream, and further, the screen video stream can be sent to the video storage area for storage.

[0173] In the specific execution process, when the service application performs screen recording in response to the screen sharing instruction, the screen recording can be performed by calling the screen recording component. Optionally, if the first terminal detects the call of the screen recording component by the service application, the screen recording is performed based on the screen recording component to obtain the screen video stream. In this embodiment, the screen recording component can be the ReplayKit Extension of the IOS client, or the Media Projection of the Android client; it can also be other components with screen recording function configured by the first terminal, which will not be described in detail in this embodiment.

[0174] In this embodiment, the first terminal and the second terminal establish a call through a call application. During the call between the first terminal and the second terminal, if the first terminal wants to share the screen with the second terminal, it can enter the service application running on the first terminal and submit the screen sharing instruction through the screen sharing control in the service application. Specifically, the first terminal enters the service application according to the access instruction to the service application to submit the screen sharing instruction to the application server through the screen sharing control in the service application. It should be noted that after the first terminal enters the service application running on the first terminal, the call application does not stop running. For example, after the first terminal and the second terminal establish a call through the call application, the first terminal displays the following Figure 3 The call interface shown; the user can exit the call interface to access the terminal interface. In the process of exiting the call interface to access the terminal interface, in order to enhance the user's awareness of the call, a call reminder can be displayed on the terminal interface. Optionally, after the first terminal enters the service application according to the access instruction to the service application, the program interface of the service application displays a call icon of the call; if it is detected that the call icon is triggered, the call application is entered and the call interface is displayed: Figure 4 As shown in 401 of the terminal interface shown in the figure, the call icon and call duration are displayed; the user can enter the service application through the access entrance 402 of the service application configured in the terminal interface, and the service interface of the service application can be configured with a screen sharing control, and the user can trigger the screen sharing control as shown in FIG. Figure 5 The screen sharing control 501 in the service interface shown submits the screen sharing instruction. In addition, in an optional implementation provided by this embodiment, after entering the service application according to the access instruction of the service application, the program interface of the service application and the call icon of the call can also be displayed. If it is detected that the call icon is triggered, the program interface jumps to the call interface of the call application. For example, the user triggers Figure 4 If the 401 in Figure 4 The terminal interface shown jumps to Figure 3 The call interface shown.

[0175] In order to enhance the user's perception of screen recording or screen sharing, a screen recording mark may be displayed through a screen interface (any interface operated by the user) during the screen recording process; wherein the screen recording mark may be the screen recording time. Optionally, during the screen recording process through the service application, a screen recording mark is displayed based on the screen interface. Figure 6The screen interface in the process of screen recording is shown, and 601 shows the sharing time T. It should be noted that during the screen recording process, the user may enter other applications. Although the call application and the service application cannot be seen from the terminal interface, the call application runs to support the call between the first terminal and the second terminal, and the service application runs to support screen recording and the storage of screen video streams. Further, the user can also end the screen sharing through the screen recording mark. Optionally, if the screen recording mark is detected to be triggered, the screen recording is terminated and the screen sharing end reminder is displayed. In addition to terminating the screen recording through the screen recording mark, after the above-mentioned screen sharing instruction is obtained, in the case of jumping from the display of the first program interface to the display of the second program interface containing the recording end control, the recording end instruction can also be submitted based on the recording end control displayed on the second program interface; Optionally, during the screen recording process, if the recording end control is detected to be triggered, the recording end instruction is obtained, and the screen recording is terminated after the recording end instruction is obtained and the screen sharing end reminder is displayed.

[0176] In this embodiment, the screen sharing instruction obtained by the call server is forwarded by the application server to the call server through the signaling server; after receiving the screen sharing instruction sent by the application server, the signaling server forwards the screen sharing instruction to the call server and returns a screen sharing response to the application server; the application server receives the screen sharing response and returns it to the service application, so that the service application records the screen of the first terminal.

[0177] In order to ensure that the second terminal can receive and display the screen video stream sent by the call server, and to avoid the second terminal being unable to receive the screen video stream but the first terminal performing screen recording and video stream storage, which causes occupation of the storage area, in an optional implementation manner provided by this embodiment, before the signaling server sends a screen sharing instruction to the call server, the following operations are further performed: Performing unilateral video negotiation on the first terminal and the second terminal; If the negotiation is successful, the screen sharing instruction is forwarded to the call service end.

[0178] Specifically, in order to ensure that the first terminal and the second terminal can successfully share the screen, the signaling server first performs unilateral video negotiation on the first terminal and the second terminal after receiving the screen sharing instruction sent by the application server. If the negotiation is successful, it forwards the screen sharing instruction to the call server and returns a screen sharing response to the application server.

[0179] In an optional implementation manner provided by this embodiment, the process of performing unilateral video negotiation on the first terminal and the second terminal is implemented in the following manner: Sending a request message with an empty session description protocol to the first terminal, and receiving a response message carrying a session description proposal returned by the first terminal; Performing encoding filtering on the session description proposal based on the access encoding format of the call service end, and performing address modification on the session description proposal based on the address information of the call service end to obtain a modified session description proposal; Sending a response request message carrying the proposal for modifying the session description to the second terminal, and receiving a session description response returned by the second terminal; A response detection is performed on the session description response, and a detection result is used as a negotiation result of the unilateral video negotiation.

[0180] Specifically, the signaling server first sends a request message with an empty session protocol to the first terminal, and receives a response message carrying a session description proposal returned by the first terminal, sends a response request message carrying a session description proposal to the second terminal, and receives a session description response returned by the second terminal, performs a response detection on the session description response, and uses the detection result as the negotiation result of the unilateral video negotiation.

[0181] In actual applications, data of the first terminal and the second terminal may pass through multiple network elements. In this embodiment, screen sharing is performed by forwarding the video stream through the call server. Therefore, after receiving the response information returned by the first terminal, the signaling server can encode and filter the session description proposal based on the access coding format of the call server, so that only the media format supported by the call server is retained, and the address information in the session description proposal is modified to the address information of the call server, to obtain the corrected session description proposal, and then send a response request message carrying the corrected session description proposal to the second terminal. When the response detection of the session description response is successful, the session description response is encoded and filtered and the address is modified to obtain the target session description response, and a confirmation response containing the target session description response is sent to the first terminal.

[0182] In an optional implementation provided by this embodiment, during the process of detecting the response to the session description response, it is detected whether the session description response contains abnormal response information. If so, it is determined that the detection fails, that is, the negotiation fails. If not, it is determined that the detection succeeds, that is, the negotiation succeeds. Optionally, the abnormal response information includes at least one of the following: The session description response is a normal response and the video port in the session description response is 0, the session description response is a normal response and the media channel in the session description response is in a closed state, and the session description response is an abnormal response.

[0183] For example, the signaling server sends an invite message that does not carry SDP (Session Description Protocol) to the first terminal, and the first terminal returns a 200 response message and carries an SDP offer (Session Description Protocol Proposal). The signaling server filters and encodes the SDP offer according to the media format supported by the call server, and modifies the media IP (Internet Protocol) and port information in the SDP offer to the media IP and port information of the call server, obtains the modified SDP offer, and sends a reinvite message carrying the modified SDP offer to the second terminal. After receiving the reinvite message, the second terminal returns a 200 response carrying the SDP answer. After receiving the 200 response returned by the second terminal, the signaling server first returns an ACK (Acknowledgment) confirmation to the second terminal, and then filters and encodes the SDP answer according to the media format supported by the call server, and modifies the media IP and port information in the SDP answer to the media IP and port information of the call server, obtains the modified SDP answer, and returns the modified SDP answer to the first terminal. answer; on this basis, the signaling server determines that the negotiation has failed if it detects that the 200 response returned by the second terminal is a 2xx response, the video port is 0, or the returned 200 response is a 2xx response, the media stream direction is inactive, or the returned 200 response is not a 2xx response, otherwise, it determines that the negotiation is successful.

[0184] In a specific implementation, the call service end obtains a screen sharing instruction sent by the signaling service end after the unilateral video negotiation is successful. Optionally, the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; the call is established based on a virtual number.

[0185] After obtaining the screen sharing instruction submitted by the service application after the call is established and forwarded by the application, the call server obtains the screen video stream obtained by screen recording through the service application according to the screen sharing instruction and sends the screen video stream to the call application. In the process of obtaining the screen video stream obtained by screen recording through the service application, the screen video stream obtained by screen recording of the first terminal through the service application can be obtained according to the call instruction carried by the screen sharing instruction.

[0186] During the specific implementation process, after receiving the screen sharing response sent by the signaling server and issued by the server, the service application calls the recording component to perform screen recording; in order to store the video stream, this embodiment can configure a video stream storage area to store the screen video stream obtained by the service application through screen recording, and the communication server can also obtain the video stream obtained by the service application through screen recording of the first terminal from the video stream storage area based on the screen sharing instruction.

[0187] In this embodiment, the service application sends the screen video stream to the video stream storage area based on the RTMP protocol, and the call service end needs to send the video stream to the call application running on the second terminal through the call application, and needs to send the video stream based on the RTP / RTCP protocol, that is, it is necessary to transmit the screen video stream based on the Internet to the second terminal through the call network. In order to ensure the effectiveness of the transmission, the screen video stream needs to be transcoded so that the screen video stream based on the Internet can be transmitted based on the call network, and a transcoded video stream that can be transmitted based on the call network is obtained. In an optional implementation provided by this embodiment, after obtaining the screen video stream obtained by the service application performing screen recording on the first terminal, the call service end transcodes the screen video stream according to the call network format to obtain a transcoded video stream; accordingly, sending the screen video stream to the call application of the second terminal corresponding to the call is sending the transcoded video stream to the call application of the second terminal.

[0188] In order to store the video stream, this embodiment can configure two video stream storage areas, one deployed at the sending end for receiving and storing the screen video stream sent by the service application; the other deployed at the receiving end for storing the transcoded video stream obtained after transcoding the screen video stream; specifically, after the service application performs screen recording, the screen video stream obtained by the screen recording and the call identifier are pushed to the first video stream storage area deployed at the sending end using the RTMP protocol; optionally, the video stream storage area can be a streaming media server; the streaming media server deployed at the sending end can be a first streaming media server, and the streaming media server deployed at the receiving end can be a second streaming media server. Specifically, the call server obtains the screen video stream obtained by the screen recording of the first terminal through the service application according to the screen sharing instruction, which can be the screen video stream obtained by the screen recording of the first terminal through the service application in the first streaming media server according to the screen sharing instruction.

[0189] In order to ensure the real-time and synchronization of screen sharing, in an optional implementation manner provided in this embodiment, in the process of transcoding the screen video stream according to the call network format to obtain the transcoded video stream, the following operations are performed: Inserting key frames into the screen video stream according to a preset video frame interval, and deleting the bidirectional prediction frames in the screen video stream to obtain an intermediate video stream; The intermediate video stream is format-converted according to the call network format to obtain the transcoded video stream.

[0190] Specifically, in order to reduce latency, the video is synchronized and detected, unsynchronized video frames are discarded, key frames are inserted into the screen video stream according to a preset video frame interval, and the bidirectional prediction frames in the screen video stream are deleted to obtain an intermediate video stream. The intermediate video stream is then transcoded according to a pre-stored call network format (a format defined by the call server) to obtain a transcoded video stream, and the transcoded video stream is stored in a second streaming media server.

[0191] In addition, the call service end obtains the screen video stream obtained by recording the screen of the first terminal through the service application according to the screen sharing instruction and transcodes the screen video stream. The process of obtaining the transcoded video stream can also be replaced by calling the transcoding service to read the screen video stream based on the call identifier carried by the screen sharing instruction, and transcoding the screen video stream according to the call network format to obtain the transcoded video stream.

[0192] Specifically, the transcoding service can read the screen video stream from the first streaming media server through RTMP based on the call identifier, and use ffmpeg to perform real-time transcoding processing on the screen video stream according to the format defined by the call media (for example, encoding format: H264, frame rate: 25-30 frames / second, frame format: AnnexB, specification: BaseLine), and then send the transcoded video stream obtained by the transcoding process to the second streaming media server through the RTMP protocol. In order to reduce the delay, RTMP reading can be read in a non-cached low-delay (-flags low_delay) mode, and the video frame is discarded (-framedrop) if the video is not synchronized. During transcoding, a key frame is inserted every ten frames (keyint=10) to reduce the video pull-up delay of the second terminal. The encoding preset and tune respectively use the fast and low-delay modes, and remove the B frames that increase the delay.

[0193] In the specific implementation process, in order to improve the efficiency of screen sharing, the call server can call the transcoding service to perform transcoding processing, obtain the transcoded video stream and store it in the second streaming media server, while reading the transcoded video stream from the second streaming media server. It should be noted that in this embodiment, the video streams in the first streaming media server and the second streaming media server are uniquely marked based on the call identifier. It should also be noted that the first streaming media server and the second streaming media server in this embodiment can be a streaming media server for storing screen video streams and transcoded video streams, which is not limited in this embodiment.

[0194] The call service end sends the screen video stream to the call application of the second terminal corresponding to the call, which may be to send the screen video stream to the call application of the second terminal through the call network; the call application of the second terminal obtains the screen video stream sent by the call service end and obtains the screen recording of the first terminal through the service application, and then displays the screen video stream on the call interface of the call application. In specific implementation, the call service end sends the screen video stream or the transcoded video stream to the second terminal through the RTP / RTCP protocol.

[0195] It should be noted that if the service application detects that the screen recording mark is triggered, in addition to terminating the screen recording and displaying a screen sharing end reminder, it also submits a screen sharing end instruction to the application server, and the application server forwards the screen sharing end instruction to the call server. The call server obtains the screen recording end instruction of the service application to end the screen recording, and ends the acquisition of the screen video stream and the sending of the screen video stream to the call application of the second terminal according to the screen recording end instruction, and sends the screen recording end instruction to the call application of the second terminal. The call application obtains the screen recording end instruction of the service application to end the screen recording sent by the call server, and based on the screen recording end instruction, displays the call interface of the call through the call application, and the call interface does not display the screen video stream.

[0196] In summary, the screen sharing processing method provided in this embodiment is that after the first terminal establishes a call with the second terminal through the call application, the client of the service application enters the service application according to the access instruction of the service application, obtains the screen sharing instruction, and submits the screen sharing instruction to the application server. The application server searches for the call of the service application according to the screen sharing instruction, adds the call identifier of the call to the screen sharing instruction, and sends the screen sharing instruction with the call identifier added to the signaling server; in this way, after logging in to the service application based on the phone number of the call during the call, "one-click screen sharing" is realized based on the service application, and screen sharing is realized without relying on other tools, which also improves the activity of the service application; After receiving the screen sharing instruction, the signaling server performs unilateral video negotiation with the first terminal and the second terminal based on the screen sharing instruction, that is, signaling negotiation. If the negotiation is successful, the screen sharing instruction is forwarded to the call server and a screen sharing response is returned to the application server. After receiving the screen sharing response, the application server sends the screen sharing response to the service application. The service application calls the screen recording component of the first terminal to perform screen recording based on the screen sharing response, and pushes the screen video stream obtained by the screen recording to the video stream storage area through the Internet. In this way, the screen sharing instruction is forwarded to the call server and the screen sharing response is returned to the application server at the same time, so that screen recording and video stream acquisition are realized at the same time, ensuring the real-time and synchronization of screen sharing; After receiving the screen sharing instruction sent by the signaling server, the call server, on the one hand, calls the transcoding service to read the screen video stream in the first video stream storage area according to the call identifier carried by the screen sharing instruction and transcodes the screen video stream according to the call network format, obtains the transcoded video stream and stores the transcoded video stream in the video stream storage area; on the other hand, reads the transcoded video stream in the second video stream storage area based on the call identifier, and sends the transcoded video stream to the second terminal through the call network, so as to display it in the call interface of the call application of the second terminal, thereby realizing screen sharing from the first terminal participating in the call to the second terminal. In this way, screen sharing can be realized only through the call without the need for the second terminal to download any service application, thereby improving the convenience of screen sharing.

[0197] The fifth screen sharing processing method embodiment provided in this specification is as follows: Reference Fig.14 , which shows a processing flow chart of a screen sharing processing method provided in this embodiment, which can be a call service end of a call application. The screen sharing processing method provided in this embodiment specifically includes the following steps S1402 to S1406.

[0198] Step S1402, obtaining the screen sharing instruction forwarded by the application server and submitted by the service application after the call is established.

[0199] Optionally, the call is established based on a call application.

[0200] In an optional implementation manner provided by this embodiment, the service application is run on the first terminal, and the first terminal establishes a call with the second terminal through the call application.

[0201] Step S1404: acquiring a screen video stream obtained by screen recording through the service application according to the screen sharing instruction.

[0202] Step S1406: Send the screen video stream to the call application.

[0203] Optionally, the call service end sends the screen video stream to a call application running on the second terminal.

[0204] Fig.15 A schematic diagram of a first screen sharing processing device provided by an embodiment of the present invention is shown in FIG. Fig.15 As shown, the device comprises: A call establishing module 1502, configured to establish a call with a second terminal through a call application; The instruction acquisition module 1504 is used to enter the service application according to the access instruction of the service application and acquire the screen sharing instruction submitted in the service application; The screen recording module 1506 is used to respond to the screen sharing instruction and perform screen recording through the service application to obtain a screen video stream for screen sharing.

[0205] The screen sharing processing device provided in this embodiment first establishes a call with the second terminal through the call application by running the call establishment module 1502, then enters the service application according to the access instruction of the service application by running the instruction acquisition module 1504, obtains the screen sharing instruction submitted in the service application, and then runs the screen recording module 1506, responds to the screen sharing instruction, performs screen recording through the service application, and obtains the screen video stream for screen sharing. Through the above-mentioned screen sharing processing, not only the screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved.

[0206] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0207] Fig.16 A schematic diagram of a second screen sharing processing device provided by an embodiment of the present invention is shown in FIG. Fig.16 As shown, the device comprises: A call establishing module 1602, configured to establish a call with the first terminal through a call application; The video stream acquisition module 1604 is used to acquire the screen video stream received by the call application and obtained by the first terminal through screen recording by the service application; The video stream display module 1606 is used to display the screen video stream on the call interface of the call application.

[0208] The screen sharing processing device provided in this embodiment first runs the call establishment module 1602 to establish a call with the first terminal through the call application, then runs the video stream acquisition module 1604 to obtain the screen video stream obtained by the first terminal through screen recording through the service application, and finally runs the video stream display module 1606 to display the screen video stream on the call interface of the call application. Through the above-mentioned screen sharing processing, not only the screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved.

[0209] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0210] Fig.17 A schematic diagram of a third screen sharing processing device provided by an embodiment of the present invention is shown in FIG. Fig.17 As shown, the device comprises: The instruction acquisition module 1702 is used to acquire a screen sharing instruction of the first terminal forwarded by the application server; the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; The video stream acquisition module 1704 is used to acquire, according to the screen sharing instruction, a screen video stream obtained by recording the screen of the first terminal through the service application; The video stream sending module 1706 is used to send the screen video stream to the second terminal corresponding to the call.

[0211] The screen sharing processing device provided in this embodiment first obtains the screen sharing instruction of the first terminal forwarded by the application server by running the instruction acquisition module 1702, and then obtains the screen video stream obtained by recording the screen of the first terminal through the service application according to the screen sharing instruction by running the video stream acquisition module 1704, and then sends the screen video stream to the second terminal corresponding to the call by running the video stream sending module 1706. Through the above-mentioned screen sharing processing, not only the screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved.

[0212] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0213] Fig.18 A schematic diagram of a fourth screen sharing processing device provided by an embodiment of the present invention is shown in FIG. Fig.18 As shown, the device comprises: An application entry module 1802, for entering the service application according to an access instruction of the service application; the access instruction is obtained after the call application establishes a call; The instruction acquisition module 1804 is used to acquire the screen sharing instruction; The screen recording module 1806 is used to perform screen recording in response to the screen sharing instruction and obtain a screen video stream for screen sharing.

[0214] The screen sharing processing device provided in this embodiment first enters module 1802 by running the application, enters the service application according to the access instruction of the service application, then obtains the screen sharing instruction by running the instruction acquisition module 1804, and finally runs the screen recording module 1806, performs screen recording in response to the screen sharing instruction, and obtains the screen video stream for screen sharing. Through the above-mentioned screen sharing processing, not only the screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved.

[0215] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0216] Fig.19 A fifth screen sharing processing device is provided as a schematic diagram of an embodiment of the present invention. Fig.19 As shown, the device comprises: The instruction acquisition module 1902 is used to acquire the screen sharing instruction submitted by the service application after the call is established and forwarded by the application server; the call is established based on the call application; The video stream acquisition module 1904 is used to acquire the screen video stream obtained by screen recording of the service application according to the screen sharing instruction; The video stream sending module 1906 is used to send the screen video stream to the call application.

[0217] The screen sharing processing device provided in this embodiment first obtains the screen sharing instruction submitted by the service application after the call is established and forwarded by the application server by running the instruction acquisition module 1902, and then obtains the screen video stream obtained by screen recording through the service application according to the screen sharing instruction by running the video stream acquisition module 1904, and finally runs the video stream sending module 1906 to send the screen video stream to the call application. Through the above-mentioned screen sharing processing, not only the screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved.

[0218] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0219] Furthermore, an embodiment of the present specification also provides a first screen sharing processing device, Fig. 20 A schematic diagram of a screen sharing processing device provided in one embodiment of the present specification is shown in FIG. Fig. 20 As shown, the device includes: a memory 2001, a processor 2002, a bus 2003 and a communication interface 2004. The memory 2001, the processor 2002 and the communication interface 2004 communicate via the bus 2003, and the communication interface 2004 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0220] Fig. 20 In the embodiment, the memory 2001 stores computer executable instructions that can be run on the processor 2002, and when the computer executable instructions are executed by the processor 2002, the following process is implemented: Establishing a call with the second terminal through the call application; Entering the service application according to the access instruction of the service application, and obtaining the screen sharing instruction submitted in the service application; In response to the screen sharing instruction, screen recording is performed through the service application to obtain a screen video stream for screen sharing.

[0221] The screen sharing processing device provided in this embodiment, through the cooperation of the memory 2001, the processor 2002, the bus 2003 and the communication interface 2004, the first terminal establishes a call with the second terminal based on the call application, and enters the service application to submit a screen sharing instruction through the screen sharing control in the service application, and responds to the screen sharing instruction. The screen is recorded through the service application to obtain a screen video stream for screen sharing, so that during the call between the first terminal and the second terminal, the first terminal can share the screen with the second terminal through the service application, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0222] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0223] Furthermore, an embodiment of the present specification also provides a second screen sharing processing device, Fig.21 A schematic diagram of a screen sharing processing device provided in one embodiment of the present specification is shown in FIG. Fig.21As shown, the device includes: a memory 2101, a processor 2102, a bus 2103 and a communication interface 2104. The memory 2101, the processor 2102 and the communication interface 2104 communicate through the bus 2103, and the communication interface 2104 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0224] Fig.21 In the embodiment, the memory 2101 stores computer executable instructions that can be run on the processor 2102, and when the computer executable instructions are executed by the processor 2102, the following process is implemented: Establishing a call with the first terminal through a call application; Acquire a screen video stream received by the call application and obtained by the first terminal through screen recording by the service application; The screen video stream is displayed on a call interface of the call application.

[0225] The screen sharing processing device provided in this embodiment, through the cooperation of the memory 2101, the processor 2102, the bus 2103 and the communication interface 2104, the second terminal first establishes a call with the first terminal through the call application, and then obtains the screen video stream obtained by the first terminal through the screen recording of the service application and displays the screen video stream in the call interface of the call application. In this way, not only the screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved.

[0226] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0227] Furthermore, an embodiment of this specification also provides a third screen sharing processing device, Fig. 22 A schematic diagram of a screen sharing processing device provided in one embodiment of the present specification is shown in FIG. Fig. 22 As shown, the device includes: a memory 2201, a processor 2202, a bus 2203 and a communication interface 2204. The memory 2201, the processor 2202 and the communication interface 2204 communicate through the bus 2203, and the communication interface 2204 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0228] Fig. 22 In the embodiment, the memory 2201 stores computer executable instructions that can be run on the processor 2202, and when the computer executable instructions are executed by the processor 2202, the following process is implemented: Acquire a screen sharing instruction of the first terminal forwarded by the application server; the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by recording the screen of the first terminal through the service application; The screen video stream is sent to a second terminal corresponding to the call.

[0229] The screen sharing processing device provided in this embodiment, through the cooperation of the memory 2201, the processor 2202, the bus 2203 and the communication interface 2204, the first terminal establishes a call with the second terminal based on the call application, and enters the service application to submit the screen sharing instruction through the screen sharing control in the service application. After the call server obtains the screen sharing instruction forwarded by the application server, it obtains the screen video stream obtained by recording the screen of the first terminal through the service application according to the screen sharing instruction and sends it to the second terminal corresponding to the call, so that during the call between the first terminal and the second terminal, the first terminal can share the screen with the second terminal through the service application, and the second terminal does not need to go through the service application, but can display the transcoded video stream of the screen sharing through the call interface, which not only realizes the screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0230] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0231] Furthermore, an embodiment of this specification also provides a fourth screen sharing processing device, Fig.23 A schematic diagram of a screen sharing processing device provided in one embodiment of the present specification is shown in FIG. Fig.23 As shown, the device includes: a memory 2301, a processor 2302, a bus 2303 and a communication interface 2304. The memory 2301, the processor 2302 and the communication interface 2304 communicate through the bus 2303, and the communication interface 2304 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0232] Fig.23 In the embodiment, the memory 2301 stores computer executable instructions that can be run on the processor 2302, and when the computer executable instructions are executed by the processor 2302, the following process is implemented: Entering the service application according to an access instruction of the service application; the access instruction is obtained after the call is established based on the call application; Get screen sharing instructions; In response to the screen sharing instruction, screen recording is performed to obtain a screen video stream for screen sharing.

[0233] The screen sharing processing device provided in this embodiment, through the cooperation of the memory 2301, the processor 2302, the bus 2303 and the communication interface 2304, the client of the service application enters the service application based on the access instruction of the service application obtained after the call is established based on the call application, obtains the screen sharing instruction, performs screen recording in response to the screen sharing instruction, and obtains the screen video stream for screen sharing, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0234] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0235] Furthermore, an embodiment of this specification also provides a fifth screen sharing processing device, Fig.24 A schematic diagram of a screen sharing processing device provided in one embodiment of the present specification is shown in FIG. Fig.24 As shown, the device includes: a memory 2401, a processor 2402, a bus 2403 and a communication interface 2404. The memory 2401, the processor 2402 and the communication interface 2404 communicate through the bus 2403, and the communication interface 2404 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0236] Fig.24 In the embodiment, the memory 2401 stores computer executable instructions that can be run on the processor 2402, and when the computer executable instructions are executed by the processor 2402, the following process is implemented: Obtaining a screen sharing instruction forwarded by the application server and submitted by the service application after establishing a call; the call is established based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by screen recording through the service application; The screen video stream is sent to the calling application.

[0237] The screen sharing processing device provided in this embodiment, through the cooperation of the memory 2401, the processor 2402, the bus 2403 and the communication interface 2404, after the server end of the call application obtains the screen sharing instruction submitted by the service application after the call is established and forwarded by the application server end, first obtains the screen video stream obtained by screen recording through the service application according to the screen sharing instruction, and then sends the screen video stream to the call application, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0238] The screen sharing processing device provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0239] Furthermore, another embodiment of the present specification also provides a first computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Establishing a call with the second terminal through the call application; Entering the service application according to the access instruction of the service application, and obtaining the screen sharing instruction submitted in the service application; In response to the screen sharing instruction, screen recording is performed through the service application to obtain a screen video stream for screen sharing.

[0240] The computer-readable storage medium provided in this embodiment is used for the first terminal to establish a call with the second terminal based on the call application, and to enter the service application to submit a screen sharing instruction through the screen sharing control in the service application. In response to the screen sharing instruction, the screen is recorded through the service application to obtain a screen video stream for screen sharing, so that during the call between the first terminal and the second terminal, the first terminal can share the screen with the second terminal through the service application, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0241] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0242] The computer-readable storage medium provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0243] Furthermore, another embodiment of the present specification also provides a second computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Establishing a call with the first terminal through a call application; Acquire a screen video stream received by the call application and obtained by the first terminal through screen recording by the service application; The screen video stream is displayed on a call interface of the call application.

[0244] The computer-readable storage medium provided in this embodiment is that the second terminal first establishes a call with the first terminal through the call application, and then obtains the screen video stream obtained by the first terminal through screen recording through the service application and displays the screen video stream in the call interface of the call application. In this way, not only screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved.

[0245] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0246] The computer-readable storage medium provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0247] Furthermore, another embodiment of the present specification also provides a third computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Acquire a screen sharing instruction of the first terminal forwarded by the application server; the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by recording the screen of the first terminal through the service application; The screen video stream is sent to a second terminal corresponding to the call.

[0248] The computer-readable storage medium provided in this embodiment is that the first terminal establishes a call with the second terminal based on the call application, and enters the service application to submit a screen sharing instruction through the screen sharing control in the service application. After the call server obtains the screen sharing instruction forwarded by the application server, it obtains the screen video stream obtained by recording the screen of the first terminal through the service application according to the screen sharing instruction and sends it to the second terminal corresponding to the call, so that during the call between the first terminal and the second terminal, the first terminal can share the screen with the second terminal through the service application, and the second terminal does not need to go through the service application, but can display the transcoded video stream of the screen sharing through the call interface, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0249] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0250] The computer-readable storage medium provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0251] Furthermore, another embodiment of the present specification provides a fourth computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Entering the service application according to an access instruction of the service application; the access instruction is obtained after the call is established based on the call application; Get screen sharing instructions; In response to the screen sharing instruction, screen recording is performed to obtain a screen video stream for screen sharing.

[0252] The computer-readable storage medium provided in this embodiment allows the client of the service application to enter the service application through the access instruction of the service application obtained after a call is established based on the call application, obtain the screen sharing instruction, perform screen recording in response to the screen sharing instruction, and obtain the screen video stream for screen sharing, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0253] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0254] The computer-readable storage medium provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0255] Furthermore, another embodiment of the present specification provides a fifth computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Obtaining a screen sharing instruction forwarded by the application server and submitted by the service application after establishing a call; the call is established based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by screen recording through the service application; The screen video stream is sent to the calling application.

[0256] The computer-readable storage medium provided in this embodiment is used for the server side of the call application to obtain the screen sharing instruction submitted by the service application after the call is established and forwarded by the application server side. The server side first obtains the screen video stream obtained by screen recording through the service application according to the screen sharing instruction, and then sends the screen video stream to the call application. This not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0257] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0258] The computer-readable storage medium provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0259] Furthermore, another embodiment of the present specification also provides a first computer program product, which implements the following process when executed by a processor: Establishing a call with the second terminal through the call application; Entering the service application according to the access instruction of the service application, and obtaining the screen sharing instruction submitted in the service application; In response to the screen sharing instruction, screen recording is performed through the service application to obtain a screen video stream for screen sharing.

[0260] In the computer program product provided in this embodiment, the first terminal establishes a call with the second terminal based on the call application, enters the service application, submits a screen sharing instruction through the screen sharing control in the service application, responds to the screen sharing instruction, performs screen recording through the service application, and obtains a screen video stream for screen sharing, so that during the call between the first terminal and the second terminal, the first terminal can share the screen with the second terminal through the service application, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0261] The computer program product provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0262] Furthermore, another embodiment of the present specification also provides a second computer program product, which implements the following process when executed by a processor: Establishing a call with the first terminal through a call application; Acquire a screen video stream received by the call application and obtained by the first terminal through screen recording by the service application; The screen video stream is displayed on a call interface of the call application.

[0263] In the computer program product provided in this embodiment, the second terminal first establishes a call with the first terminal through the call application, and then obtains the screen video stream obtained by the first terminal through screen recording through the service application and displays the screen video stream in the call interface of the call application. In this way, not only screen sharing during the call is realized, but also the convenience of screen sharing during the call is improved.

[0264] The computer program product provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0265] Furthermore, another embodiment of the present specification also provides a third computer program product, which implements the following process when executed by a processor: Acquire a screen sharing instruction of the first terminal forwarded by the application server; the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by recording the screen of the first terminal through the service application; The screen video stream is sent to a second terminal corresponding to the call.

[0266] In the computer program product provided by this embodiment, the first terminal establishes a call with the second terminal based on the call application, and enters the service application to submit a screen sharing instruction through the screen sharing control in the service application. After the call server obtains the screen sharing instruction forwarded by the application server, it obtains the screen video stream obtained by recording the screen of the first terminal through the service application according to the screen sharing instruction and sends it to the second terminal corresponding to the call, so that during the call between the first terminal and the second terminal, the first terminal can share the screen with the second terminal through the service application, and the second terminal does not need to go through the service application, but can display the transcoded video stream of the screen sharing through the call interface, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0267] The computer program product provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0268] Furthermore, another embodiment of this specification provides a fourth computer program product, which implements the following process when executed by a processor: Entering the service application according to an access instruction of the service application; the access instruction is obtained after the call is established based on the call application; Get screen sharing instructions; In response to the screen sharing instruction, screen recording is performed to obtain a screen video stream for screen sharing.

[0269] In the computer program product provided in this embodiment, after a call is established based on the call application, the client of the service application obtains the access instruction of the service application, enters the service application, obtains the screen sharing instruction, performs screen recording in response to the screen sharing instruction, and obtains the screen video stream for screen sharing, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0270] The computer program product provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0271] Furthermore, another embodiment of the present specification also provides a fifth computer program product, which implements the following process when the computer program product is executed by a processor: Obtaining a screen sharing instruction forwarded by the application server and submitted by the service application after establishing a call; the call is established based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by screen recording through the service application; The screen video stream is sent to the calling application.

[0272] In the computer program product provided by this embodiment, after the server of the call application obtains the screen sharing instruction submitted by the service application after the call is established and forwarded by the application server, it first obtains the screen video stream obtained by screen recording through the service application according to the screen sharing instruction, and then sends the screen video stream to the call application, which not only realizes screen sharing during the call, but also improves the convenience of screen sharing during the call.

[0273] The computer program product provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0274] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0275] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0276] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0277] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0278] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0279] Memory may include non-permanent storage in a computer-readable storage medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable storage medium.

[0280] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0281] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0282] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0283] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A screen sharing processing method, characterized in that: Applied to a first terminal, the method includes: Establishing a call with the second terminal through the call application; Entering the service application according to the access instruction of the service application, and obtaining the screen sharing instruction submitted in the service application; In response to the screen sharing instruction, screen recording is performed through the service application to obtain a screen video stream for screen sharing.

2. The screen sharing processing method according to claim 1, characterized in that: The obtaining of the screen sharing instruction submitted in the service application comprises: Obtaining a service access instruction for a screen sharing service submitted in the service application; In response to the service access instruction, displaying the call record data of the call through the program interface of the service application program; Obtain a screen sharing instruction submitted through the call record data in the service application.

3. The screen sharing processing method according to claim 1, characterized in that: After entering the service application program according to the access instruction of the service application program, the method further comprises: Displaying the program interface of the service application and the call icon of the call; If it is detected that the call icon is triggered, the program interface jumps to the call interface of the call application program.

4. The screen sharing processing method according to claim 1, characterized in that: During the screen recording by the service application, displaying a screen recording mark based on the screen interface; If it is detected that the screen recording mark is triggered, the screen recording is terminated and a screen sharing end reminder is displayed.

5. The screen sharing processing method according to claim 1, characterized in that: After the step of establishing a call with the second terminal through the call application is performed, the method further includes: In response to the establishment of the call, recording audio through the call application to obtain a first audio stream; and / or, The second audio stream of the second terminal is received through the call application, and the broadcast component is called to perform broadcast processing on the second audio stream.

6. The screen sharing processing method according to claim 1, characterized in that: The obtaining of the screen sharing instruction submitted in the service application comprises: Obtaining a service access instruction for a screen sharing service submitted in the service application; In response to the service access instruction, presenting a screen sharing control through a program interface of the service application; If it is detected that the screen sharing control is triggered, the screen sharing instruction submitted in the service application is obtained.

7. The screen sharing processing method according to claim 6, characterized in that: After the step of obtaining the screen sharing instruction submitted in the service application is executed, the method further includes: Jump from the program interface displaying the screen sharing control to the program interface displaying the recording end control within the service application.

8. The screen sharing processing method according to claim 7, characterized in that: During the process of performing screen recording through the service application, if it is detected that the recording end control is triggered, the recording end instruction submitted in the service application is obtained.

9. The screen sharing processing method according to claim 1, characterized in that: The method further comprises: Acquiring, through the call application, a shared screen video stream obtained by the second terminal through screen recording through the service application; The shared screen video stream is displayed through the call interface of the call application.

10. A screen sharing processing method, characterized in that: Applied to the second terminal, the method includes: Establishing a call with the first terminal through a call application; Acquire a screen video stream received by the call application and obtained by the first terminal through screen recording by the service application; The screen video stream is displayed on a call interface of the call application.

11. The screen sharing processing method according to claim 10, characterized in that: The method further comprises: Obtaining a screen recording end instruction for the first terminal to end screen recording; Based on the screen recording end instruction, the call interface of the call is displayed through the call application.

12. The screen sharing processing method according to claim 10, characterized in that: The method further comprises: In response to the establishment of the call, recording the audio through the call application to obtain a second audio stream; and / or, The first audio stream of the first terminal is received through the call application, and the broadcast component is called to perform broadcast processing on the first audio stream.

13. The screen sharing processing method according to claim 10, characterized in that: The method further comprises: Enter the service application according to the access instruction of the service application, and obtain the screen sharing instruction submitted in the service application; In response to the screen sharing instruction, screen recording is performed through the service application to obtain a shared screen video stream for screen sharing.

14. A screen sharing processing method, characterized in that: Applied to a call server, the method comprises: Acquire a screen sharing instruction of the first terminal forwarded by the application server; the screen sharing instruction enters the service application and is submitted after the first terminal establishes a call based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by recording the screen of the first terminal through the service application; The screen video stream is sent to a second terminal corresponding to the call.

15. The screen sharing processing method according to claim 14, characterized in that: After the step of acquiring a screen video stream obtained by recording the screen of the first terminal by the service application according to the screen sharing instruction is executed, and before the step of sending the screen video stream to the second terminal corresponding to the call is executed, the method further includes: Transcoding the screen video stream according to the call network format to obtain a transcoded video stream; Correspondingly, sending the screen video stream to the second terminal corresponding to the call includes: The transcoded video stream is sent to the second terminal.

16. The screen sharing processing method according to claim 15, characterized in that: The step of transcoding the screen video stream according to the call network format to obtain the transcoded video stream includes: Inserting key frames into the screen video stream according to a preset video frame interval, and deleting the bidirectional prediction frames in the screen video stream to obtain an intermediate video stream; The intermediate video stream is format-converted according to the call network format to obtain the transcoded video stream.

17. The screen sharing processing method according to claim 14, characterized in that: The screen sharing instruction is forwarded by the application server to the call server through the signaling server; after receiving the screen sharing instruction sent by the application server, the signaling server forwards the screen sharing instruction to the call server and returns a screen sharing response to the application server; The application server receives the screen sharing response and returns it to the service application, so that the service application performs screen recording on the first terminal.

18. The screen sharing processing method according to claim 17, characterized in that: Before the signaling server forwards the screen sharing instruction to the call server, the signaling server further performs the following operations: Performing unilateral video negotiation on the first terminal and the second terminal; If the negotiation is successful, the screen sharing instruction is forwarded to the call server.

19. The screen sharing processing method according to claim 18, characterized in that: The performing unilateral video negotiation on the first terminal and the second terminal includes: Sending a request message with an empty session description protocol to the first terminal, and receiving a response message carrying a session description proposal returned by the first terminal; Performing encoding filtering on the session description proposal based on the access encoding format of the call server, and performing address modification on the session description proposal based on the address information of the call server to obtain a modified session description proposal; Sending a response request message carrying the proposal for modifying the session description to the second terminal, and receiving a session description response returned by the second terminal; A response detection is performed on the session description response, and a detection result is used as a negotiation result of the unilateral video negotiation.

20. A screen sharing processing method, characterized in that: Applied to a client of a service application, the method comprises: Entering the service application according to an access instruction of the service application; the access instruction is obtained after the call is established based on the call application; Get screen sharing instructions; In response to the screen sharing instruction, screen recording is performed to obtain a screen video stream for screen sharing.

21. The screen sharing processing method according to claim 20, characterized in that: The obtaining of the screen sharing instruction comprises: Get the service access instruction of the screen sharing service; Perform a call record data query based on the service access instruction, obtain the call record data of the call and display it; Obtain a screen sharing instruction submitted based on the call record data.

22. The screen sharing processing method according to claim 20, characterized in that: The step of performing screen recording in response to the screen sharing instruction and obtaining a screen video stream for screen sharing includes: Perform screen recording according to the screen sharing response returned by the application server to obtain the screen video stream; Correspondingly, after the step of performing screen recording according to the screen sharing response returned by the application server and obtaining the screen video stream is executed, the step further includes: The screen video stream is sent to a video storage area for storage.

23. The screen sharing processing method according to claim 20, characterized in that: The obtaining of the screen sharing instruction comprises: Get the service access instruction of the screen sharing service; Based on the service access instruction, display a first program interface including a screen sharing control; If it is detected that the screen sharing control is triggered, it is determined that the screen sharing instruction is acquired.

24. The screen sharing processing method according to claim 23, characterized in that: After the step of obtaining the screen sharing instruction is executed, the method further includes: Jump from displaying the first program interface to displaying the second program interface including the recording end control.

25. The screen sharing processing method according to claim 24, characterized in that: During the screen recording process, if it is detected that the recording end control is triggered, a recording end instruction is obtained.

26. A screen sharing processing method, characterized in that: The method applied to a call service end of a call application comprises: Obtaining a screen sharing instruction forwarded by the application server and submitted by the service application after establishing a call; the call is established based on the call application; According to the screen sharing instruction, obtaining a screen video stream obtained by screen recording through the service application; The screen video stream is sent to the calling application.

27. The screen sharing processing method according to claim 26, characterized in that: The service application is run on the first terminal, and the first terminal establishes a call with the second terminal through the call application; The call service end sends the screen video stream to a call application running on the second terminal.

Images