Conference screen sharing method and system, program product and medium
By synthesizing a composite code stream on the shared end, the first mainstream and shared auxiliary stream on the first receiving end are synthesized, and the problem of inconsistency between the portrait displayed on the receiving end in an online meeting is solved, which improves the conference experience and improves the transmission efficiency.
Patent Information
- Application Number
- CN202311628775.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-25
- Filing Date
- 2023-11-30
- Publication Date
- 2025-04-25
AI Technical Summary
In online meetings, the sharing end can only process the mainstream of local portraits, which causes the portrait displayed on the receiving end to be inconsistent with the actual speaking end, affecting the meeting experience.
By obtaining the first mainstream of the first receiving end on the shared end and synthesizing it with the local shared auxiliary stream, a composite code stream is obtained to ensure that the displayed second image remains consistent with the image of the speech end.
It realizes dynamic recognition of the speech end image in the conference screen sharing, improves the conference effect and experience, and reduces the uplink transmission bandwidth of the conference and improves transmission efficiency.
Smart Images
Figure CN119892804A_ABST
Abstract
Description
[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on October 25, 2023, with application number 202311393711.5 and invention name “A portrait display method and device”, the entire contents of which are incorporated by reference in this application. Technical Field
[0002] The embodiments of the present application relate to the field of conferences, and in particular to a conference screen sharing method, system, program product, and medium. Background Art
[0003] Online meetings have become a form of communication in the work process. During a meeting, there can be multiple participants, including sharing terminals and receiving terminals. The user device that is currently speaking in the meeting is the speaking terminal. The sharing terminal often needs to demonstrate the content to be explained by sharing the desktop, such as images, videos, or operation demonstrations, and the receiving terminal can receive the voice and shared content in the meeting. However, in the actual communication process, the expression, status, or body movements of the speaking terminal are also important components of the communication. If the video information of the speaking terminal is conveyed at the same time during the sharing of materials, it will increase the receiving terminal's understanding of the communication content and greatly improve the meeting experience.
[0004] The sharing end can cut out the portrait in the local portrait mainstream to obtain the target portrait mainstream after cutting out. The sharing end sends two streams to the server, including the target portrait mainstream and the shared auxiliary stream. The target portrait mainstream includes the portrait floating window image, and the shared auxiliary stream includes the screen sharing image of the sharing end. The server combines the two streams and then forwards the combined composite stream to the receiving end, which receives the stream and decodes it for display.
[0005] Since the sharing end can only process the local portrait mainstream, and the speaking end and the sharing end are not necessarily the same participant, the portrait displayed on the receiving end may be inconsistent with the actual speaking end. Summary of the invention
[0006] The embodiments of the present application provide a conference screen sharing method, system, program product and medium, which are applied to the field of screen sharing, used to implement conference screen sharing, and can dynamically identify the first image of the speaking end.
[0007] In a first aspect, an embodiment of the present application provides a conference screen sharing method, which is applied to a conference screen sharing system. The conference screen sharing system includes multiple user devices, in which the user device initiating screen sharing is called a sharing end, and the user device receiving screen sharing is called a receiving end. The system may have one or more receiving ends, and the conference screen sharing system includes at least a first receiving end. In the system, a user device in a speaking state is a speaking end.
[0008] The sharing end shares the screen image through the shared auxiliary stream, and the shared auxiliary stream includes the screen sharing image. The first receiving end is the speaking end, and the mainstream including the first image of the first receiving end is the first mainstream. The sharing end obtains the first mainstream of the first receiving end, and synthesizes the first mainstream with the local shared auxiliary stream to obtain a composite code stream. The composite code stream includes a second image, which is synthesized by the first image and the screen sharing image. The sharing end displays the second image on the local screen.
[0009] In the embodiment of the present application, the sharing end obtains the first mainstream of the first receiving end, and synthesizes the image with the shared auxiliary stream of the local end to obtain a composite code stream. Therefore, even if the speaking end is not the sharing end, the second image displayed in the composite code stream can still be consistent with the image of the speaking end.
[0010] In some optional implementations, the first image is a first portrait of the first receiving end, and the first portrait may be a camera video stream of the first receiving end, or may be a virtual portrait generated by the first receiving end.
[0011] In the embodiment of the present application, when the first image is a portrait, the second image displayed in the composite code stream can reflect the actions, expressions and other information of the speaking user, thereby improving the conference effect.
[0012] In some optional implementations, when the sharing end and the first receiving end are in the same local area network, the first mainstream from the first receiving end is directly received.
[0013] In some optional implementations, when the sharing end and the first receiving end are in the same local area network, the sharing end may first receive the second mainstream from the first receiving end, and then perform cutout processing on the second mainstream to obtain the first mainstream.
[0014] In some optional implementations, after the sharing end synthesizes the composite code stream, and when the sharing end and the first receiving end are in the same local area network, the sharing end may directly send the composite code stream to the receiving end.
[0015] In the embodiment of the present application, since the sharing end combines the shared auxiliary stream and the first main stream into one stream, the sharing end only needs to send one code stream to the receiving end, which reduces the conference uplink transmission bandwidth and improves the transmission efficiency.
[0016] In some optional implementations, the conference screen sharing system also includes a server, which is used to detect the speaking end and perform media processing such as forwarding, transcoding or synthesis of the code stream. The server receives video mainstreams from multiple receiving ends, including the second mainstream of the first receiving end. The server sends all video mainstreams, that is, the third mainstream, to the sharing end. The server also sends an indication message to the sharing end, indicating the second mainstream among all video mainstreams. The sharing end determines the second mainstream from the third mainstream according to the indication message.
[0017] In some optional implementations, after receiving the second mainstream from the first receiving end, the server end may first perform a cutout process on the second mainstream to obtain the first mainstream, and the server end may directly send the cutout first mainstream to the sharing end.
[0018] In some optional implementations, if the first mainstream does not need to be cut out, after receiving the first mainstream from the receiving end, the server sends all video mainstreams, i.e., the fourth mainstream, to the sharing end, and then instructs the sharing end of the first mainstream through an indication message. The sharing end determines the first mainstream from the fourth mainstream according to the indication message.
[0019] In some optional implementations, the sharing end may send the synthesized composite code stream to the server end, and then the server end forwards the composite code stream to the receiving end.
[0020] In some optional implementations, the conference screen sharing system further includes a second receiving end, and the server receives the second mainstream from the first receiving end, and sends all video mainstreams, i.e., the third mainstream, to the second receiving end, including the second mainstream of the first receiving end. The server uses an instruction message to enable the second receiving end to determine the second mainstream of the first receiving end from the third mainstream, and perform a cutout process on the second mainstream of the first receiving end to obtain the first mainstream. The second receiving end synthesizes the first mainstream and the shared auxiliary stream to obtain a second image and displays it on the screen of the second receiving end.
[0021] In some optional implementations, the second receiving end may also receive the cutout first mainstream from the server, and the server cuts out the second mainstream of the first receiving end and sends it to the second receiving end. The second receiving end synthesizes the first mainstream and the shared auxiliary stream to obtain a second image and displays it on the screen of the second receiving end.
[0022] In some optional implementations, the server can obtain the second mainstream of the first receiving end, and perform cutout processing on the second mainstream to obtain the first mainstream. The server synthesizes the first mainstream and the shared auxiliary stream to obtain a composite stream, and sends the composite stream to the first receiving end.
[0023] In some optional embodiments, the second image is synthesized based on the principle of minimizing occlusion, and the first image is composited to a first area in the screen sharing image, where the first area includes a blank area or an area with the simplest texture in the screen sharing image, and the first area can be obtained by an algorithm.
[0024] In some optional implementations, the first image is composited into a second area set by the sharing terminal in the screen sharing image. In the second image synthesized based on the method, the position of the first image can be manually set by a user of the sharing terminal.
[0025] A second aspect of an embodiment of the present application provides a conference screen sharing system, including:
[0026] A sharing end, used for acquiring a first mainstream of a first receiving end, where the first receiving end is the speaking end, and the first mainstream includes a first image;
[0027] The sharing end is further used to synthesize and encode the shared auxiliary stream and the first main stream to obtain a composite code stream, wherein the composite code stream includes a second image, which is synthesized by a screen sharing image and the first image, and the shared auxiliary stream includes the screen sharing image;
[0028] The sharing end is further used to display the second image.
[0029] The steps performed by the conference screen sharing system described in the second aspect may refer to the conference screen sharing method described in the first aspect, and will not be repeated here.
[0030] A third aspect of an embodiment of the present application provides a computing device cluster, characterized in that it includes at least one computing device, each computing device includes a processor and a memory;
[0031] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method as described in the first aspect above.
[0032] A fourth aspect of an embodiment of the present application provides a computer program product comprising instructions, wherein when the instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect above.
[0033] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, characterized in that it includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1A system architecture diagram in an embodiment of the present application;
[0035] Figure 2 Another system architecture diagram in an embodiment of the present application;
[0036] Figure 3 This is a schematic diagram of an embodiment of synthesizing a composite code stream at a sharing end in a conference screen sharing method in an embodiment of the present application;
[0037] Figure 4 This is a schematic diagram of an embodiment of a conference screen sharing method in an embodiment of the present application;
[0038] Figure 5 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0039] Figure 6 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0040] Figure 7 This is a schematic diagram of an embodiment of synthesizing a composite code stream at the server end in the conference screen sharing method in the embodiment of the present application;
[0041] Figure 8 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0042] Fig. 9 This is a schematic diagram of an embodiment of synthesizing a composite code stream at a receiving end in a conference screen sharing method in an embodiment of the present application;
[0043] Fig.10 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0044] Fig.11 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0045] Fig.12 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0046] Fig.13 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0047] Fig.14 This is a schematic diagram of another embodiment of synthesizing a composite code stream at a sharing end in a conference screen sharing method in an embodiment of the present application;
[0048] Fig.15 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0049] Fig.16This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0050] Fig.17 This is a schematic diagram of another embodiment of synthesizing a composite code stream at a receiving end in a conference screen sharing method in an embodiment of the present application;
[0051] Fig.18 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0052] Fig.19 This is a schematic diagram of another embodiment of the conference screen sharing method in the embodiment of the present application;
[0053] Fig. 20 This is a schematic diagram of an embodiment of a conference screen sharing system in an embodiment of the present application;
[0054] Fig.21 This is a schematic diagram of the structure of a computing device in an embodiment of the present application;
[0055] Fig. 22 A schematic diagram of the structure of a computing device cluster in an embodiment of the present application;
[0056] Fig.23 Another structural diagram of a computing device cluster in an embodiment of the present application. DETAILED DESCRIPTION
[0057] The embodiments of the present application disclose a conference screen sharing method, system, program product and medium, which are applied to the field of screen sharing, used to realize conference screen sharing, and can dynamically identify the first image of the speaking end.
[0058] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0059] The terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0060] The following is a brief description of the system architecture on which the conference screen sharing method in the embodiment of the present application is based.
[0061] The system architecture is a conference screen sharing system, such as Figure 1 As shown, terminal 101 initiates screen sharing, and terminals 103 and 104 receive screen sharing. Terminal 101 is called a sharing terminal, terminals 103 and 104 are called receiving terminals, and all code streams are forwarded through server 102. Server 102 can be connected to multiple receiving terminals, and the terminal that speaks in the conference is called a speaking terminal.
[0062] Figure 2 Another system architecture of the conference screen sharing system is shown in FIG. Figure 2 As described above, terminal 201 and terminal 202 are in the same local area network, terminal 201 initiates screen sharing, and terminal 202 receives screen sharing. Terminal 201 is called a sharing terminal, terminal 202 is called a receiving terminal, and the terminal that speaks in the conference is called a speaking terminal. Since the sharing terminal and the receiving terminal are in the same local area network, there is no need for the server to forward the code stream.
[0063] Based on the above architecture, in the embodiment of the present application, the sharing end obtains the first mainstream of the speaking end, synthesizes it with the local shared auxiliary stream to obtain a composite stream, or the composite stream can be synthesized by the server or the receiving end. The first mainstream can be a video mainstream including a portrait floating window image, or a video stream including the head portrait of the speaking end user, which is not limited here. In the embodiment of the present application, the first mainstream is described as a video mainstream including a portrait floating window image.
[0064] Based on the differences in the speaking end and the device that synthesizes the composite stream, there are many ways to share the conference screen. The following are the following:
[0065] 1. The shared end synthesizes the composite stream and forwards it to the receiving end through the server.
[0066] See also Figure 3 In this embodiment, the sharing end obtains the first mainstream of the speaking end, and synthesizes the first mainstream of the speaking end and the local shared auxiliary stream to obtain a composite code stream. The sharing end sends the composite code stream to the server end, and the server end forwards the composite code stream to all receiving ends.
[0067] In this embodiment, based on the different speaking terminals, the sharing terminal may obtain the first mainstream in various ways:
[0068] 1. The speaking end is the shared end;
[0069] See also Figure 4 In an embodiment of the present application, a conference screen sharing method includes:
[0070] 401. The sharing terminal obtains the first mainstream;
[0071] Since the speaking end is a sharing end, the sharing end can obtain the main video stream from the local end. The sharing end obtains the second main stream, which is the local portrait main stream of the sharing end, that is, the camera video stream of the sharing end. The sharing end processes the second main stream with the portrait cutout algorithm, calculates the area in the second main stream that only contains the portrait foreground, and then makes the portrait background transparent to obtain the portrait floating window image.
[0072] 402. The sharing end combines the first main stream and the shared auxiliary stream;
[0073] The first mainstream includes a portrait floating window image, and the shared auxiliary stream includes a screen sharing image. The sharing end composites the portrait floating window image in each frame into the corresponding screen sharing image. The sharing end can use a texture algorithm on the screen sharing image based on the principle of minimizing occlusion to calculate the blank area or the area with the simplest texture in the screen sharing image. For example, the sharing end divides the screen sharing image into multiple areas, and uses a texture algorithm to score the texture of each area from 0 to 10, 0 points for the area with the simplest texture, and 10 points for the area with the most complex texture. The sharing end selects the area with the lowest score for composite. If there are multiple areas with the same score, a region is randomly selected for composite. The sharing end composites the portrait floating window image into the area to form an image, namely the second image, and the portrait floating window image is located on the upper layer of the screen sharing image.
[0074] The sharing end may also set an area in the screen sharing image for synthesis, and the area is set by the sharing end receiving instructions from the user. The sharing end displays the portrait floating window image at the position of the area, and synthesizes the screen sharing image and the portrait floating window image by means of desktop screenshot to obtain a second image. Specifically, the sharing end places the portrait floating window image on the upper layer of the screen sharing image, takes a screenshot of each frame of the screen, and obtains the second image. Since the portrait floating window image and the screen sharing image are not synthesized into one image, this method supports the sharing end user to manually adjust the size or position of the portrait floating window image. In actual applications, the sharing end may also have other ways of combining images, which are not limited here.
[0075] 403. The sharing end sends a composite code stream to the server end;
[0076] The sharing end encodes the synthesized image to obtain a composite code stream. The sharing end sends the composite code stream to the server through the line for sending the shared auxiliary stream. At this time, the sharing end only sends one code stream to the server.
[0077] In the embodiment of the present application, since the sharing end combines the first mainstream stream and the shared auxiliary stream of the speaking end into a composite stream, only one stream needs to be sent to the server end, which reduces the transmission bandwidth of the uplink in the conference system and improves the transmission efficiency.
[0078] 404. The server sends a composite code stream to the receiver;
[0079] The server receives the composite code stream from the sharing end and forwards the composite code stream to the receiving end.
[0080] 405. The receiving end decodes and displays the composite code stream;
[0081] After receiving the composite code stream from the server, the receiver decodes the composite code stream to obtain a composite image. The receiver displays the composite image to achieve screen sharing.
[0082] 2. The speaking end is the receiving end;
[0083] See also Figure 5 In an embodiment of the present application, a conference screen sharing method includes:
[0084] 501. The receiving end sends the second main stream to the server end;
[0085] When the speaking end is the receiving end, the receiving end sends the second mainstream to the server end. At this time, the second mainstream is the local portrait mainstream of the receiving end.
[0086] 502. The server sends the third main stream to the sharing terminal;
[0087] When all terminals turn on their cameras, they need to send the local portrait mainstream to the server. The server sends the third mainstream to the sharing terminal, which includes all portrait mainstreams.
[0088] 503. The server sends an indication message to the sharing terminal;
[0089] The server detects the speaker and generates an indication message according to the speaker, and sends the indication message to the sharing terminal, so that the sharing terminal determines the portrait mainstream of the speaker from all portrait mainstreams, that is, the second mainstream.
[0090] 504. The sharing end performs cutout processing on the second main stream;
[0091] After the sharing end determines the second mainstream from the third mainstream, it performs portrait cutout algorithm processing on the second mainstream, calculates the area in the second mainstream that only contains the portrait foreground, and then makes the portrait background transparent to obtain a portrait floating window image.
[0092] 505. The sharing end combines the first main stream and the shared auxiliary stream;
[0093] 506. The sharing end sends a composite code stream to the server end;
[0094] 507. The server sends a composite code stream to the receiver;
[0095] 508. The receiving end decodes and displays the composite code stream;
[0096] Steps 505 to 508 in this embodiment are similar to those in the previous embodiment. Figure 4 Steps 402 to 405 in the illustrated embodiment are similar and will not be described in detail herein.
[0097] In the embodiment of the present application, when the receiving end is the speaking end, the sharing end can not only receive the second mainstream forwarded by the server end, but also receive the first mainstream obtained by the server end by performing cutout processing on the second mainstream of the receiving end.
[0098] See also Figure 6 In an embodiment of the present application, a conference screen sharing method includes:
[0099] 601. The receiving end sends the second main stream to the server end;
[0100] Step 601 in this embodiment is similar to the aforementioned Figure 5 Step 501 in the illustrated embodiment is similar and will not be described in detail here.
[0101] 602. The server performs cutout processing on the second main stream;
[0102] The server detects the speaker and determines the second mainstream from all the portrait mainstreams. The server processes the second mainstream with a portrait cutout algorithm, calculates the area in the second mainstream that only contains the portrait foreground, and then makes the portrait background transparent to obtain a portrait floating window image.
[0103] 603. The server sends the first mainstream to the sharing terminal;
[0104] The server sends the main stream of the portrait after cutting out to the sharing end, which synthesizes it into a composite code stream.
[0105] 604. The sharing end combines the first main stream and the shared auxiliary stream;
[0106] 605. The sharing end sends a composite code stream to the server end;
[0107] 606. The server sends a composite code stream to the receiver.
[0108] 607. The receiving end decodes and displays the composite code stream;
[0109] Steps 604 to 607 in this embodiment are similar to those in the previous Figure 5 Steps 505 to 508 in the illustrated embodiment are similar and will not be described in detail herein.
[0110] In the embodiment of the present application, the sharing end can dynamically identify the floating window image of the person at the speaking end by acquiring the first mainstream of the speaking end, and when the receiving end displays it, the portrait image can change according to the speaking end. At the same time, because the first mainstream and the shared auxiliary stream are synthesized into a composite stream, the receiving end can receive not only the sound and auxiliary stream content of the speaking end, but also the video mainstream of the speaking end, including actions, expressions and other information, thereby improving the conference effect.
[0111] 2. The server synthesizes the composite code stream and sends it to the receiver;
[0112] See also Figure 7 In this embodiment, the sharing end sends a shared auxiliary stream to the server end, and the server end obtains the first main stream and synthesizes the composite stream. The server end then sends the composite stream to all receiving ends. The server end also sends the first main stream to the sharing end, and the sharing end synthesizes the second image and displays it.
[0113] See also Figure 8 In an embodiment of the present application, a conference screen sharing method includes:
[0114] 801. The sharing end sends the second main stream to the server end;
[0115] The sharing end sends the second mainstream to the server end. At this time, the second mainstream is the local portrait mainstream of the sharing end.
[0116] 802. The receiving end sends the second main stream to the server end;
[0117] Step 802 in this embodiment is similar to the above Figure 5 Step 501 in the illustrated embodiment is similar and will not be described in detail here.
[0118] In this embodiment, when the sharing end is the speaking end, step 801 is executed; when the receiving end is the speaking end, step 802 is executed.
[0119] 803. The sharing end sends the shared auxiliary stream to the server end;
[0120] The sharing end sends the local shared auxiliary stream to the server end, and the server end decodes the shared auxiliary stream to obtain the screen sharing image.
[0121] 804. The server performs cutout processing on the second main stream;
[0122] Step 804 in this embodiment is similar to the above Figure 6 Step 602 in the illustrated embodiment is similar and will not be described in detail here.
[0123] 805. The server synthesizes the first main stream and the shared auxiliary stream;
[0124] The server uses a texture algorithm to calculate the screen sharing image in the shared auxiliary stream, determines the blank area or the area with the simplest texture in the screen sharing image, and composites the portrait floating window image into the area to form a single image including the screen sharing image and the portrait floating window image, i.e., the second image.
[0125] 806. The server sends a composite code stream to the receiver.
[0126] 807. The receiving end decodes and displays the composite code stream;
[0127] Steps 806 to 807 in this embodiment are similar to those in the previous embodiment. Figure 5 Steps 507 to 508 in the illustrated embodiment are similar and will not be described in detail herein.
[0128] 808. The server sends the first mainstream to the sharing terminal;
[0129] The server sends the main image of the cutout portrait to the sharing terminal, so that the sharing terminal can synthesize the second image and display it.
[0130] 809. The sharing end combines the first main stream and the shared auxiliary stream;
[0131] Step 809 in this embodiment is similar to the above Figure 4 Step 402 in the illustrated embodiment is similar and will not be described in detail here.
[0132] 3. The receiving end obtains the first mainstream, locally synthesizes the second image and displays it;
[0133] See also Fig. 9 In this embodiment, the sharing end sends a shared auxiliary stream to the server end, and the server end forwards the shared auxiliary stream to the receiving end. The receiving end obtains the first mainstream and synthesizes the first mainstream with the shared auxiliary stream from the server end. Since the receiving end does not need to encode the synthesized image, the second image can be directly obtained. When the speaking end is the receiving end, the receiving end also sends the first mainstream to the sharing end through the server end.
[0134] Based on the different speaking ends, there are multiple ways for the receiving end to obtain the first mainstream:
[0135] 1. The speaking end is the shared end;
[0136] See also Fig.10 In an embodiment of the present application, a conference screen sharing method includes:
[0137] 1001. The sharing end sends the second main stream to the server end;
[0138] Step 1001 in this embodiment is similar to the above Figure 8 Step 801 in the illustrated embodiment is similar and will not be described in detail here.
[0139] 1002. The server sends the third main stream to the receiving end;
[0140] When all terminals turn on their cameras, they need to send the local portrait mainstream to the server. The server sends the third mainstream to the receiving end, which includes all portrait mainstreams.
[0141] 1003. The server sends an indication message to the receiver;
[0142] The server detects the speaker and generates an indication message according to the speaker. The server sends the indication message to the sharing terminal, so that the sharing terminal determines the portrait mainstream of the speaker from all portrait mainstreams, that is, the second mainstream.
[0143] 1004. The receiving end performs a cutout process on the second bit stream;
[0144] The receiving end processes the second main stream with a portrait cutout algorithm, calculates the area in the second main stream that only contains the portrait foreground, and then makes the portrait background transparent to obtain a portrait floating window image.
[0145] 1005. The sharing end sends the shared auxiliary stream to the server end;
[0146] The server receives the shared auxiliary stream from the sharing end and forwards it to the receiving end;
[0147] 1006. The server sends the shared auxiliary stream to the receiving end;
[0148] The server forwards the shared auxiliary stream from the sharing end to the receiving end;
[0149] This embodiment does not limit the time sequence between step 1004 and step 1006. In practical applications, step 1006 may be performed first, and then step 1004, which is not specifically limited here.
[0150] 1007. The receiving end combines the first main stream and the shared auxiliary stream;
[0151] The receiving end composites the portrait floating window image in each frame into the corresponding screen sharing image, and uses the texture algorithm on the screen sharing image based on the principle of minimizing occlusion to calculate the blank area or the area with the simplest texture in the screen sharing image. The sharing end composites the portrait floating window image into the area to form an image, namely the second image, and the portrait floating window image is located on the upper layer of the screen sharing image.
[0152] The sharing end can also set an area in the screen sharing image for synthesis, and the area is set by the receiving end receiving the user's instructions. The receiving end displays the portrait floating window image in the area, and uses the desktop screenshot method to synthesize the screen sharing image and the portrait floating window image to obtain a second image. This method supports the receiving end user to manually adjust the size or position of the portrait floating window image. In actual applications, the sharing end can also have other ways of combining images, which are not limited here.
[0153] 1008. The receiving end sends the first main stream to the server end;
[0154] The receiving end sends the main stream of the portrait after the cutout, that is, the first main stream, to the server end, which is forwarded by the server end to the sharing end.
[0155] 1009. The server sends the first mainstream to the sharing terminal;
[0156] 1010. The sharing end combines the first main stream and the shared auxiliary stream;
[0157] Steps 1009 to 1010 in this embodiment are similar to those in the previous embodiment. Figure 8 Steps 808 to 809 in the illustrated embodiment are similar and will not be described in detail herein.
[0158] In the embodiment of the present application, when the sharing end is the speaking end, the receiving end may also receive a first mainstream from the serving end, the first mainstream being obtained by the serving end performing cutout processing on the second mainstream after receiving the second mainstream from the sharing end.
[0159] See also Fig.11 In an embodiment of the present application, a conference screen sharing method includes:
[0160] 1101. The sharing end sends the second main stream to the server end;
[0161] Step 1001 in this embodiment is similar to the above Figure 8 Step 801 in the illustrated embodiment is similar and will not be described in detail here.
[0162] 1102. The server performs cutout processing on the second main stream;
[0163] Step 1102 in this embodiment is similar to the aforementioned Figure 6 Step 602 in the illustrated embodiment is similar and will not be described in detail here.
[0164] 1103. The server sends a first mainstream to the receiving end;
[0165] The server sends the first mainstream after the cutout processing to the receiving end, so that the receiving end can directly use the first mainstream for image composite without further cutout processing.
[0166] 1104. The sharing end sends the shared auxiliary stream to the server end;
[0167] 1105. The server sends the shared auxiliary stream to the receiving end;
[0168] 1106. The receiving end combines the first main stream and the shared auxiliary stream;
[0169] Steps 1104 to 1106 in this embodiment are similar to those in the previous embodiment. Fig.10 Steps 1005 to 1007 in the illustrated embodiment are similar and will not be described in detail herein.
[0170] 1107. The server sends the first mainstream to the sharing terminal;
[0171] 1108. The sharing end combines the first main stream and the shared auxiliary stream;
[0172] Steps 1107 to 1108 in this embodiment are similar to those in the previous embodiment. Figure 8 Steps 808 to 809 in the illustrated embodiment are similar and will not be described in detail herein.
[0173] 2. The speaking end is the receiving end;
[0174] In this embodiment, there may be multiple receiving terminals in the conference screen sharing system, including a first receiving terminal and a second receiving terminal. As an example, the implementation of the first receiving terminal is described below. Similarly, the implementation of the second receiving terminal can refer to the implementation of the first receiving terminal. Fig.12 When the speaking end is the local first receiving end, a conference screen sharing method in an embodiment of the present application includes:
[0175] 1201. A first receiving end obtains a first main stream;
[0176] The first receiving end obtains the local second mainstream, performs cutout processing on the second mainstream, and obtains the first mainstream including the portrait floating window image.
[0177] 1202. The sharing end sends a shared auxiliary stream to the server end;
[0178] 1203. The server sends a shared auxiliary stream to the first receiving end;
[0179] Steps 1202 to 1203 in this embodiment are similar to those in the previous embodiment. Fig.10 Steps 1004 to 1005 in the illustrated embodiment are similar and will not be described in detail herein.
[0180] 1204. The first receiving end synthesizes the first main stream and the shared auxiliary stream;
[0181] 1205. The first receiving end sends a first mainstream to the server end;
[0182] 1206. The server sends the first mainstream to the sharing terminal;
[0183] 1207. The sharing end combines the first main stream and the shared auxiliary stream;
[0184] Steps 1204 to 1207 in this embodiment are similar to those in the previous Fig.10 Steps 1007 to 1010 in the illustrated embodiment are similar and will not be described in detail herein.
[0185] See also Fig.13 , when the speaking end is the second receiving end, a conference screen sharing method in an embodiment of the present application includes:
[0186] 1301. The second receiving end sends a second main stream to the server end;
[0187] Step 1301 of this embodiment is the same as the aforementioned Figure 8 Step 802 in the illustrated embodiment is similar and will not be described in detail here.
[0188] 1302. The server sends the third main stream to the receiving end;
[0189] 1303. The server sends an indication message to the receiver;
[0190] 1304. The receiving end performs a cutout process on the second main stream;
[0191] Steps 1302 to 1304 in this embodiment are similar to those in the previous embodiment. Fig.10 Steps 1002 to 1004 in the illustrated embodiment are similar and will not be described in detail herein.
[0192] 1305. The server performs cutout processing on the second mainstream;
[0193] 1306. The server sends the first main stream to the receiving end;
[0194] Steps 1305 to 1306 in this embodiment are similar to those in the previous embodiment. Fig.10 Steps 1102 to 1103 in the illustrated embodiment are similar and will not be described in detail herein.
[0195] In this embodiment, only steps 1302 to 1304 may be executed without executing steps 1305 to 1305; or only steps 1305 to 1306 may be executed without executing steps 1302 to 1304. The specifics are not limited here.
[0196] 1307. The sharing end sends the shared auxiliary stream to the server end;
[0197] 1308. The server sends the shared auxiliary stream to the receiving end;
[0198] 1309. The receiving end combines the first main stream and the shared auxiliary stream;
[0199] 1310. The server sends the first mainstream to the sharing terminal;
[0200] 1311. The sharing end combines the first main stream and the shared auxiliary stream;
[0201] Steps 1306 to 1310 in this embodiment are similar to the aforementioned Fig.11 Steps 1104 to 1108 in the illustrated embodiment are similar and will not be described in detail here.
[0202] 4. In the same LAN, the sharing end synthesizes and sends the composite code stream to the receiving end;
[0203] The shared end can combine the first main stream and the shared auxiliary stream into one stream, and send the stream directly between terminals in the same LAN, so it does not need to be forwarded by the server. Fig.14 As shown in the figure, the sharing end synthesizes the first main stream and the shared auxiliary stream, and then sends the composite stream to the receiving end. Depending on the speaking end, the sharing end obtains the first main stream in a variety of ways:
[0204] 1. The speaking end is the shared end;
[0205] See also Fig.15 In an embodiment of the present application, a conference screen sharing method includes:
[0206] 1501. The sharing terminal obtains the first mainstream;
[0207] 1502. The sharing end combines the first main stream and the shared auxiliary stream;
[0208] Steps 1501 to 1502 in this embodiment are similar to those in the previous embodiment. Figure 4 Steps 401 to 402 in the illustrated embodiment are similar and will not be described in detail herein.
[0209] 1503. The sharing end sends a composite code stream to the receiving end;
[0210] After the sharing end encodes the second image, a composite code stream is obtained. Since the sharing end and the receiving end are in the same local area network, the sharing end can directly send the code stream to the receiving end without passing through the server end.
[0211] 1504. The receiving end decodes and displays the composite code stream;
[0212] Step 1504 in this embodiment is similar to the above Figure 4 Step 405 in the illustrated embodiment is similar and will not be described in detail here.
[0213] 2. The speaking end is the receiving end;
[0214] See also Fig.16 In an embodiment of the present application, a conference screen sharing method includes:
[0215] 1601. The receiving end sends the second main stream to the sharing end;
[0216] In the same LAN, the receiving end can directly send the second mainstream to the sharing end. The second mainstream includes the local portrait mainstream of the receiving end.
[0217] 1602. The sharing end performs cutout processing on the second main stream;
[0218] 1603. The sharing end combines the first main stream and the shared auxiliary stream;
[0219] Steps 1602 to 1603 in this embodiment are similar to those in the previous embodiment. Figure 5 Steps 504 to 505 in the illustrated embodiment are similar and will not be described in detail herein.
[0220] 1604. The sharing end sends a composite code stream to the receiving end;
[0221] 1605. The receiving end decodes and displays the composite code stream;
[0222] Steps 1604 to 1605 in this embodiment are similar to those in the previous embodiment. Fig.15 Steps 1503 to 1504 in the illustrated embodiment are similar and will not be described in detail here.
[0223] 5. In the same local area network, the receiving end directly synthesizes the second image;
[0224] like Fig.17 As shown, the sharing end sends the shared auxiliary stream to the receiving end, and the receiving end obtains the first main stream and synthesizes the second image with the shared auxiliary stream, which is directly displayed on the receiving end. The receiving end then sends the first main stream to the sharing end, so that the sharing end can synthesize the second image and display it. Depending on the speaking end, the receiving end obtains the first main stream in a variety of ways:
[0225] 1. The speaking end is the shared end;
[0226] See also Fig.18 In an embodiment of the present application, a conference screen sharing method includes:
[0227] 1801. The sharing end sends the second main stream to the receiving end;
[0228] In the same LAN, the sharing end can directly send the second mainstream to the receiving end. The second mainstream includes the local portrait mainstream of the sharing end.
[0229] 1802. The receiving end performs cutout processing on the second main stream;
[0230] Step 1802 in this embodiment is similar to the above Fig.10 Step 1006 in the illustrated embodiment is similar and will not be described in detail here.
[0231] 1803. The sharing end sends the shared auxiliary stream to the receiving end;
[0232] The sharing end sends a shared auxiliary stream to the receiving end, where the shared auxiliary stream includes a screen sharing image for the receiving end to synthesize a second image.
[0233] 1804. The receiving end combines the first main stream and the shared auxiliary stream;
[0234] Step 1804 in this embodiment is similar to the above Fig.10 Step 1007 in the illustrated embodiment is similar and will not be described in detail here.
[0235] 1805. The receiving end sends the first main stream to the sharing end;
[0236] The receiving end sends the first mainstream after the cutout processing to the sharing end, and the sharing end synthesizes the first mainstream to obtain the second image.
[0237] 1806. The sharing end combines the first main stream and the shared auxiliary stream;
[0238] Step 1806 in this embodiment is similar to the above Fig.10 Step 1010 in the illustrated embodiment is similar and will not be described in detail here.
[0239] 2. The speaking end is the receiving end;
[0240] See also Fig.19 In an embodiment of the present application, a conference screen sharing method includes:
[0241] 1901. The receiving end obtains the first mainstream;
[0242] Step 1901 in this embodiment is similar to the above Fig.12 Step 1201 in the illustrated embodiment is similar and will not be described in detail here.
[0243] 1902. The sharing end sends the shared auxiliary stream to the receiving end;
[0244] 1903. The receiving end combines the first main stream and the shared auxiliary stream;
[0245] 1904. The receiving end sends the first main stream to the sharing end;
[0246] 1905. The sharing end combines the first main stream and the shared auxiliary stream.
[0247] Steps 1902 to 1905 in this embodiment are similar to those in the previous embodiment. Fig.18 Steps 1803 to 1805 in the illustrated embodiment are similar and will not be described in detail here.
[0248] The present application also provides a conference screen sharing system 2000, such as Fig. 20 As shown, including:
[0249] The sharing end 2001 is used to obtain a first mainstream of a first receiving end 2002, where the first receiving end 2002 is a speaking end, and the first mainstream includes a first image;
[0250] The sharing end 2001 is further configured to synthesize and encode the shared auxiliary stream and the first main stream to obtain a composite code stream, wherein the composite code stream includes a second image, which is synthesized by the screen sharing image and the first image, and the shared auxiliary stream includes the screen sharing image;
[0251] The sharing terminal 2001 is also used to display the second image.
[0252] The shared end and the first receiving end can be implemented by software or hardware. Exemplarily, the implementation of the shared end is introduced below. Similarly, the implementation of the first receiving end can refer to the implementation of the shared end.
[0253] As an example of a software functional unit, the shared end may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above-mentioned computing instance may be one or more. For example, the shared end may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.
[0254] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0255] As an example of a hardware functional unit, the shared end may include at least one computing device, such as a server, etc. Alternatively, the shared end may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0256] The multiple computing devices included in the shared end can be distributed in the same region or in different regions. The multiple computing devices included in the shared end can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the shared end can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0257] The present application also provides a computing device 2100. Fig.21 As shown, the computing device 2100 includes: a bus 2101, a processor 2102, a memory 2103 and a communication interface 2104. The processor 2102, the memory 2103 and the communication interface 2104 communicate with each other through the bus 2101. The computing device 2100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 2100.
[0258] The bus 2101 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus 2102 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 2102 may include a path for transmitting information between various components of the computing device 2100 (for example, the memory 2103, the processor 2102, and the communication interface 2104).
[0259] The processor 2102 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0260] The memory 2103 may include a volatile memory, such as a random access memory (RAM). The processor 2102 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0261] The memory 2103 stores executable codes, and the processor 2102 executes the executable codes to respectively implement the functions of the aforementioned sharing terminal and the first receiving terminal, thereby implementing the conference screen sharing method. That is, the memory 2103 stores instructions for executing the conference screen sharing method.
[0262] The communication interface 2104 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 2100 and other devices or communication networks.
[0263] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.
[0264] like Fig. 22 As shown, the computing device cluster includes at least one computing device 2100. The memory 2103 in one or more computing devices 2100 in the computing device cluster may store the same instructions for executing the conference screen sharing method.
[0265] In some possible implementations, the memory 2103 of one or more computing devices 2100 in the computing device cluster may also store partial instructions for executing the conference screen sharing method. In other words, the combination of one or more computing devices 2100 may jointly execute instructions for executing the conference screen sharing method.
[0266] It should be noted that the memory 2103 in different computing devices 2100 in the computing device cluster can store different instructions for executing part of the functions of the database cluster. That is, the instructions stored in the memory 2103 in different computing devices 2100 can implement the functions of one or more nodes in the sharing end and the first receiving end.
[0267] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.23 A possible implementation is shown. Fig.23 As shown, two computing devices 2100A and 2100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 2103 in the computing device 2100A stores instructions for executing the functions of the sharing end. At the same time, the memory 2103 in the computing device 2100B stores instructions for executing the functions of the first receiving end.
[0268] It should be understood that Fig.23 The functions of the computing device 2100A shown in FIG. 2100A may also be completed by multiple computing devices 2100. Similarly, the functions of the computing device 2100B may also be completed by multiple computing devices 2100.
[0269] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes a conference screen sharing method or a conference screen sharing method.
[0270] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device to execute a conference screen sharing method, or instruct a computing device to execute a conference screen sharing method.
[0271] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. A conference screen sharing method, characterized in that: The method is applied to a conference screen sharing system, wherein the conference screen sharing system includes a sharing end and a first receiving end, wherein the sharing end is a user device that initiates screen sharing in a conference, and the first receiving end is a user device that receives screen sharing in a conference. A user device in a speaking state in the conference screen sharing system is a speaking end. The method includes: The sharing end obtains a first mainstream of the first receiving end, where the first receiving end is the speaking end, and the first mainstream includes a first image; The sharing end performs synthesis encoding on the shared auxiliary stream and the first main stream to obtain a composite code stream, wherein the composite code stream includes a second image, the second image is synthesized by a screen sharing image and the first image, and the shared auxiliary stream includes the screen sharing image; The sharing end displays the second image.
2. The conference screen sharing method according to claim 1, characterized in that: The first image is a first portrait.
3. The conference screen sharing method according to claim 2, characterized in that: The sharing end obtains the first main stream of the first receiving end, including: The sharing end receives a second mainstream from the first receiving end, where the second mainstream is a portrait mainstream including a background generated by the first receiving end; The sharing end performs cutout processing on the second mainstream to obtain the first mainstream.
4. The conference screen sharing method according to any one of claims 1 to 3, characterized in that: The method further comprises: The sharing end sends the composite code stream to the first receiving end.
5. The conference screen sharing method according to claim 3, characterized in that: The conference screen sharing system further includes a server end, the server end is used to detect the speaking end and perform media processing on the code stream, and the sharing end receives the second mainstream from the first receiving end, including: The server receives a second mainstream from the first receiving end, where the second mainstream is a portrait mainstream generated by the first receiving end; The sharing end receives a third mainstream and an indication message from the service end, wherein the third mainstream includes the second mainstream and mainstreams generated by other receiving ends; The sharing terminal obtains the second mainstream from the third mainstream according to the indication message.
6. The conference screen sharing method according to any one of claims 1 or 2, characterized in that: The conference screen sharing system further includes a server end, the server end is used to detect the speaking end and perform media processing on the code stream, and the sharing end receives the first mainstream from the first receiving end, including: The server receives a second mainstream from the first receiving end, where the second mainstream is a portrait mainstream including a background generated by the first receiving end; The server performs cutout processing on the second mainstream to obtain the first mainstream; The sharing end receives the first mainstream from the service end.
7. The conference screen sharing method according to claim 1 or 2, characterized in that: The conference screen sharing system further includes a server end, which is used to detect the speaking end and perform media processing on the code stream. The sharing end obtains the first mainstream of the first receiving end, including: The server receives the first mainstream from the receiving end; The sharing end receives a fourth mainstream and an indication message from the service end, wherein the fourth mainstream includes the first mainstream and mainstreams generated by other receiving ends; The sharing terminal determines the first mainstream from the fourth mainstream according to the indication message.
8. The conference screen sharing method according to any one of claims 5 to 7, characterized in that: The method further comprises: The sharing end sends the composite code stream to the service end.
9. The conference screen sharing method according to any one of claims 5 to 7, characterized in that: The conference screen sharing system further includes a second receiving end and a server end, wherein the second receiving end is a user device that receives screen sharing in the conference, and the server end is used to detect the speaking end and perform media processing on the code stream. The method further includes: The server receives a second mainstream from the first receiving end, where the second mainstream is a portrait mainstream generated by the first receiving end; The second receiving end receives a third mainstream from the server end, wherein the third mainstream includes the second mainstream; The sharing end receives a third mainstream and an indication message from the service end, wherein the third mainstream includes the second mainstream and mainstreams generated by other receiving ends; The sharing terminal acquires the second mainstream from the third mainstream according to the indication message; The second receiving end performs a cutout process on the second mainstream to obtain the first mainstream; The second receiving end synthesizes the shared auxiliary stream and the first main stream to obtain and display the second image.
10. The conference screen sharing method according to claim 6, characterized in that: The conference screen sharing system further includes a second receiving end, where the second receiving end is a user device that receives screen sharing in the conference. The method further includes: The second receiving end receives the first mainstream from the server end; The second receiving end synthesizes the shared auxiliary stream and the first main stream to obtain and display the second image.
11. The conference screen sharing method according to claim 6, characterized in that: The method further comprises: The server synthesizes the shared auxiliary stream and the first main stream to obtain the composite stream; The server sends the composite code stream to the first receiver.
12. The conference screen sharing method according to any one of claims 1 to 11, characterized in that: The second image is synthesized based on a principle of occlusion minimization, and the first image is composited to a first area in the screen sharing image, where the first area includes a blank area or an area with the simplest texture in the screen sharing image.
13. The conference screen sharing method according to any one of claims 1 to 11, characterized in that: The first image is composited into a second area in the screen sharing image, and the second area is set by a user corresponding to the sharing end.
14. A conference screen sharing system, characterized in that: The conference screen sharing system comprises: A sharing end, used for acquiring a first mainstream of a first receiving end, where the first receiving end is the speaking end, and the first mainstream includes a first image; The sharing end is further configured to synthesize and encode the shared auxiliary stream and the first main stream to obtain a composite code stream, wherein the composite code stream includes a second image, the second image is synthesized by a screen sharing image and the first image, and the shared auxiliary stream includes the screen sharing image; The sharing end is further used to display the second image.
15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 13.
16. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 13.
17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 13.