Conference data processing method and device, equipment, medium and product

Abstract

The invention provides a conference data processing method and device, equipment, a medium and a product. The method comprises the following steps: receiving a conference data processing request sent by a conference client; analyzing the conference data processing request to obtain a target resource address and a processing parameter; generating a processing instruction; and sending the processing instruction to an idle engine instance allocated to the conference client, so that the idle engine instance pulls the conference data streams from the corresponding conference client according to the target resource address, processing the pulled conference data streams according to the processing parameters to obtain processed conference data streams, and sending the processed conference data streams to the conference client according to the processed conference data streams. And sending the processed conference data stream to a corresponding conference client, so that the corresponding conference client performs rendering according to the processed conference data stream. The method does not need a server for transfer, realizes real-time rendering of millisecond-level delay, reduces bandwidth consumption, improves the real-time performance and interaction experience of multi-party cooperation, and meets the requirements of real-time interaction and content co-creation in a complex conference scene.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a conference data processing method, device, equipment, medium, and product. Background Art

[0002] With the increasing popularity of remote work and online education, intelligent conferencing systems have become a core collaboration tool. However, local clients currently rely on servers to relay and process data streams, resulting in high latency and high bandwidth consumption, making it difficult to meet the low-latency interaction requirements of conferencing scenarios. Summary of the Invention

[0003] In response to at least one of the above technical problems, embodiments of the present invention provide a conference data processing method, apparatus, device, medium, and product.

[0004] According to a first aspect, a conference data processing method provided by an embodiment of the present invention is applied to a conference engine management device, the method comprising:

[0005] Receive conference data processing requests sent by conference clients;

[0006] Parsing the conference data processing request to obtain a target resource address and processing parameters; wherein the target resource address is the URL of the conference data stream of the conference client of all participants;

[0007] generating a processing instruction according to the target resource address and the processing parameters;

[0008] Sending the processing instruction to an idle engine instance allocated to the conference client, so that the idle engine instance pulls the conference data stream from the corresponding conference client according to the target resource address in the processing instruction, processes each pulled conference data stream according to the processing parameters in the processing instruction to obtain a processed conference data stream, and sends the processed conference data stream to the corresponding conference client, so that the corresponding conference client performs rendering according to the processed conference data stream;

[0009] The idle engine instance is an engine instance in an idle state in the engine cluster. The engine instances in the engine cluster are deployed in a distributed manner, and each engine instance can process multiple conference data streams in parallel.

[0010] In one embodiment, before receiving the conference data processing request sent by the conference client, the method further includes:

[0011] Receive service application requests sent by conference clients;

[0012] Verify the user's permissions in the service application request;

[0013] After the permission verification is passed, query the current status of each engine instance in the engine cluster;

[0014] According to the current status of each engine instance, an idle engine instance is selected and allocated to the conference client.

[0015] In one embodiment, the performing of authority verification on the user in the service application request includes:

[0016] Verify whether the user in the service application request is a legitimate user;

[0017] If the user is a legitimate user, verify whether the user is the conference host;

[0018] If it is the conference host, the authority verification is passed;

[0019] If you are not the conference host, the permission verification will fail.

[0020] In one embodiment, selecting an idle engine instance to allocate to the conference client according to the current status of each engine instance includes:

[0021] According to the current status of each engine instance, an engine instance that is idle and supports the mobile terminal low bit rate technology is selected and allocated to the conference client.

[0022] In one embodiment, the method further comprises: determining a stream address and encryption parameters of the idle engine instance;

[0023] The sending of the processing instruction to the idle engine instance allocated to the conference client includes: encrypting the processing instruction according to the encryption parameter and then sending the encrypted instruction to the idle engine instance corresponding to the stream address.

[0024] In one embodiment, the conference data processing request includes any one of the following: a split-screen layout request, a real-time subtitle request, and a recording and editing request.

[0025] According to a second aspect, an embodiment of the present invention provides a conference data processing device configured on a conference engine management device, the device comprising:

[0026] A first receiving module is used to receive a conference data processing request sent by a conference client;

[0027] A request parsing module, configured to parse the conference data processing request to obtain a target resource address and processing parameters; wherein the target resource address is the URL of the conference data stream of the conference client of all participants;

[0028] An instruction generation module, configured to generate a processing instruction according to the target resource address and the processing parameters;

[0029] An instruction sending module is used to send the processing instruction to an idle engine instance allocated to the conference client, so that the idle engine instance pulls the conference data stream from the corresponding conference client according to the target resource address in the processing instruction, processes each pulled conference data stream according to the processing parameters in the processing instruction to obtain a processed conference data stream, and sends the processed conference data stream to the corresponding conference client, so that the corresponding conference client renders according to the processed conference data stream; wherein the idle engine instance is an engine instance in an idle state in the engine cluster, and each engine instance in the engine cluster adopts a distributed deployment, and each engine instance can process multiple conference data streams in parallel.

[0030] According to a third aspect, an embodiment of the present invention provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided in the first aspect is implemented.

[0031] According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method provided in the first aspect.

[0032] According to a fifth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program, which implements the method provided in the first aspect when executed by a processor.

[0033] The conference data processing method, apparatus, device, medium, and product provided by the embodiments of the present invention are as follows: after receiving a conference data processing request sent by a conference client, the conference engine management device parses the conference data processing request, generates a processing instruction, and sends the processing instruction to an idle engine instance allocated to the conference client. In this way, the idle engine instance pulls the conference data stream from the corresponding conference client according to the target resource address in the processing instruction, processes each pulled conference data stream according to the processing parameters in the processing instruction, obtains a processed conference data stream, and sends the processed conference data stream to the corresponding conference client so that the corresponding conference client renders according to the processed conference data stream. It can be seen that the idle engine instance directly pulls the conference data stream from the conference client, processes the pulled conference data stream, and directly sends the processed conference data stream to the conference client of each participant after processing, without the need for server transfer, achieving real-time rendering with millisecond-level delay, reducing bandwidth consumption, improving the real-time and interactive experience of multi-party collaboration, and meeting the real-time interaction and content co-creation needs in complex conference scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 1 is a flow chart of a conference data processing method according to an embodiment of the present invention;

[0035] Figure 2 FIG. 1 is a structural block diagram of a conference data processing device in one embodiment of the present invention. DETAILED DESCRIPTION

[0036] In a first aspect, the present invention provides a conference data processing method, which is applied to a conference engine management device, see Figure 1 , the method comprises the following steps:

[0037] S110, receiving a conference data processing request sent by a conference client;

[0038] The conference data processing method provided by the present invention involves a conference client, a conference engine management device, an engine cluster and a resource storage device.

[0039] After a conference user initiates a conference data processing request on the conference client, the conference client sends the conference data processing request to the conference engine management device. The conference client also receives and renders the processed conference data stream pushed by the engine instance.

[0040] The conference data processing request may include any one of the following: a split-screen layout request, a real-time subtitle request, and a recording and editing request.

[0041] The recording and editing request, for example, extracts the 00:00-00:10 segment from participant A's video stream and merges it with participant B's audio stream.

[0042] Of course, in addition to the aforementioned conference data processing requests, dynamic layout synthesis instructions can also be included, such as automatically adjusting the screen size based on the speaker's volume, with the speaker's screen occupying 80% of the screen. They can also include real-time special effects rendering instructions, such as adding a conference-specific watermark to all video streams with 50% transparency.

[0043] The conference data processing request may be, but is not limited to, an HTTP request.

[0044] S120: Parse the conference data processing request to obtain a target resource address and processing parameters; wherein the target resource address is the URL of the conference data stream of the conference client of all participants;

[0045] The conference data processing request may be parsed by the secondary management interface of the conference engine management device.

[0046] S130, generating a processing instruction according to the target resource address and the processing parameters;

[0047] The processing instruction may be, but is not limited to, a TCP instruction.

[0048] S140. Send the processing instruction to the idle engine instance allocated to the conference client, so that the idle engine instance pulls the conference data stream from the corresponding conference client according to the target resource address in the processing instruction, processes each pulled conference data stream according to the processing parameters in the processing instruction to obtain a processed conference data stream, and sends the processed conference data stream to the corresponding conference client, so that the corresponding conference client renders according to the processed conference data stream; wherein the idle engine instance is an engine instance in an idle state in the engine cluster, each engine instance in the engine cluster adopts a distributed deployment, and each engine instance can process multiple conference data streams in parallel.

[0049] As can be seen, the engine cluster contains multiple engine instances, each deployed in a distributed manner. Each engine instance can process multiple conference data streams in parallel (real-time synthesis of ≥10 1080p video streams), thereby improving data stream processing efficiency. The engine instances support distributed parallel processing, distributing processing instructions through a load balancing algorithm to ensure efficient video processing in conferences.

[0050] For example, a conference data processing request is for a three-person split-screen. The target resource address includes the URLs of the video streams for participants A, B, and C. Processing parameters include the split-screen ratio (1:1:1), screen coordinates ((0,0), (640,0), (0,640)), and resolution (640×480). The generated processing instruction includes: "Synthesize 3 streams, input = [A, B, C], layout = split 3 equally, output resolution = 640×480." The idle engine instance assigned to the conference client is instance E1. The conference engine management device sends the processing instruction to instance E1, which pulls the conference data streams from the conference clients of participants A, B, and C. It scales the three conference data streams according to the processing parameters (the resolution after scaling is 640×480) and positions them, synthesizing them into a new data stream, the processed conference data stream. Instance E1 sends the processed conference data stream to the conference clients of participants A, B, and C, allowing these three conference clients to directly render the content into a split-screen image with a resolution of 640×480 without decoding the original conference data stream, thereby reducing device power consumption.

[0051] Of course, in the above processing, instance E1 stores the generated temporary composite file in the resource storage device, and the participants can download the complete recorded video after the meeting.

[0052] Among them, a two-way data channel can be established between the conference client and the engine instance through WebRTC (Web Real-Time Communication) to support real-time interactive operations: participants drag the screen in the conference client to adjust the split-screen layout, and the operation instructions are transmitted to the engine instance in real time through the data channel, triggering layout recalculation; the engine instance can also push processing progress information (for example, video rendering is 60% complete) to the conference client to display loading status or error prompts.

[0053] In one embodiment, before receiving the conference data processing request sent by the conference client, the method may further include:

[0054] Receive service application requests sent by conference clients;

[0055] Verify the user's permissions in the service application request;

[0056] After the permission verification is passed, query the current status of each engine instance in the engine cluster;

[0057] According to the current status of each engine instance, an idle engine instance is selected and allocated to the conference client.

[0058] In a real-world scenario, a participant opens a conference link and accesses the conference client. The client then sends a service request to the conference engine management device, including information such as the participant ID and device type. Upon receiving the request, the engine management device verifies the permissions of the participant corresponding to the participant ID in the request. Once the permissions are verified, the device queries the current status of each engine instance in the engine cluster and selects an idle engine instance to allocate to the conference client.

[0059] Furthermore, the authorization verification of the user in the service application request may include:

[0060] Verify whether the user in the service application request is a legitimate user;

[0061] If the user is a legitimate user, verify whether the user is the conference host;

[0062] If it is the conference host, the authority verification is passed;

[0063] If you are not the conference host, the permission verification will fail.

[0064] Among them, legal users refer to users registered in the conference client.

[0065] Among them, the legitimacy and authority can be verified through the first-level management interface of the conference engine management device.

[0066] It can be seen that the user who has registered on the conference client and is the conference host has passed the authority verification, so the user as the conference host can initiate a conference data processing request on the conference client.

[0067] Furthermore, selecting an idle engine instance to allocate to the conference client according to the current status of each engine instance may include:

[0068] According to the current status of each engine instance, an engine instance that is idle and supports the mobile terminal low bit rate technology is selected and allocated to the conference client.

[0069] As you can see, when selecting an engine instance, in addition to considering availability, you should also consider whether the engine instance supports mobile low-bitrate technology. Mobile low-bitrate technology addresses the limited network bandwidth and wide variations in hardware performance of mobile devices. By optimizing video encoding, transmission protocols, and resource allocation strategies, it reduces data transmission while ensuring video quality. This reduces data transmission while maintaining video quality even when network bandwidth is limited or hardware performance is low, thus ensuring smooth execution of the engine instance's processing.

[0070] In one embodiment, the method may further include: determining the stream address and encryption parameters of the idle engine instance; sending the processing instruction to the idle engine instance allocated to the conference client includes: encrypting the processing instruction according to the encryption parameters and sending it to the idle engine instance corresponding to the stream address.

[0071] The stream address may be a WebRTC stream address.

[0072] The encryption parameter may be a DTLS (Datagram Transport Layer Security) encryption parameter.

[0073] It can be seen that encryption can ensure data security, and stream addresses can ensure that processing instructions can be delivered to the idle engine instance.

[0074] Based on the above description, the conference engine management device includes:

[0075] Level 1 management interface: handles user login authentication (e.g., legitimacy verification, permission verification), monitors the load status of the conference engine cluster, and dynamically allocates idle instances.

[0076] Secondary processing interface: parses the conference data processing request from the conference client, generates a processing instruction containing the target resource address and processing parameters, and sends it to the specified idle engine instance.

[0077] Among them, the engine instances in the engine cluster can directly communicate with the conference client and resource storage device through WebRTC.

[0078] Among them, the resource storage device is used to store all data generated in the meeting, including: original input streams (for example, camera video, microphone audio, screen sharing data); processing intermediate files (for example, temporary frame data generated during the editing process); final output files (for example, meeting recording video, synthesized multi-screen files, audio streams with special effects).

[0079] It can be seen that the engine instance in the present invention directly pulls the conference data stream from the conference client, processes the pulled conference data stream, and directly sends the processed conference data stream to the conference client of each participant after processing, without the need for server transfer, to achieve real-time rendering with millisecond-level delay. The selection of the engine instance is achieved through the conference engine management device. It can be seen that the present invention solves the efficiency and real-time problems of video processing in smart conferences by constructing a conference engine management device and an engine cluster, realizes online real-time editing, multi-stream synthesis and efficient rendering of conference videos, reduces device dependence, breaks through platform limitations, improves the real-time and interactive experience of multi-party collaboration, and meets the real-time interaction and content co-creation needs in complex conference scenarios.

[0080] In a second aspect, an embodiment of the present invention provides a conference data processing device configured in a conference engine management device, see Figure 2 , the device comprises:

[0081] A first receiving module is used to receive a conference data processing request sent by a conference client;

[0082] A request parsing module, configured to parse the conference data processing request to obtain a target resource address and processing parameters; wherein the target resource address is the URL of the conference data stream of the conference client of all participants;

[0083] An instruction generation module, configured to generate a processing instruction according to the target resource address and the processing parameters;

[0084] An instruction sending module is used to send the processing instruction to an idle engine instance allocated to the conference client, so that the idle engine instance pulls the conference data stream from the corresponding conference client according to the target resource address in the processing instruction, processes each pulled conference data stream according to the processing parameters in the processing instruction to obtain a processed conference data stream, and sends the processed conference data stream to the corresponding conference client, so that the corresponding conference client renders according to the processed conference data stream; wherein the idle engine instance is an engine instance in an idle state in the engine cluster, and each engine instance in the engine cluster adopts a distributed deployment, and each engine instance can process multiple conference data streams in parallel.

[0085] In one embodiment, the apparatus further comprises:

[0086] The instance selection module is used to receive a service application request sent by the conference client before the first receiving module receives the conference data processing request sent by the conference client; verify the authority of the user in the service application request; after the authority verification is passed, query the current status of each engine instance in the engine cluster; and select an idle engine instance to allocate to the conference client based on the current status of each engine instance.

[0087] In one embodiment, the instance selection module performs permission verification on the user in the service application request, including: verifying whether the user in the service application request is a legitimate user; if it is a legitimate user, verifying whether the user is the meeting host; if it is the meeting host, the permission verification passes; if it is not the meeting host, the permission verification fails.

[0088] In one embodiment, the instance selection module selects an idle engine instance to be assigned to the conference client based on the current status of each engine instance, including: selecting an engine instance that is idle and supports mobile low bit rate technology to be assigned to the conference client based on the current status of each engine instance.

[0089] In one embodiment, the apparatus further comprises:

[0090] a parameter determination module, configured to determine a stream address and encryption parameters of the idle engine instance;

[0091] The instruction sending module is specifically configured to: encrypt the processing instruction according to the encryption parameter and send the encrypted instruction to the idle engine instance corresponding to the stream address.

[0092] In one embodiment, the conference data processing request includes any one of the following: a split-screen layout request, a real-time subtitle request, and a recording and editing request.

[0093] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the device provided by the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0094] In a third aspect, the present application provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any one of the above embodiments.

[0095] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided by the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0096] In a fourth aspect, the present application provides a computer-readable medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the processor executes the method provided in the first aspect.

[0097] Specifically, a system or device equipped with a storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program codes stored in the storage medium.

[0098] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0099] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0100] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0101] In addition, it can be understood that the program code read from the storage medium is written into a memory provided in an expansion board inserted into the computer or into a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0102] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer-readable medium provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0103] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the method provided in the first aspect when executed by a processor.

[0104] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer program product provided by the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0105] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0106] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented using hardware, software, widgets, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0107] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A conference data processing method, characterized in that: Applied to a conference engine management device, the method includes: Receive conference data processing requests sent by conference clients; Parsing the conference data processing request to obtain a target resource address and processing parameters; wherein the target resource address is the URL of the conference data stream of the conference client of all participants; generating a processing instruction according to the target resource address and the processing parameters; Sending the processing instruction to an idle engine instance allocated to the conference client, so that the idle engine instance pulls the conference data stream from the corresponding conference client according to the target resource address in the processing instruction, processes each pulled conference data stream according to the processing parameters in the processing instruction to obtain a processed conference data stream, and sends the processed conference data stream to the corresponding conference client, so that the corresponding conference client performs rendering according to the processed conference data stream; The idle engine instance is an engine instance in an idle state in the engine cluster. The engine instances in the engine cluster are deployed in a distributed manner, and each engine instance can process multiple conference data streams in parallel.

2. The method according to claim 1, characterized in that Before receiving the conference data processing request sent by the conference client, the method further includes: Receive service application requests sent by conference clients; Verify the user's permissions in the service application request; After the permission verification is passed, query the current status of each engine instance in the engine cluster; According to the current status of each engine instance, an idle engine instance is selected and allocated to the conference client.

3. The method according to claim 2, characterized in that The performing of authority verification on the user in the service application request includes: Verify whether the user in the service application request is a legitimate user; If the user is a legitimate user, verify whether the user is the conference host; If it is the conference host, the authority verification is passed; If you are not the conference host, the permission verification will fail.

4. The method according to claim 2, characterized in that According to the current status of each engine instance, selecting an idle engine instance to allocate to the conference client includes: According to the current status of each engine instance, an engine instance that is idle and supports the mobile terminal low bit rate technology is selected and allocated to the conference client.

5. The method according to claim 1, wherein Also includes: Determining a stream address and encryption parameters for the idle engine instance; The sending of the processing instruction to the idle engine instance allocated to the conference client includes: encrypting the processing instruction according to the encryption parameter and then sending the encrypted instruction to the idle engine instance corresponding to the stream address.

6. The method according to claim 1, wherein The conference data processing request includes any one of the following: a split-screen layout request, a real-time subtitle request, and a recording and editing request.

7. A conference data processing device, characterized in that: Configured in a conference engine management device, the device includes: A first receiving module is used to receive a conference data processing request sent by a conference client; A request parsing module, configured to parse the conference data processing request to obtain a target resource address and processing parameters; wherein the target resource address is the URL of the conference data stream of the conference client of all participants; An instruction generation module, configured to generate a processing instruction according to the target resource address and the processing parameters; An instruction sending module is used to send the processing instruction to an idle engine instance allocated to the conference client, so that the idle engine instance pulls the conference data stream from the corresponding conference client according to the target resource address in the processing instruction, processes each pulled conference data stream according to the processing parameters in the processing instruction to obtain a processed conference data stream, and sends the processed conference data stream to the corresponding conference client, so that the corresponding conference client renders according to the processed conference data stream; wherein the idle engine instance is an engine instance in an idle state in the engine cluster, and each engine instance in the engine cluster adopts a distributed deployment, and each engine instance can process multiple conference data streams in parallel.

8. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images