A data processing method, device, apparatus, and storage medium

Abstract

Embodiments of the present application disclose a data processing method, device and equipment, and a storage medium, which can be applied to various fields or scenarios such as cloud technology, artificial intelligence, block chain, Internet of Vehicles, intelligent transportation, smart home and the like. The data processing method comprises: obtaining network data of a plurality of first clients; calling an audio / video filtering engine to perform filtering processing on the network data of each first client in the plurality of first clients to obtain audio / video data of each first client; calling a mixed flow processing module to perform mixed flow processing on the audio / video data of the plurality of first clients to obtain a mixed audio / video data stream; and sending the mixed audio / video data stream to a second client to enable the second client to display the mixed audio / video data stream. By using the present application, mixed flow services can be performed by using an intelligent network card, CPU resources are released, and the processing performance of a media server is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of mobile internet, multi-person video services have gradually emerged. In multi-person video conferencing or live video streaming scenarios, we often need to display the video feeds of multiple people on one screen so that participants or hosts can interact through video and voice. This requires the use of audio-video mixing technology to combine multiple audio and video streams into one.

[0003] Currently, audio and video mixing is usually handled by the central processing unit (CPU) of the media server. The network card of the media server needs to write the received audio and video data into memory for the CPU to process. The CPU has a high workload of processing multiple audio and video data at the same time, which makes the media server slow and affects the user experience. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium that can efficiently respond to audio and video data mixing requests through a smart network card, thereby improving the user experience.

[0005] On one hand, embodiments of this application provide a data processing method applied to a smart network interface card (NIC) of a media server. The smart NIC includes an audio / video filtering engine and a mixing processing module. The method includes:

[0006] Obtain network data from multiple first clients;

[0007] The audio and video filtering engine is invoked to filter the network data of each of the multiple first clients, thereby obtaining the audio and video data of each first client;

[0008] The mixing module is invoked to mix the audio and video data from multiple first clients, resulting in a mixed audio and video data stream.

[0009] Send a mixed audio and video data stream to the second client so that the second client can display the mixed audio and video data stream.

[0010] On the other hand, embodiments of this application provide a data processing apparatus, including:

[0011] A receiving unit is used to acquire network data from multiple first clients;

[0012] The processing unit is used to call the audio and video filtering engine to filter the network data of each of the multiple first clients to obtain the audio and video data of each first client; and to call the mixing processing module to mix the audio and video data of the multiple first clients to obtain a mixed audio and video data stream.

[0013] The sending unit is used to send a mixed audio and video data stream to the second client so that the second client can display the mixed audio and video data stream.

[0014] Accordingly, this application provides a computer device, including: a memory, a processor, and a network interface. The processor is connected to the memory and the network interface. The network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the method in this application embodiment.

[0015] Accordingly, embodiments of this application provide a computer-readable storage medium, including: a computer program stored in the computer-readable storage medium, which, when executed by a processor, implements the method in the embodiments of this application.

[0016] Accordingly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads and executes the computer instructions from the computer-readable storage medium, causing the computer device to perform the method described in this application.

[0017] By implementing the embodiments of this application, audio and video data from multiple first clients can be obtained through the smart network card of the media server. The mixing processing module of the smart network card is then invoked to perform mixing processing on the audio and video data from the multiple first clients, resulting in a mixed audio and video data stream. This mixed audio and video data stream is then pushed to a second client so that the second client can display the mixed audio and video data stream. In this way, the audio and video data mixing processing of the media service can be shifted from the CPU to the mixing processing module of the smart network card, thereby reducing the CPU load, improving the speed of the media server's response to the mixing service, and providing a better user experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0021] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0022] Figure 4a This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0023] Figure 4b This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0024] Figure 4c This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] When a media server mixes audio and video data sent from multiple clients, it typically processes the data using its CPU before pushing the mixed audio and video streams to each client. This method of audio and video data mixing generally has three characteristics: 1. The media server requires significant computing resources to process the audio and video data; 2. Clients send large amounts of audio and video data; 3. The media server's mixing service is both CPU-intensive and I / O (Input / Output)-intensive. This leads to two consequences: 1. The media server's CPU resources are under high workload due to processing audio and video data from multiple clients simultaneously, easily causing slow response times; 2. The media server's network card needs to continuously write received audio and video data into memory for CPU processing, requiring multiple memory data copy operations, which affects the response speed of the mixing service.

[0029] Based on this, embodiments of this application provide a data processing method, apparatus, device, and storage medium. This method can be applied to a smart network interface card (NIC) of a media server. The smart NIC may include an audio / video filtering engine and a mixing processing module. By implementing embodiments of this application, network data from multiple first clients can be acquired through the smart NIC. The audio / video filtering engine of the smart NIC is then invoked to filter the acquired network data, resulting in audio / video data from multiple first clients. The intelligent mixing processing module is then invoked to mix the audio / video data from multiple first clients, resulting in a mixed audio / video data stream. After pushing this mixed audio / video data stream to a second client, it can be displayed. Thus, the business logic of audio / video data mixing is implemented using the smart NIC of the media server, freeing up CPU resources and improving the processing performance of the media server.

[0030] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application. As shown in the figure, the architecture of this data processing system may include a terminal 10, a media server 12 configured with a smart network card 11, and a terminal 13. The terminal 10 and the terminal 13 communicate with the media server through a network. The terminal may be a personal computer, a laptop computer, a smartphone, a tablet computer, a smartwatch, a smart voice interaction device, a smart home appliance, an in-vehicle terminal, or a smart wearable device, but is not limited to these. The media server may be a single server, a server cluster consisting of multiple servers, or a cloud server providing cloud computing services.

[0031] In this embodiment, terminal 10 may be equipped with a first client, and terminal 13 may be equipped with a second client. The first client and the second client may be the same service client on different terminals, or the same service client on the same terminal. Terminal 10 and terminal 13 may be the same terminal or different terminals. Media server 12 can respond to multiple first client's mixed streaming service requests. Smart network card 11 includes an audio / video filtering engine and a mixed streaming processing module. The audio / video filtering engine can filter the network data of multiple first clients to obtain audio / video data of multiple first clients. The mixed streaming processing module can perform mixed streaming processing on the audio / video data of multiple clients to obtain a mixed audio / video data stream. The obtained mixed audio / video data stream can be pushed to the second client for display. The smart network card 11 can interact with the target storage space through a direct data access method, which may be a direct memory access (DMA) method or a remote direct memory access (RDMA) method.

[0032] The method provided in this application can be applied to most media servers related to audio and video services, and can respond quickly even in scenarios with a large number of client audio and video data streams. As long as the media server meets the interface and configuration requirements of the smart network card, it can be adapted to the method described in this application to provide mixed streaming services for client audio and video data. Furthermore, the method provided in this application can also be applied to cloud conferencing service environments. Cloud conferencing is an efficient, convenient, and low-cost conferencing format based on cloud computing technology. Users can operate through an internet interface to quickly and efficiently share audio and video data synchronously with teams and clients, thereby improving the stability, security, and availability of meetings.

[0033] It should be noted that the server involved in this application can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0034] For further details, please see Figure 2, Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method can be executed by a smart network interface card (NIC) of a media server, which includes an audio / video filtering engine and a mixing module, such as... Figure 2 As shown, the method may include, but is not limited to, the following steps.

[0035] S201: Obtain network data from multiple first clients.

[0036] In this embodiment of the application, the smart network card can receive network data sent by multiple first clients.

[0037] In one embodiment, the smart network interface card (NIC) can be equipped with an FPGA (Field-Programmable Gate Array), and the NIC can receive network data from multiple first clients via an FPGA-based Ethernet MAC (Media Access Control) protocol.

[0038] S202: Call the audio and video filtering engine to filter the network data of each of the multiple first clients to obtain the audio and video data of each first client.

[0039] In this embodiment of the application, the smart network card can call the audio and video filtering engine to filter the network data of each of the multiple first clients to obtain the audio and video data of each first client that can be processed.

[0040] In one embodiment, the audio and video filtering engine of the smart network card can unpack and parse the network data of each first client, retain the data containing audio and video signaling, and correspondingly filter out other network data that does not contain audio and video signaling, thereby obtaining the audio and video data of each first client.

[0041] In one embodiment, after obtaining the audio and video data of each first client, the smart network interface card (NIC) can acquire the service identifier carried in the audio and video data of each first client. Based on the service identifier, it can determine whether there is a correlation between the services corresponding to the audio and video data of multiple first clients. Each first client's audio and video data carries its own service identifier, which is used to identify the service corresponding to the audio and video data. For example, the service identifier may include information about the service platform to which the audio and video data belongs, or it may include information about the specific service name to which the audio and video data belongs.

[0042] In one embodiment, the smart network interface card (NIC) can determine whether there is a correlation between the services corresponding to the audio and video data of multiple first clients based on preset service association rules. Specifically, audio and video data from multiple first clients belonging to the same service platform can be determined to be related audio and video data. For example, when multiple first clients belong to the same live streaming platform, assuming two users, A1 and A2, use the first client for live streaming interaction, the audio and video data of the first client corresponding to user A1 can be the audio and video data generated by user A1 as the host during the live stream, and the audio and video data of the first client corresponding to user A2 can be the audio and video data generated by user A2 as the viewer interacting with host A1. The audio and video data of the first client corresponding to user A1 and the audio and video data of the first client corresponding to user A2 are thus related audio and video data. Correspondingly, the smart NIC can also pre-store a service information table containing service association rules, and can periodically update the service information table as needed.

[0043] In one embodiment, when the smart network interface card (NIC) determines that some of the audio and video data corresponding to multiple first clients are related, it can prioritize processing the related audio and video data of the first clients, and then continue to determine whether the remaining audio and video data corresponding to the first clients are related. This process is repeated until all the unprocessed audio and video data corresponding to the first clients are unrelated. At this point, the unrelated audio and video data corresponding to the first clients can be discarded, and an error message indicating that the discarded audio and video data of the first clients is unrelated can be reported.

[0044] S203: Call the mixing module to perform mixing processing on the audio and video data of multiple first clients to obtain a mixed audio and video data stream.

[0045] In this embodiment of the application, the smart network card can call the mixing processing module to perform mixing processing on the audio and video data of multiple first clients associated with the service, so as to obtain a mixed audio and video data stream.

[0046] In one embodiment, the smart network interface card (NIC) can invoke a mixing processing module and a direct storage method between the NIC and the target storage space to mix audio and video data from multiple first clients to obtain a mixed audio and video data stream. The target storage space can be the memory of the media server or the memory of the smart NIC. The direct storage method between the smart NIC and the target storage space refers to a method where the smart NIC and the target storage space directly interact with each other; that is, the smart NIC does not need to interact with the CPU. The direct storage method can include direct data access methods, which can be direct memory access methods or remote direct data access methods. Using this direct storage method can reduce CPU resource consumption, thereby reducing the workload of the media server.

[0047] In one embodiment, the mixing module of the smart network interface card (NIC) may include a decoding unit, a mixing unit, and an encoding unit. The smart NIC can call the decoding unit in the mixing module to decode the audio and video data of multiple first clients, obtaining decoded audio and video data for each first client. Then, it can call the mixing unit in the mixing module to perform mixing processing on the decoded audio and video data from the multiple first clients. Finally, it can call the encoding unit in the mixing module to complete the mixing encoding, thereby obtaining a mixed audio and video data stream.

[0048] S204: Send a mixed audio and video data stream to the second client so that the second client can display the mixed audio and video data stream.

[0049] In this embodiment of the application, the smart network card can send a mixed audio and video data stream to a second client, and the second client can display the mixed audio and video data stream to the user.

[0050] In one embodiment, the second client and the first client can be clients corresponding to the same user, or they can be clients corresponding to different users. When the second client and the first client are clients corresponding to the same user, the application scenario of this application can be that user A watches video footage of himself and others through the second client. When the second client and the first client are clients corresponding to different users, the application scenario of this application can be that user A watches video footage of user B and user C interacting live through the first client through the second client.

[0051] By implementing the embodiments of this application, the smart network card of the media server can be used to realize the mixed streaming processing of audio and video data of multiple first clients and the pushing of second clients, which frees up most of the CPU resources. Even when the audio and video data increases, the pushing speed will not slow down, thus improving the experience of audio and video services.

[0052] For further details, please see Figure 3 , Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. This method can be executed by a smart network interface card (NIC) of a media server, which includes an audio / video filtering engine and a mixing module, such as... Figure 3 As shown, the method may include, but is not limited to, the following steps.

[0053] S301: Obtain network data from multiple first clients.

[0054] S302: Call the audio and video filtering engine to filter the network data of each of the multiple first clients to obtain the audio and video data of each first client.

[0055] In this embodiment of the application, the execution processes of steps S301 and S302 can be found in the following references: Figure 2 The specific descriptions of steps S201 and S202 are not repeated here.

[0056] S303: Call the mixing processing module to decode the audio and video data of each of the multiple first clients, and obtain the decoded audio and video data of each first client.

[0057] In this embodiment of the application, the smart network card can call the mixing processing module to decode the audio and video data of each first client, so that the decoded audio and video data of each first client can be processed.

[0058] S304: The decoded audio and video data of each first client is stored in the target storage space through the direct data access method.

[0059] In this embodiment, the smart network interface card (NIC) can store the decoded audio and video data of each first client to a target storage space using a direct data access (DDATA) method. This DDATA method can be either a direct memory access (DMI) method or a remote direct data access (RDA) method. The DMI method directly copies data from one address space to another, allowing communication between hardware devices of different speeds without relying on a large CPU interrupt load. The RDA method addresses the latency in server-side data processing during network transmission. It quickly moves data from one system to a remote system's memory, eliminating the overhead of external memory copying and thus freeing up memory bandwidth and CPU cycles.

[0060] In one embodiment, the target storage space can be the memory of the media server, the memory of the smart network interface card (NIC), or a combination of both. When the target storage space is the memory of the media server, the smart NIC can store the decoded audio and video data of each first client using the remote direct data access method, enabling fast read and write operations. When the target storage space is the memory of the smart NIC, the smart NIC can store the decoded audio and video data of each first client in memory using the direct memory access method, thus improving processing efficiency. When the target storage space is a combination of the memory of the media server and the memory of the smart NIC, the smart NIC can set a memory space threshold. When the remaining storage space in the smart NIC's memory is less than or equal to the memory space threshold, the decoded audio and video data is stored in the memory of the media server. In this way, data can be read directly using the smart NIC, reducing frequent copying between the media server's CPU and memory, and improving processing efficiency.

[0061] S305: Call the mixing processing module to obtain the texture feature data corresponding to the decoded audio and video data of each first client from the target storage space.

[0062] In this embodiment, the smart network card can invoke the mixing processing module to obtain texture feature data corresponding to the audio and video data of each of the multiple decoded first clients from the target storage space. The texture feature data may, for example, be the structural information of the texture.

[0063] S306: Based on the texture feature data corresponding to the audio and video data of each first client after decoding, call the mixing processing module to perform mixing encoding on the audio and video data of multiple first clients to obtain a mixed audio and video data stream.

[0064] In this embodiment of the application, the smart network card can call the mixing processing module to perform mixing encoding on the audio and video data of the multiple first clients based on the texture feature data corresponding to the audio and video data of each of the multiple first clients after decoding, so as to obtain the final mixed audio and video data stream.

[0065] In one embodiment, the application scenario of this application can be a multi-person video conferencing scenario. Assuming there are three participants, B1, B2, and B3, with B1 as the main speaker, when the mixing processing module of the smart network card is called to perform mixing encoding on the texture feature data corresponding to the audio and video data of the first client after decoding by the three participants, the texture feature data corresponding to the main speaker B1 is placed as the background at the bottom layer. Then, the texture feature data corresponding to B2 and B3 are overlaid on the texture feature data corresponding to B1. Then, all the texture feature data are scaled and coordinate calculated pixel by pixel, and finally, color fusion is performed to complete the mixing encoding of audio and video data of multiple first clients, resulting in a mixed audio and video data stream.

[0066] S307: Send a mixed audio and video data stream to the second client so that the second client can display the mixed audio and video data stream.

[0067] In this embodiment, the smart network interface card (NIC) can send a mixed audio and video data stream to a client, and the second client can display the mixed audio and video data to the user. The second client can be a video playback client or a web browser client.

[0068] By implementing the embodiments of this application, the smart network interface card (NIC) can store the decoded audio and video data of each first client in the target storage space, and call the decoding module to obtain the texture feature data corresponding to the decoded audio and video data from the target storage space. Then, it performs mixing encoding based on the texture feature data, and finally sends the resulting mixed audio and video data stream downstream to the second client. In this way, data interaction between the CPU and the smart NIC is no longer required. Compared to the method of performing mixing processing by the CPU and then writing the results into memory, this method offers better flexibility and scalability, and also improves the mixing and downstream streaming speed of the entire media server.

[0069] For further details, please see Figure 4a , Figure 4a This is a flowchart illustrating another data processing method provided in an embodiment of this application. This method can be executed by a smart network interface card (NIC) of a media server, which includes an audio / video filtering engine and a mixing module, such as... Figure 4a As shown, the method may include, but is not limited to, the following steps.

[0070] S401: Obtain network data from multiple first clients.

[0071] S402: Call the audio and video filtering engine to filter the network data of each of the multiple first clients to obtain the audio and video data of each first client.

[0072] S403: Call the mixing processing module to decode the audio and video data of each of the multiple first clients, and obtain the decoded audio and video data of each first client.

[0073] In this embodiment, the execution processes of steps S401, S402, and S403 can be found in the following references: Figure 2 Steps S201, S202 and Figure 3 The specific description of step S303 will not be repeated here.

[0074] S404: Create a first data structure, a second data structure, and a decoding buffer corresponding to the second data structure for each first client in the target storage space.

[0075] In one embodiment, the smart network interface card can create a first data structure and a second data structure corresponding to each first client, as well as a decoding buffer corresponding to the second data structure, in the target storage space.

[0076] In one embodiment, the smart network interface card (NIC) can create a first data structure, such as a Texture data structure, for each first client in the memory of the media server using a remote direct data access method, and create a second data structure corresponding to the first data structure. For example, the second data structure can be a SurfaceTexture data structure corresponding to the Texture data structure. Correspondingly, the smart NIC can also create a decoding buffer corresponding to the second data structure in the memory of the media server. For example, the smart NIC can allocate a Surface memory block in the memory of the media server as a decoding buffer for the SurfaceTexture.

[0077] S405: Synchronize the decoded audio and video data of each first client to the decoding buffer according to the data storage method of the second data structure.

[0078] In this application, the smart network card can synchronize the decoded audio and video data of each first client to the decoding buffer according to the data storage method of the second data structure.

[0079] In one embodiment, the audio and video data corresponding to each first client can be synchronized to the decoding buffer according to the storage method of the second data structure after each frame of audio and video data is decoded, or the audio and video data corresponding to each first client can be synchronized to the decoding buffer according to the storage method of the second data structure after all the audio and video data is decoded.

[0080] S406: Synchronize the decoded audio and video data of each first client from the decoding buffer to the target storage space according to the data storage method of the first data structure.

[0081] In this embodiment of the application, the smart network card can synchronize the decoded audio and video data of each first client from the decoding buffer to the target storage space according to the data storage method of the first data structure.

[0082] In one embodiment, when the decoded audio and video data of each first client is completely stored in the decoding buffer, the smart network card can receive a signal that it has been successfully synchronized to the decoding buffer, and then synchronize the decoded audio and video data of each first client to the target storage space according to the first data structure (such as the Texture data structure).

[0083] S407: Extract the texture feature data corresponding to the decoded audio and video data of each first client from the target storage space according to the first data structure.

[0084] In this embodiment, the smart network interface card (NIC) can extract texture feature data corresponding to the decoded audio and video data of each first client from the target storage space according to a first data structure. The first data structure may include the texture feature data of the decoded audio and video data of each first client.

[0085] S408: Call the mixing processing module to perform pixel-by-pixel scaling and coordinate calculations on the texture feature data corresponding to the decoded audio and video data of each first client.

[0086] In this embodiment of the application, the smart network card can call the mixing processing module to perform pixel-by-pixel scaling and coordinate calculation on the texture feature data corresponding to the decoded audio and video data of each first client.

[0087] In one embodiment, the smart network interface card (NIC) can perform a fusion operation on the texture feature data corresponding to the audio and video data of each first client. Specifically, the smart NIC can place the texture feature data corresponding to the main video frame as the background at the bottom layer, then overlay the texture feature data corresponding to the main video frame with the texture feature data corresponding to other video frames, and then perform pixel-by-pixel scaling and coordinate calculations. For example, the main video frame could be the video frame of the host opening a live streaming room, and the other video frames could be video frames of other hosts interacting with that host.

[0088] S409: Call the mixing processing module to perform color fusion processing on the texture feature data corresponding to the audio and video data of each first client after pixel-by-pixel scaling and coordinate calculation, and obtain a mixed audio and video data stream.

[0089] In this embodiment of the application, after scaling and coordinate calculation of the texture feature data corresponding to the audio and video data of each first client pixel by pixel, the smart network card can call the mixing processing module to perform color fusion processing on the texture feature data corresponding to the audio and video data of each first client, thus completing the mixing and obtaining a mixed audio and video data stream.

[0090] S410: Send a mixed audio and video data stream to the second client so that the second client can display the mixed audio and video data stream.

[0091] In this embodiment of the application, the execution process of step S410 can be found in [reference needed]. Figure 2 In step S204 and Figure 3 The specific description of step S307 will not be repeated here.

[0092] In one embodiment, such as Figure 4b As shown, Figure 4b This is a flowchart illustrating another data processing method provided in this application embodiment. The method can be executed by the smart network card of the media server. The smart network card includes an audio and video filtering engine and a mixing processing module. The method may include, but is not limited to, the following steps.

[0093] S411: The smart network card can call the audio and video filtering engine to obtain audio and video data uploaded by multiple first clients.

[0094] S412: The smart network card calls the mixing processing module to unpack and parse the audio and video data uploaded by multiple first clients. This mixing processing module can be an FPGA-based mixing processing module.

[0095] S413: The smart network card can use FPGA programming capabilities to perform mixed-stream calculations on the audio and video data of multiple first clients after unpacking and parsing, and obtain a mixed audio and video data stream.

[0096] S414: The smart network card can transmit the mixed audio and video data stream to a second client so that the second client can display the mixed audio and video data stream.

[0097] In one embodiment, such as Figure 4c As shown, Figure 4c This is a flowchart illustrating another data processing method provided in this application embodiment. The method can be executed by the smart network card of the media server. The smart network card includes an audio and video filtering engine and a mixing processing module. The method may include, but is not limited to, the following steps.

[0098] S421: The smart network card can detect data events, that is, when the first client sends data to the multimedia service, the smart network card can respond.

[0099] S422: The smart network card can receive network data from multiple first clients via Ethernet MAC.

[0100] S423: The smart network card can call the audio and video filtering engine to filter the network data received from multiple first clients, and obtain audio and video data of multiple first clients containing audio and video signaling.

[0101] S424: After the smart network card obtains audio and video data from multiple first clients, it can perform mixed streaming processing on the audio and video data from multiple first clients. The smart network card can call the mixed streaming processing module to obtain the audio and video data from multiple first clients from the buffer memory area of ​​the smart network card, perform mixed streaming calculation, and obtain a mixed audio and video data stream.

[0102] S425: The smart network card can transmit the mixed audio and video data stream obtained from the mixing process to the second client via Ethernet MAC, so that the second client can display the mixed audio and video data stream.

[0103] By implementing the embodiments of this application, the mixing and processing of audio and video data from multiple first clients can be performed by the mixing and processing module of the smart network card instead of the CPU of the media server. As a result, most of the CPU resources are released, greatly reducing the CPU workload of the media server, thereby speeding up the streaming speed and improving the user experience.

[0104] For further details, please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. In one embodiment, such as... Figure 5 As shown, the data processing device 50 may include:

[0105] The receiving unit 501 is used to acquire network data from multiple first clients.

[0106] The processing unit 502 is used to call the audio and video filtering engine to filter the network data of each of the multiple first clients to obtain the audio and video data of each first client; and to call the mixing processing module to mix the audio and video data of the multiple first clients to obtain a mixed audio and video data stream.

[0107] The sending unit 503 is used to send a mixed audio and video data stream to the second client so that the second client can display the mixed audio and video data stream.

[0108] In one embodiment, the processing unit 502 can also be used to obtain the service identifier carried by the audio and video data of each first client; determine whether there is a correlation between the services corresponding to the audio and video data of multiple first clients based on the service identifier; if there is a correlation, execute the step of calling the mixing processing module to perform mixing processing on the audio and video data of multiple first clients to obtain a mixed audio and video data stream.

[0109] In one embodiment, the processing unit 502 can also be used to call the mixing processing module and the direct storage method between the smart network card and the target storage space to perform mixing processing on the audio and video data of multiple first clients to obtain a mixed audio and video data stream.

[0110] In one embodiment, the processing unit 502 can also be used to call the mixing processing module to decode the audio and video data of each of the multiple first clients to obtain the decoded audio and video data of each first client; store the decoded audio and video data of each first client in the target storage space through the direct data access method; call the mixing processing module to obtain the texture feature data corresponding to the decoded audio and video data of each first client from the target storage space; and call the mixing processing module to perform mixing encoding on the audio and video data of the multiple first clients according to the texture feature data corresponding to the decoded audio and video data of each first client to obtain a mixed audio and video data stream.

[0111] In one embodiment, the processing unit 502 can also be used to create a first data structure, a second data structure, and a decoding buffer corresponding to the second data structure for each first client in the target storage space; synchronize the decoded audio and video data of each first client to the decoding buffer according to the data storage method of the second data structure; and synchronize the decoded audio and video data of each first client from the decoding buffer to the target storage space according to the data storage method of the first data structure. The first data structure includes the texture feature data of the decoded audio and video data of each first client.

[0112] In one embodiment, the processing unit 502 can also be used to extract texture feature data corresponding to the decoded audio and video data of each first client from the target storage space according to the first data structure.

[0113] In one embodiment, the processing unit 502 can also be used to call the mixing processing module to perform pixel-by-pixel scaling and coordinate calculation on the texture feature data corresponding to the audio and video data of each first client after decoding; and to call the mixing processing module to perform color fusion processing on the texture feature data corresponding to the audio and video data of each first client after pixel-by-pixel scaling and coordinate calculation to obtain a mixed audio and video data stream.

[0114] It should be noted that, Figure 5 For details not mentioned in the corresponding embodiments and the specific implementation methods of each step, please refer to [link to relevant documentation]. Figures 2-4c The illustrated embodiments and the foregoing content will not be repeated here.

[0115] For further details, please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may include a network interface 601, a memory 602, and a processor 603. The network interface 601, memory 602, and processor 603 are connected via one or more communication buses, which are used to enable communication between these components. The network interface 601 may include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 602 may include volatile memory, such as random-access memory (RAM); the memory 602 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 602 may also include a combination of the above types of memory. The processor 603 may be a central processing unit (CPU). The processor 603 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The aforementioned PLD can be a field-programmable gate array (FPGA), a generic array logic (GAL), etc.

[0116] Optionally, the memory 602 is also used to store program instructions, which the processor 603 can also call to perform the following: acquire network data from multiple first clients; call the audio and video filtering engine to filter the network data of each of the multiple first clients to obtain audio and video data of each first client; call the mixing processing module to mix the audio and video data of the multiple first clients to obtain a mixed audio and video data stream; and send the mixed audio and video data stream to a second client so that the second client can display the mixed audio and video data stream.

[0117] In one embodiment, the memory 602 is further used to store program instructions, and the processor 603 can also call the program instructions to implement: obtaining the service identifier carried by the audio and video data of each first client; determining whether there is a correlation between the services corresponding to the audio and video data of multiple first clients based on the service identifier; if there is a correlation, executing the step of calling the mixing processing module to perform mixing processing on the audio and video data of multiple first clients to obtain a mixed audio and video data stream.

[0118] In one embodiment, the memory 602 is also used to store program instructions, and the processor 603 can also call the program instructions to: call the mixing processing module and the direct storage method between the smart network card and the target storage space to perform mixing processing on the audio and video data of multiple first clients to obtain a mixed audio and video data stream.

[0119] In one embodiment, the memory 602 is further used to store program instructions, which the processor 603 can also call to perform the following: calling the mixing processing module to decode the audio and video data of each of the multiple first clients to obtain the decoded audio and video data of each first client; storing the decoded audio and video data of each first client to a target storage space using a direct data access method; calling the mixing processing module to obtain the texture feature data corresponding to the decoded audio and video data of each first client from the target storage space; and, based on the texture feature data corresponding to the decoded audio and video data of each first client, calling the mixing processing module to perform mixing encoding on the audio and video data of the multiple first clients to obtain a mixed audio and video data stream.

[0120] In one embodiment, the memory 602 is further used to store program instructions, which the processor 603 can also call to perform the following: creating a first data structure, a second data structure, and a decoding buffer corresponding to the second data structure for each first client in the target storage space; synchronizing the decoded audio and video data of each first client to the decoding buffer according to the data storage method of the second data structure; and synchronizing the decoded audio and video data of each first client from the decoding buffer to the target storage space according to the data storage method of the first data structure, wherein the first data structure includes texture feature data of the decoded audio and video data of each first client.

[0121] In one embodiment, the memory 602 is also used to store program instructions, which the processor 603 can also call to implement: extracting texture feature data corresponding to the decoded audio and video data of each first client from the target storage space according to the first data structure.

[0122] In one embodiment, the memory 602 is further used to store program instructions, and the processor 603 can also call the program instructions to: call the mixing processing module to perform pixel-by-pixel scaling and coordinate calculation on the texture feature data corresponding to the decoded audio and video data of each first client; call the mixing processing module to perform color fusion processing on the texture feature data corresponding to the audio and video data of each first client after pixel-by-pixel scaling and coordinate calculation, so as to obtain a mixed audio and video data stream.

[0123] It should be understood that the principles and beneficial effects of the computer device 60 in solving the problem described in the embodiments of this application are the same as those in this application. Figures 2-4c The embodiments shown and the foregoing content have similar principles and beneficial effects in solving the problem, and will not be repeated here for the sake of brevity.

[0124] In addition, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in the foregoing embodiments.

[0125] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the foregoing embodiments.

[0126] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0127] The units in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0129] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments, and equivalent changes made in accordance with the claims of this application, still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, A smart network interface card (NIC) for use in a media server, the smart NIC including an audio / video filtering engine and a mixing module, the method comprising: Obtain network data from multiple first clients; The audio and video filtering engine is invoked to filter the network data of each of the plurality of first clients to obtain the audio and video data of each first client; The mixing processing module is invoked to perform mixing processing on the audio and video data of the multiple first clients to obtain a mixed audio and video data stream; The mixed audio and video data stream is sent to the second client so that the second client can display the mixed audio and video data stream; The step of calling the mixing processing module to perform mixing processing on the audio and video data of the plurality of first clients to obtain a mixed audio and video data stream includes: The mixing processing module is invoked to decode the audio and video data of each of the plurality of first clients to obtain the decoded audio and video data of each first client; The decoded audio and video data of each first client is stored in the target storage space using a direct data access method; The mixing processing module is invoked to obtain the texture feature data corresponding to the decoded audio and video data of each first client from the target storage space; based on the texture feature data corresponding to the decoded audio and video data of each first client, the mixing processing module is invoked to perform mixing encoding on the audio and video data of the multiple first clients to obtain a mixed audio and video data stream.

2. The method according to claim 1, characterized in that, Before invoking the mixing processing module to perform mixing processing on the audio and video data of the plurality of first clients to obtain a mixed audio and video data stream, the method further includes: Obtain the service identifier carried in the audio and video data of each first client; Determine whether there is a correlation between the services corresponding to the audio and video data of the multiple first clients based on the service identifier; If a correlation exists, the step of calling the mixing processing module to perform mixing processing on the audio and video data of the multiple first clients to obtain a mixed audio and video data stream is executed.

3. The method according to claim 1, characterized in that, The step of storing the decoded audio and video data of each first client into the target storage space via the direct data access method includes: In the target storage space, create a first data structure, a second data structure, and a decoding buffer corresponding to the second data structure for each first client; The decoded audio and video data of each first client is synchronized to the decoding buffer according to the data storage method of the second data structure; According to the data storage method of the first data structure, the decoded audio and video data of each first client is synchronized from the decoding buffer to the target storage space. The first data structure includes the texture feature data of the decoded audio and video data of each first client.

4. The method according to claim 3, characterized in that, The step of calling the mixing processing module to obtain the texture feature data corresponding to the decoded audio and video data of each first client from the target storage space includes: Based on the first data structure, extract the texture feature data corresponding to the decoded audio and video data of each first client from the target storage space.

5. The method according to any one of claims 1, 3, and 4, characterized in that, The step of calling the mixing processing module to perform mixing encoding on the audio and video data of the multiple first clients based on the texture feature data corresponding to the decoded audio and video data of each first client, to obtain a mixed audio and video data stream, includes: The mixing processing module is invoked to perform pixel-by-pixel scaling and coordinate calculations on the texture feature data corresponding to the decoded audio and video data of each first client. The mixing processing module is invoked to perform color fusion processing on the texture feature data corresponding to the audio and video data of each first client after pixel-by-pixel scaling and coordinate calculation, so as to obtain the mixed audio and video data stream.

6. A data processing apparatus, characterized in that, include: A receiving unit is used to acquire network data from multiple first clients; The processing unit is used to call the audio and video filtering engine to filter the network data of each of the plurality of first clients to obtain the audio and video data of each first client. The mixing processing module is invoked to perform mixing processing on the audio and video data of the multiple first clients to obtain a mixed audio and video data stream; The sending unit is configured to send the mixed audio and video data stream to the second client so that the second client can display the mixed audio and video data stream; Specifically, the processing unit is used to call the mixing processing module to decode the audio and video data of each of the plurality of first clients to obtain the decoded audio and video data of each first client; and to store the decoded audio and video data of each first client to the target storage space through a direct data access method. The mixing processing module is invoked to obtain the texture feature data corresponding to the decoded audio and video data of each first client from the target storage space; based on the texture feature data corresponding to the decoded audio and video data of each first client, the mixing processing module is invoked to perform mixing encoding on the audio and video data of the multiple first clients to obtain a mixed audio and video data stream.

7. A computer device, characterized in that, The device includes a memory, a processor, and a network interface. The processor is connected to the memory and the network interface. The network interface is used to provide network communication functions. The memory is used to store program code. The processor is used to call the program code to execute the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions that, when executed by a processor, implement the method of any one of claims 1-5.

Images