Multi-path data flow mixing method, device and equipment and storage medium

Abstract

The invention discloses a multipath data flow mixing method and device, equipment and a storage medium. The method comprises the following steps: acquiring at least two paths of original data materials to be subjected to stream mixing processing; and performing frame rate conversion processing on each path of original data material to obtain at least two paths of to-be-mixed stream data. And on the basis of a preset image merging specification, merging image frames in the at least two paths of to-be-mixed data to obtain a merged data image. And based on the image frame number corresponding to the merged data image, performing integration processing on the merged data image to obtain the target mixed stream data so as to support direct mixed stream of online resources and control the number of playing times, so that a user can perform flexible picture scheduling conveniently, and the usability, flexibility and effect of mixed stream are improved.

Description

Technical Field

[0001] The present invention relates to the field of media processing technology, and in particular to a multi-channel data mixing method, device, equipment and storage medium. Background Art

[0002] In the live broadcast business, requirements such as picture-in-picture and screen switching are generally achieved through audio and video mixing technology.

[0003] The online audio and video mixing products currently provided in the industry are used to merge multiple input streams and push the merged results back to the output stream. When users want to mix online videos, online pictures and other formats of materials into the screen, they need to convert these materials into audio and video streams first, which is very cumbersome. Since the audio and video streams are processed, the number of playbacks cannot be controlled. When the material is played, the user needs to call the mixing service again to adjust the screen.

[0004] Currently, the mixing products on the market do not support mixing of transparent materials with alpha channels. Some products support transparent processing of non-transparent materials through green screen cutouts, but there will be green residue on the edges of the processed images, and the mixing effect is not good. Summary of the invention

[0005] The present invention provides a multi-channel data mixing method, device, equipment and storage medium to support direct mixing of online resources and control the number of playback times, facilitate users to perform flexible picture scheduling, and improve the ease of use, flexibility and effect of mixing.

[0006] According to one aspect of the present invention, a method for mixing multiple data streams is provided. The method comprises:

[0007] Obtain at least two channels of original data materials to be processed by mixed streams;

[0008] Perform frame rate conversion on each channel of the original data material to obtain at least two channels of data to be mixed;

[0009] Based on a preset image merging specification, merging the image frames in the at least two channels of the to-be-mixed stream data to obtain a merged data image;

[0010] Based on the image frame number corresponding to the merged data image, the merged data image is integrated to obtain target mixed stream data.

[0011] According to another aspect of the present invention, a multi-channel data mixing device is provided. The device comprises:

[0012] The original material acquisition module is used to acquire at least two channels of original data materials to be processed by mixed streams;

[0013] A module for determining data to be mixed, used for performing frame rate conversion processing on each channel of the original data material to obtain at least two channels of data to be mixed;

[0014] A merged image determination module, configured to merge the image frames in the at least two channels of the to-be-mixed stream data based on a preset image merging specification to obtain a merged data image;

[0015] The mixed stream data determination module is used to integrate the merged data image based on the image frame number corresponding to the merged data image to obtain target mixed stream data.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-channel data mixing method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-channel data mixing method described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention is to obtain at least two channels of original data materials to be mixed. Each channel of the original data materials is subjected to frame rate conversion processing to obtain at least two channels of data to be mixed. Based on a preset image merging specification, the image frames in the at least two channels of data to be mixed are merged to obtain a merged data image. Based on the image frame number corresponding to the merged data image, the merged data image is integrated to obtain the target mixed data, which effectively solves the existing problems of audio and video mixing technology in live broadcast scenarios, thereby enabling direct mixing of online resources and controlling the number of playbacks, facilitating users to perform flexible picture scheduling, and improving the ease of use, flexibility and effect of mixing.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a multi-channel data mixing method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of a multi-channel data mixing method provided according to Embodiment 2 of the present invention;

[0026] Figure 3 is a schematic diagram of a multi-channel data mixing method provided according to Embodiment 3 of the present invention;

[0027] Figure 4 is a structural diagram of a multi-channel data mixing device provided according to a fourth embodiment of the present invention;

[0028] Figure 5 It is a structural schematic diagram of an electronic device for implementing the multi-channel data mixing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1This is a flowchart of a multi-channel data mixing method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where multiple online resources are directly mixed. The method can be executed by a multi-channel data mixing device. The multi-channel data mixing device can be implemented in the form of hardware and / or software. The multi-channel data mixing device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S101: Obtain at least two channels of original data materials to be mixed.

[0034] This technical solution aims to solve the existing problems of audio and video mixing technology in live broadcast scenarios, especially the cumbersome operation of mixing online pictures, videos and other materials, the lack of support for mixing alpha channel transparent materials, and poor mixing effects. It allows multiple audio and video streams from different sources (such as cameras, screen recordings, etc.) to be merged into a single output stream, which can enhance the user experience.

[0035] The original data materials may be multiple local materials and / or online materials, which may include videos, pictures, audios, and the like.

[0036] Specifically, multiple channels of original data materials (eg, different video sources) to be mixed are received. It should be noted that the present invention can support mixing of at least two channels of original data materials.

[0037] S102: Perform frame rate conversion processing on each channel of the original data material to obtain at least two channels of data to be mixed.

[0038] The data to be mixed may refer to multiple channels of original data materials with the same frame rate.

[0039] Specifically, each of the original data materials can be subjected to frame rate conversion processing to obtain at least two channels of data to be mixed. Exemplarily, for picture materials, they are downloaded and converted into MP4 format for subsequent processing. For video materials with non-preset frame rates (e.g., 25 frames per second), they are also downloaded first, then the frame rate is adjusted to the preset frame rate, and converted into MP4 format to obtain the data to be mixed.

[0040] It should be noted that before performing the frame rate conversion process, you can also perform an input parameter integrity check, and verify whether the provided resources are valid and available by checking the file header or trying to pull the stream to ensure that all necessary parameters are provided and in the correct format.

[0041] S103: Based on a preset image merging specification, merge the image frames in the at least two channels of the to-be-mixed stream data to obtain a merged data image.

[0042] The preset image merging specification may refer to the merging requirement of image frames, or may refer to the display screen requirement of mixed stream data, such as screen size and screen layout.

[0043] Specifically, each channel of mixed stream data is decoded to obtain an image frame of each channel of mixed stream data, and then the image frame is scaled, laid out, image fused, etc. according to a preset image merging specification to obtain a merged data image.

[0044] S104: Based on the image frame number corresponding to the merged data image, integrate the merged data image to obtain target mixed stream data.

[0045] The target mixed stream data may refer to mixed stream data obtained through multi-channel mixed streams.

[0046] Specifically, according to the image frame number of the merged data image, the merged data image is subjected to integration alignment and pixel alignment and other processing, so as to obtain target mixed stream data.

[0047] The technical solution of the embodiment of the present invention is to obtain at least two channels of original data materials to be mixed. Each channel of the original data materials is subjected to frame rate conversion processing to obtain at least two channels of data to be mixed. Based on a preset image merging specification, the image frames in the at least two channels of data to be mixed are merged to obtain a merged data image. Based on the image frame number corresponding to the merged data image, the merged data image is integrated to obtain the target mixed data, which effectively solves the existing problems of audio and video mixing technology in live broadcast scenarios, thereby enabling direct mixing of online resources and controlling the number of playbacks, facilitating users to perform flexible picture scheduling, and improving the ease of use, flexibility and effect of mixing.

[0048] Exemplarily, after obtaining the target mixed stream data, the method further includes: encoding the target mixed stream data to obtain a target mixed stream video, and pushing the target mixed stream video to a preset online channel for display processing.

[0049] Specifically, the output stream encoding module encodes the fused target mixed stream data into a pushable stream format to obtain the target mixed stream video for subsequent data push. The encoded target mixed stream video is pushed to the preset online channel so that the preset online channel performs display processing. Exemplarily, when there is a problem with the push stream communication, a disconnection and reconnection operation will be performed to ensure that the mixed stream data can be pushed to the designated channel again as soon as possible.

[0050] Exemplarily, the method further includes:

[0051] In the process of displaying the target mixed stream video, in response to the secondary mixed stream request, secondary mixed stream data is constructed according to the target mixed stream data; and based on the mixed stream request time corresponding to the secondary mixed stream request, the secondary mixed stream data is displayed.

[0052] The secondary mixing request may refer to a request for re-mixing the target mixed video. The secondary mixing data may refer to mixed data obtained by re-mixing the target mixed video.

[0053] Specifically, when the user plays the target mixed stream video, it can respond to the secondary mixed stream request triggered by the user. When the target mixed stream video needs to adjust the picture, it can quickly construct the secondary mixed stream data and realize the rapid switching of the display of the secondary mixed stream data at the mixing request moment corresponding to the secondary mixed stream request.

[0054] Exemplarily, in response to the secondary mixing request, secondary mixing data is constructed according to the target mixing data, including: determining the currently played video frame in the target mixed stream video when the secondary mixing request is received, and using the video frame after the currently played video frame as the first mixed stream video frame for the secondary mixing; using the material frame in the secondary mixing material corresponding to the secondary mixing request as the second mixed stream video frame for the secondary mixing; and merging the first mixed stream video frame and the second mixed stream video frame according to the secondary mixing requirement in the secondary mixing request to obtain secondary mixing data.

[0055] Among them, the first mixed stream video frame may refer to the video frame that constitutes the target mixed stream video. The second mixed stream video frame may refer to the video frame in the secondary mixed stream material. It should be noted that the secondary mixed stream material may also be a local material and / or an online material. The secondary mixed stream requirement is similar to the preset image merging specification, which may refer to the merging requirement of the secondary image frame, or the display screen requirement of the secondary mixed stream data, such as the screen size and screen layout.

[0056] Specifically, the current moment after receiving the secondary mixed stream request is determined, and the currently played video frame in the target mixed stream video is determined by the current moment. The video frame located after the currently played video frame in the target mixed stream video is determined as the first mixed stream video frame. The secondary mixed stream material is subjected to integrity check, validity check, frame rate conversion, material decoding and other operations, and the material frame in the secondary mixed stream material is used as the second mixed stream video frame of the secondary mixed stream. Referring to the secondary mixed stream requirements, each frame in the first mixed stream video frame is merged with the corresponding video frame in the second mixed stream video frame to obtain the secondary mixed stream data. Exemplarily, the video frames in the first mixed stream video frame and the second mixed stream video frame can be numbered respectively. The video frames with the same number in the first mixed stream video frame and the second mixed stream video frame are merged to obtain the secondary mixed stream data.

[0057] Exemplarily, the method further includes: after displaying the secondary mixed stream data in response to the secondary mixed stream request, the target mixed stream video located after the time of the secondary mixed stream request may be deleted to reduce data cache and improve mixed stream processing performance.

[0058] Embodiment 2

[0059] Figure 2 This is a flowchart of a multi-channel data mixing method provided by Embodiment 2 of the present invention. Based on the above embodiments, this embodiment refines the process of merging image frames in at least two channels of the to-be-mixed data to obtain a merged data image. Figure 2 As shown, the method includes:

[0060] S201: Obtain at least two channels of original data materials to be mixed.

[0061] S202: Perform frame rate conversion on each channel of the original data material to obtain at least two channels of data to be mixed.

[0062] S203: For each channel of the to-be-mixed stream data, decode the to-be-mixed stream data to obtain a decoded data image.

[0063] The decoded data image may refer to a video frame image obtained by decoding the mixed stream data.

[0064] Specifically, each channel of to-be-mixed stream data is decoded and processed respectively to obtain a decoded data image corresponding to each channel of to-be-mixed stream data.

[0065] Exemplarily, the decoding the to-be-mixed stream data to obtain a decoded data image includes:

[0066] Determine the color channel type corresponding to each channel of the to-be-mixed stream data. For each channel of the to-be-mixed stream data, decode the to-be-mixed stream data based on a decoding method corresponding to the color channel type to obtain a decoded data image.

[0067] Among them, the color channel types include RGB channel and RGBA channel. RGB channel refers to a color model for representing colors in digital images and videos, based on the additive color mixing principle of the three basic colors of red, green, and blue. The RGBA channel adds an Alpha channel to the RGB color model to indicate the transparency of the image. The value of the Alpha channel is usually an integer between 0 and 255 (for 8-bit / channel images), where 0 represents complete transparency, 255 represents complete opacity, and values ​​in between represent varying degrees of translucency.

[0068] When two or more images are superimposed, the alpha channel is used to determine how the colors of each pixel are mixed, and the alpha channel is used to achieve seamless merging of multiple images. Alpha channels are often used to create and edit masks to accurately select and manipulate parts of an image. For example, use the alpha channel to cut out an object and separate it from the background without losing details.

[0069] Specifically, according to the decoding method corresponding to the color channel type of each channel of the mixed stream data, the mixed stream data is decoded and processed to obtain a decoded data image. Exemplarily, when the color channel type is an RGB channel type, H264 technology is used for decoding to obtain a decoded data image. When the color channel type is an RGBA channel type, libvpx-vp9 technology is used for decoding, and the pixel format is set to YUVA format to obtain a decoded data image. For video streams containing alpha channels, more complex algorithms can be used to accurately restore the transparency and color information of the image. For video streams that do not contain alpha channels, a simpler and faster algorithm can be used to avoid unnecessary computational overhead.

[0070] S204: Based on a preset image merging requirement and according to image frame numbers corresponding to the decoded data images, the decoded data images respectively corresponding to the at least two channels of the to-be-mixed stream data are merged to obtain a merged data image.

[0071] Specifically, referring to the preset image merging requirement, the decoded data images with the same image frame number in at least two channels of the to-be-mixed stream data are merged to obtain a merged data image.

[0072] S205 . Based on the image frame number corresponding to the merged data image, integrate the merged data image to obtain target mixed stream data.

[0073] The technical solution of the embodiment of the present invention decodes the stream data to be mixed for each channel to obtain a decoded data image. Based on the preset image merging requirements, according to the image frame number corresponding to the decoded data image, the decoded data images corresponding to the at least two channels of the stream data to be mixed are merged to obtain a merged data image. By distinguishing different types of stream data to be mixed, computing resources can be allocated more effectively to ensure the real-time and smoothness of decoding, and at the same time, it can better adapt to different types of application scenarios and hardware platforms, providing users with a better quality and smoother video playback experience.

[0074] Embodiment 3

[0075] Figure 3 A schematic diagram of a multi-channel data mixing method provided in Embodiment 3 of the present invention is shown in FIG. Figure 3 As shown, the method includes:

[0076] After receiving the audio and video mixing request for the first time, it will go through five main modules: input parameter verification and preprocessing, input stream decoding, filter graph construction, output stream encoding, and data streaming.

[0077] The input parameter verification and preprocessing module will verify the legality of the input stream according to the following steps and perform preprocessing before decoding:

[0078] (1) Input parameter integrity check

[0079] (2) Verify the validity of online resources and audio and video streams by reading file headers and pulling streams.

[0080] (3) Download online image materials and convert them to MP4

[0081] (4) Download non-25fps video materials and convert them to 25fps MP4

[0082] The input stream decoding module decodes each input stream after pre-processing. The processing steps are as follows:

[0083] (1) For non-transparent materials that do not contain an alpha channel, use H264 technology to decode

[0084] (2) For transparent materials containing alpha channels, use libvpx-vp9 technology to decode and set the pixel format to YUVA format.

[0085] The filter graph construction step module will scale, layout, and image fuse the input stream, which is the key to the mixed stream technology. The specific steps are:

[0086] (1) Decode each input stream, synchronize the fixed frame rate and pts, set a fixed length buffer, and scale and layout the input stream.

[0087] (2) During decoding, the I frames of each input stream are aligned, and the pixel alignment of each input stream video frame is performed, and finally the final picture stream is merged.

[0088] (3) When image adjustment is required, in response to a secondary stream mixing request, a second set of filter graphs is constructed according to the new image requirements.

[0089] (4) When the secondary mixing request is initiated, the latest PTS in the old image buffersink is recorded as the synchronization point

[0090] (5) The alignment point uses the synchronization point PTS to infer the PTS of the video frame in the fixed buffer of the old image input stream.

[0091] (6) The second set of filter graphs remixes the video frames at maximum speed

[0092] (7) The switching point is when the playback progress of the new filter image catches up with the original filter image.

[0093] (8) Finally, the image is replaced and the resources of the old filter image are released.

[0094] The output stream encoding module has a relatively simple function and is only responsible for encoding the fused image into a pushable stream format.

[0095] The data push module is responsible for pushing the encoded data to the online channel. When there is a problem with the push communication, the module will reconnect to ensure that the mixed stream data can be pushed to the specified target channel as soon as possible.

[0096] Embodiment 4

[0097] Figure 4 This is a schematic diagram of the structure of a multi-channel data mixing device provided in Embodiment 4 of the present invention. Figure 4 As shown, the device comprises:

[0098] The original material acquisition module 301 is used to acquire at least two channels of original data materials to be mixed;

[0099] The module 302 for determining the data to be mixed is used to perform frame rate conversion processing on each of the original data materials to obtain at least two channels of data to be mixed;

[0100] A merged image determination module 303 is used to merge the image frames in the at least two channels of the to-be-mixed stream data based on a preset image merging specification to obtain a merged data image;

[0101] The mixed stream data determination module 304 is used to integrate the merged data image based on the image frame number corresponding to the merged data image to obtain target mixed stream data.

[0102] The technical solution of the embodiment of the present invention is to obtain at least two channels of original data materials to be mixed. Each channel of the original data materials is subjected to frame rate conversion processing to obtain at least two channels of data to be mixed. Based on a preset image merging specification, the image frames in the at least two channels of data to be mixed are merged to obtain a merged data image. Based on the image frame number corresponding to the merged data image, the merged data image is integrated to obtain the target mixed data, which effectively solves the existing problems of audio and video mixing technology in live broadcast scenarios, thereby enabling direct mixing of online resources and controlling the number of playbacks, facilitating users to perform flexible picture scheduling, and improving the ease of use, flexibility and effect of mixing.

[0103] Optionally, the merged image determination module 303 includes:

[0104] A decoded image obtaining unit, configured to decode each channel of the to-be-mixed stream data to obtain a decoded data image;

[0105] The merged image determination unit is used to merge the decoded data images corresponding to the at least two channels of the to-be-mixed stream data respectively based on the preset image merging requirement and according to the image frame number corresponding to the decoded data image to obtain a merged data image.

[0106] Optionally, the decoded image obtaining unit is specifically configured to:

[0107] Determine the color channel type corresponding to each channel of the to-be-mixed data, wherein the color channel type includes an RGB channel and an RGBA channel;

[0108] For each channel of the to-be-mixed stream data, the to-be-mixed stream data is decoded based on a decoding method corresponding to the color channel type to obtain a decoded data image.

[0109] Optionally, the device further comprises: a mixed flow data display module. The mixed flow data display module is used to:

[0110] The target mixed stream data is encoded to obtain a target mixed stream video, and the target mixed stream video is pushed to a preset online channel for display processing.

[0111] Optionally, the mixed flow data display module further includes:

[0112] A mixed stream data construction unit, configured to construct secondary mixed stream data according to the target mixed stream data in response to a secondary mixed stream request during display of the target mixed stream video;

[0113] The mixed flow data display unit is used to display the secondary mixed flow data based on the mixed flow request time corresponding to the secondary mixed flow request.

[0114] Optionally, the mixed flow data construction unit is specifically used to:

[0115] Determine a currently played video frame in the target mixed stream video when the secondary mixed stream request is received, and use a video frame after the currently played video frame as a first mixed stream video frame for the secondary mixed stream;

[0116] Using the material frame in the secondary mixed stream material corresponding to the secondary mixed stream request as the second mixed stream video frame of the secondary mixed stream;

[0117] According to the secondary stream mixing requirement in the secondary stream mixing request, the first stream mixing video frame and the second stream mixing video frame are merged to obtain secondary stream mixing data.

[0118] Optionally, the device further comprises: a mixed stream data deletion module. The mixed stream data deletion module is used to:

[0119] The target mixed stream video located after the time of the secondary mixed stream request is deleted.

[0120] The multi-channel data mixing device provided in the embodiment of the present invention can execute the multi-channel data mixing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] Embodiment 5

[0122] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0123] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0125] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a multi-channel data mixing method.

[0126] In some embodiments, the multi-channel data mixing method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the multi-channel data mixing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the multi-channel data mixing method in any other appropriate manner (for example, by means of firmware).

[0127] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0129] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0132] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0133] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0134] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-channel data mixing method, characterized in that: include: Obtain at least two channels of original data materials to be processed by mixed streams; Perform frame rate conversion on each channel of the original data material to obtain at least two channels of data to be mixed; Based on a preset image merging specification, merging the image frames in the at least two channels of the to-be-mixed stream data to obtain a merged data image; Based on the image frame number corresponding to the merged data image, the merged data image is integrated to obtain target mixed stream data.

2. The method according to claim 1, characterized in that: The step of merging the image frames in the at least two channels of the to-be-mixed stream data based on a preset image merging specification to obtain a merged data image includes: For each channel of the to-be-mixed stream data, the to-be-mixed stream data is decoded to obtain a decoded data image; Based on the preset image merging requirement, according to the image frame number corresponding to the decoded data image, the decoded data images respectively corresponding to the at least two channels of the to-be-mixed stream data are merged to obtain a merged data image.

3. The method according to claim 2, characterized in that The step of decoding the to-be-mixed stream data to obtain a decoded data image includes: Determine the color channel type corresponding to each channel of the to-be-mixed data, wherein the color channel type includes an RGB channel and an RGBA channel; For each channel of the to-be-mixed stream data, the to-be-mixed stream data is decoded based on a decoding method corresponding to the color channel type to obtain a decoded data image.

4. The method according to claim 1, characterized in that The method further comprises: The target mixed stream data is encoded to obtain a target mixed stream video, and the target mixed stream video is pushed to a preset online channel for display processing.

5. The method according to claim 4, characterized in that The method further comprises: In the process of displaying the target mixed stream video, in response to a secondary mixed stream request, constructing secondary mixed stream data according to the target mixed stream data; Based on the stream mixing request time corresponding to the secondary stream mixing request, the secondary stream mixing data is displayed and processed.

6. The method according to claim 5, characterized in that The step of constructing secondary stream mixing data according to the target stream mixing data in response to the secondary stream mixing request includes: Determine a currently played video frame in the target mixed stream video when the secondary mixed stream request is received, and use a video frame after the currently played video frame as a first mixed stream video frame for the secondary mixed stream; Using the material frame in the secondary mixed stream material corresponding to the secondary mixed stream request as the second mixed stream video frame of the secondary mixed stream; According to the secondary stream mixing requirement in the secondary stream mixing request, the first stream mixing video frame and the second stream mixing video frame are merged to obtain secondary stream mixing data.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: The target mixed stream video located after the time of the secondary mixed stream request is deleted.

8. A multi-channel data mixing device, characterized in that: include: The original material acquisition module is used to acquire at least two channels of original data materials to be processed by mixed streams; A module for determining data to be mixed, used for performing frame rate conversion processing on each channel of the original data material to obtain at least two channels of data to be mixed; A merged image determination module, configured to merge the image frames in the at least two channels of the to-be-mixed stream data based on a preset image merging specification to obtain a merged data image; The mixed stream data determination module is used to integrate the merged data image based on the image frame number corresponding to the merged data image to obtain target mixed stream data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-channel data mixing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-channel data mixing method according to any one of claims 1 to 7 when executed.

Images