Live broadcast data processing method and device and storage medium
By texture processing and coordinate conversion of the anchor video picture and the live broadcast auxiliary associated image on the push streaming server, the problem of inconsistency between the handwriting board content and the live broadcast video picture synthesis in the existing technology is solved, and the consistency between the anchor and the audience is achieved , improves the accuracy of the live broadcast screen and the audience experience.
Patent Information
- Application Number
- CN202411149377.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-22
AI Technical Summary
When the prior art synthesizes the contents of tools such as handwriting boards and live video images, it is restricted by rendering and other methods, resulting in inconsistent with the live video images displayed by the audience in real time with the pictures displayed by the anchor, resulting in poor accuracy of the picture processing. Influence the audience experience.
On the streaming server side, the host video picture and live assisted related images are obtained respectively, texture processing and synthesis are performed, coordinate system is converted, and target video picture is generated to ensure that the live assisted related images and anchor video picture are rendered and synthesized under the same coordinate system, so as to achieve Real-time preview and encoding streaming.
Through real-time rendering and synthesis, the consistency between the anchor and the audience-side pictures is ensured, and the accuracy of the live broadcast picture and the audience experience are improved.
Smart Images

Figure CN119011896A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a live broadcast data processing method, device and storage medium. Background Art
[0002] With the development of mobile live broadcast technology, mobile live broadcast has become the most flexible and convenient form of live broadcast. The implementation form of mobile live broadcast is mainly to collect video images and audio data through image and audio acquisition devices such as cameras and microphones. Taking video acquisition as an example, the collected video of the anchor needs to be previewed locally on the one hand, and the preview data is sent to the streaming media live broadcast server in real time for CDN (Content Delivery Network) forwarding, so as to realize the live broadcast viewing of the audience. When broadcasting live, due to the influence of the screen size of the mobile phone, tools such as handwriting tablets are usually used to transmit the relevant written content to the anchor through the local area network. The anchor synthesizes the video screen and the content in the handwriting tablet, and pushes the synthesized picture to the live broadcast server for the audience to watch.
[0003] However, currently, when the content in tools such as handwriting tablets are synthesized with live video images, due to limitations in rendering and other methods, the live images seen by the audience are inconsistent with the images displayed in real time by the host, resulting in poor accuracy in image processing and affecting the audience's experience. Summary of the invention
[0004] In order to solve the above problems, this application provides the following technical solutions:
[0005] A live broadcast data processing method, applied to a streaming server, comprising:
[0006] Obtain the anchor video screen and live broadcast auxiliary associated images corresponding to the anchor end respectively;
[0007] Performing texture processing on each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image to obtain a first synthetic image corresponding to each frame;
[0008] Convert the coordinates of the first composite image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor;
[0009] The first synthetic image corresponding to each frame after the coordinate conversion is processed to generate a target video screen.
[0010] Optionally, the obtaining of the anchor video screen and the live broadcast auxiliary associated screen corresponding to the anchor terminal respectively includes:
[0011] Receive the host video screen and the image path of the live broadcast auxiliary associated image sent by the host end;
[0012] Based on the image path, a live broadcast auxiliary associated image corresponding to the anchor video screen of the current frame is obtained; wherein the image path of the live broadcast auxiliary associated image is an image storage path generated after the anchor end converts the received live broadcast auxiliary associated image into a target image format.
[0013] Optionally, performing texture processing on each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image to obtain a first synthetic image corresponding to each frame includes:
[0014] Processing each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image respectively to obtain a first texture image of the anchor video screen and a second texture image of the live broadcast auxiliary associated image;
[0015] Rendering the first texture image to obtain a first layer;
[0016] Rendering the second texture image on the first layer to obtain a second layer;
[0017] The first layer and the second layer are synthesized to obtain a first synthesized image corresponding to each frame.
[0018] Optionally, obtaining the live broadcast auxiliary associated image corresponding to the anchor video picture of the current frame based on the image path includes:
[0019] Based on the image path, a live broadcast auxiliary associated coded image corresponding to the anchor video picture of the current frame is obtained;
[0020] The live broadcast auxiliary associated coded image is decoded to obtain a live broadcast auxiliary associated image.
[0021] Optionally, converting the coordinates of the first synthesized image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor includes:
[0022] Obtaining a first resolution of the operating system screen and a second resolution corresponding to a coordinate system of a rendering processor;
[0023] Based on the first resolution and the second resolution, the coordinates of the first synthesized image corresponding to each frame in the screen coordinate system of the operating system screen are converted into coordinates in the coordinate system of the rendering processor.
[0024] Optionally, the converting, based on the first resolution and the second resolution, coordinates of the first synthesized image corresponding to each frame in a screen coordinate system of the operating system screen into coordinates in a coordinate system of a rendering processor includes:
[0025] If the first resolution and the second resolution are the same, obtaining the width and height of the first composite image corresponding to each frame on the operating system screen, and the width and height of the operating system;
[0026] Based on the width and height of the first composite image on the operating system screen and the width and height of the operating system, the coordinates of the first composite image in the coordinate system of the rendering processor are determined.
[0027] Optionally, processing the first synthetic image corresponding to each frame after coordinate transformation to generate a target video picture includes:
[0028] Performing view processing on the first synthetic image corresponding to each frame after the coordinate transformation to obtain a processed image;
[0029] The processed image is encoded and formatted to obtain the target video image.
[0030] Optionally, it also includes:
[0031] The target video screen is sent to the anchor end, and the target video screen is live-streamed; wherein the target video screen is presented on the anchor end as a live preview screen of the anchor end.
[0032] A live broadcast data processing device, applied to a streaming server, comprising:
[0033] An acquisition unit, used to respectively obtain a host video screen and a live broadcast auxiliary associated image corresponding to the host terminal;
[0034] A first processing unit is used to perform texture processing on each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image to obtain a first synthetic image corresponding to each frame;
[0035] A conversion unit, used to convert the coordinates of the first composite image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor;
[0036] The second processing unit is used to process the first synthetic image corresponding to each frame after the coordinate conversion to generate a target video picture.
[0037] A storage medium, characterized in that the storage medium stores executable instructions, and when the instructions are executed by a processor, a live broadcast data processing method as described in any one of the above is implemented.
[0038] Compared with the prior art, the present application provides a live data processing method, device and storage medium, which is applied to a streaming server, including: respectively obtaining the anchor video screen and the live auxiliary associated image corresponding to the anchor end; performing texture processing on each frame of the anchor video screen and the corresponding live auxiliary associated image to obtain the first synthetic image corresponding to each frame; converting the coordinates of the first synthetic image corresponding to each frame in the screen coordinate system of the operating system screen into the coordinates in the coordinate system of the rendering processor; processing the first synthetic image corresponding to each frame after the coordinate conversion to generate a target video screen. In the embodiment of the present application, the live auxiliary associated image and the anchor video screen are rendered on the streaming server, and the live auxiliary associated image and the anchor video screen are rendered and synthesized in real time in the same coordinate system, and then the obtained target video screen is previewed and encoded and streamed in real time, thereby ensuring the consistency between the anchor end screen and the audience end screen. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0040] Figure 1 A flowchart of a live broadcast data processing method provided in an embodiment of the present application;
[0041] Figure 2 A processing flow chart of image coordinate conversion provided in an embodiment of the present application;
[0042] Figure 3 A schematic diagram of the structure of a live data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0044] The terms "first" and "second" and the like in the specification and claims of this application and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.
[0045] In an embodiment of the present application, a live broadcast data processing method is provided, which is applied to a streaming server, which is a server used to send the final video stream to the viewer end and the anchor end. Figure 1 , the method may include the following steps:
[0046] S101, respectively obtaining a host video screen and a live broadcast auxiliary associated image corresponding to the host end.
[0047] The anchor video screen corresponding to the anchor end is the real-time video screen of the anchor during the live broadcast, which can be collected by the live broadcast terminal device used by the anchor end. The live broadcast auxiliary associated image refers to the fact that in order to make the content display more comprehensive during the live broadcast, the anchor can use devices such as handwriting boards to assist the live broadcast. For example, relevant information during the live broadcast can be written on a whiteboard. The live broadcast auxiliary screen presented by these handwriting boards and other devices is called a live broadcast auxiliary associated screen. The live broadcast auxiliary associated screen is collected by a device that presents the live broadcast auxiliary associated screen (such as a whiteboard device), and each frame of the collected live broadcast auxiliary associated screen is sent to the anchor end, so that the anchor end can store each frame of the live broadcast auxiliary associated screen as a picture of the corresponding format, such as a picture stored in jpg format, and then send the storage path to the streaming server, thereby realizing the isolation of the business layer (i.e., the live broadcast screen) from the SDK (Software Development Kit) of the streaming server.
[0048] S102: Perform texture processing on each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image to obtain a first synthetic image corresponding to each frame.
[0049] Based on the time when the host video screen is collected and the time when the live broadcast auxiliary associated image is collected, the two are matched, that is, the host video screen and the live broadcast auxiliary associated image corresponding to each frame are obtained, and then the host video screen and the live broadcast auxiliary associated image corresponding to each frame are texture processed to obtain the first synthetic image corresponding to each frame. Correspondingly, when the push stream server receives the storage path information of the live broadcast auxiliary associated image sent by the host end, the corresponding live broadcast auxiliary associated image can be obtained through the storage path information, and the live broadcast auxiliary associated image is decoded. The decoded image can be uploaded to the graphics processor GPU through OpenGL ES, and a 2D layer is synthesized with the host video screen in the form of a texture. Among them, OpenGL ES is a mobile cross-platform rendering toolkit that can be used for 2D and 3D video rendering. Correspondingly, other rendering tools can also be used, and examples are not given one by one in the embodiments of this application. The first synthetic image corresponding to each frame obtained by texture synthesis can be applied to subsequent three-dimensional videos, so that the final video screen is more matched with the real scene and the screen presentation effect is better.
[0050] S103 : Convert the coordinates of the first synthesized image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor.
[0051] The streaming server converts the screen coordinates of the first composite image corresponding to each frame on the operating system screen into coordinates in the coordinate system of the rendering processor, wherein the horizontal coordinates of the operating system screen may be the View coordinates, and the coordinate system of the rendering processor may be the adopted OpenGL ES coordinates. Thus, the position of the live broadcast auxiliary associated image (such as a whiteboard image) can be changed at the View layer, and further, the real-time scaling animation of the live broadcast auxiliary associated image can be achieved by setting the scaling factor and scaling time.
[0052] Among them, the View coordinates belong to the UI coordinate system, that is, the coordinates in the controls provided by the operating system. Android and iOS systems have their own coordinate systems, which are generally based on the upper left corner as the origin and right and downward as the positive directions. The rendering tool has its own coordinate system, with the center of the image as the origin and right and upward as the positive directions. Therefore, in order to provide better image display effects, the coordinate system needs to be converted. That is, the coordinate system operated by the anchor end is the coordinate system in the UI VIEW, and the operating screen is the screen in OpenGL ES, which is an operation form in which two spaces are combined into one. Therefore, a conversion is required to convert the coordinates in the View to the OpenGL ES rendering space.
[0053] S104: Process the first synthetic image corresponding to each frame after the coordinate conversion to generate a target video picture.
[0054] The first synthetic image corresponding to each frame after the coordinate conversion is processed, and the streaming server may perform a reverse conversion on the coordinates in step S103, so as to calculate the drag increment of the business layer in real time, realize the drag animation of the live broadcast auxiliary associated screen, and then obtain the final target video screen.
[0055] In one implementation of the embodiment of the present application, the target video screen is sent to the anchor end, and the target video screen is live streamed; wherein the target video screen is presented on the anchor end as a live preview screen of the anchor end. Specifically, the streaming server can send the synthesized target video screen to the anchor end for live preview, and encode it and stream it to the server end for distribution, so that the audience end obtains the target video screen.
[0056] The embodiment of the present application provides a live data processing method, which is applied to a streaming server, including: respectively obtaining the anchor video screen and the live auxiliary associated image corresponding to the anchor end; performing texture processing on each frame of the anchor video screen and the corresponding live auxiliary associated image to obtain the first synthetic image corresponding to each frame; converting the coordinates of the first synthetic image corresponding to each frame in the screen coordinate system of the operating system screen into the coordinates in the coordinate system of the rendering processor; processing the first synthetic image corresponding to each frame after the coordinate conversion to generate a target video screen. In the embodiment of the present application, the live auxiliary associated image and the anchor video screen are rendered on the streaming server, and the live auxiliary associated image and the anchor video screen are rendered and synthesized in real time in the same coordinate system, and then the obtained target video screen is previewed and encoded and streamed in real time, thereby ensuring the consistency between the anchor end screen and the audience end screen.
[0057] In the embodiment of the present application, the live broadcast auxiliary associated screen is no longer displayed in the form of an overlay of the user interface UI layer, but these data are directly sent to the streaming server for rendering processing. By calculating the associated information (such as position, size, scaling factor, etc.) between the live broadcast auxiliary associated screen and the host video screen in real time, the live broadcast auxiliary associated screen and the host video screen are drawn and synthesized through the OpenGL ES rendering engine, and then the synthesized screen is previewed and encoded and streamed in real time. In this way, the layer scaling, dragging and other animations can be efficiently realized, thereby ensuring the consistency of the host end screen and the audience end screen, and achieving a WYSIWYG visual effect.
[0058] The following describes the relevant implementation methods in the embodiments of the present application in conjunction with specific application scenarios. The live broadcast auxiliary associated images collected by the device for generating the live broadcast auxiliary associated images can be sent directly to the streaming server, or the storage path of the live broadcast auxiliary associated images can be sent by the host end to the streaming server in order to achieve matching with the host video screen, thereby ensuring that the streaming server can receive the live broadcast auxiliary associated images that match the host video screen. In one implementation of the embodiments of the present application, the obtaining of the host video screen and the live broadcast auxiliary associated screen corresponding to the host end respectively includes:
[0059] Receive the host video screen and the image path of the live broadcast auxiliary associated image sent by the host end;
[0060] Based on the image path, a live broadcast auxiliary associated image corresponding to the anchor video screen of the current frame is obtained; wherein the image path of the live broadcast auxiliary associated image is an image storage path generated after the anchor end converts the received live broadcast auxiliary associated image into a target image format.
[0061] Furthermore, the obtaining of the live broadcast auxiliary associated image corresponding to the anchor video screen of the current frame based on the image path includes: obtaining the live broadcast auxiliary associated coded image corresponding to the anchor video screen of the current frame based on the image path; decoding the live broadcast auxiliary associated coded image to obtain the live broadcast auxiliary associated image. In order to compress the image size for easy transmission, the image is generally encoded by image encoding technology, so it is necessary to perform corresponding decoding processing after obtaining the image.
[0062] After the streaming server obtains each frame of the host video screen and the corresponding live broadcast auxiliary associated image, texture processing will be performed to provide a more realistic data basis for the subsequent presentation of the video image. In one embodiment, the texture processing of each frame of the host video screen and the corresponding live broadcast auxiliary associated image to obtain the first synthetic image corresponding to each frame includes:
[0063] Processing each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image respectively to obtain a first texture image of the anchor video screen and a second texture image of the live broadcast auxiliary associated image;
[0064] Rendering the first texture image to obtain a first layer;
[0065] Rendering the second texture image on the first layer to obtain a second layer;
[0066] The first layer and the second layer are synthesized to obtain a first synthesized image corresponding to each frame.
[0067] When performing coordinate system conversion in the embodiment of the present application, it is necessary to process according to the resolution of the operating system screen so that the image after coordinate conversion can better meet the output scene when output. Specifically, the coordinates of the first synthetic image corresponding to each frame in the screen coordinate system of the operating system screen are converted into coordinates in the coordinate system of the rendering processor, including:
[0068] Obtaining a first resolution of the operating system screen and a second resolution corresponding to a coordinate system of a rendering processor;
[0069] Based on the first resolution and the second resolution, the coordinates of the first synthesized image corresponding to each frame in the screen coordinate system of the operating system screen are converted into coordinates in the coordinate system of the rendering processor.
[0070] Furthermore, based on the first resolution and the second resolution, converting the coordinates of the first synthesized image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor includes:
[0071] If the first resolution and the second resolution are the same, obtaining the width and height of the first composite image corresponding to each frame on the operating system screen, and the width and height of the operating system;
[0072] Based on the width and height of the first composite image on the operating system screen and the width and height of the operating system, the coordinates of the first composite image in the coordinate system of the rendering processor are determined.
[0073] See also Figure 2 , is a processing flow chart of image coordinate system conversion provided in an embodiment of the present application.
[0074] The image displayed in the view window can be a horizontal screen image or a vertical screen image. First, the width and height of the view window presenting the image are obtained, expressed as ar (aspect ratio, aspect ratio or screen width-to-height ratio). The coordinate system of the current operating point (x, y) is known, and the coordinate system of the rendering processor (also called the rendering engine) is also known, that is, the image display area in the rendering processor is also known, that is, the rendering processor also corresponds to a rendering width and rendering height. By judging the aspect ratio of ar and the display area of the rendering processor, corresponding calculations are performed to perform coordinate conversion. Among them, it is necessary to calculate the image position and pixel ratio. The pixel ratio is the ratio of the width to the height of a single pixel in the image in each pixel. The angle of the image is determined according to the image position. If the image angle is 0 degrees, it is a vertical screen image. The real area of the image is obtained by ar=w / h, where w is the width of the view window presenting the image, and h is the height of the window; if the image angle is 90 degrees, it means that the presented image is a horizontal screen image. The real area of the image is obtained by ar=h / w, and then the coordinates in the real area (that is, the real rendering area) are calculated through coordinate transformation.
[0075] If ar is greater than the aspect ratio of the rendering window in the rendering processor, the height of the image's actual rendering area rectHeight = rectHeight; the width of the image's actual rendering area rectWidth = rectHeight*ar. If ar is less than the aspect ratio of the rendering window in the rendering processor, the height of the image's actual rendering area rectWidth = rectWidth; the width of the image's actual rendering area rectHeight = rectWidth / ar.
[0076] The actual rendering coordinates of the first composite image under the rendering processor are obtained by the following calculation:
[0077] Rendering coordinates horizontal coordinate x = (screen coordinate point x / rectWidth)*2-1
[0078] Rendering coordinate ordinate y = -(screen coordinate point y / rectHeight)*2+1
[0079] In one implementation of the embodiment of the present application, the processing of the first synthetic image corresponding to each frame after the coordinate transformation to generate the target video picture includes:
[0080] Performing view processing on the first synthesized image corresponding to each frame after coordinate transformation to obtain a processed image; encoding and format packaging the processed image to obtain a target video screen.
[0081] See also Figure 3In an embodiment of the present application, a live data processing device is also provided, which is applied to a streaming server, including:
[0082] An acquisition unit 301 is used to respectively obtain a host video screen and a live broadcast auxiliary associated image corresponding to the host terminal;
[0083] The first processing unit 302 is used to perform texture processing on each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image to obtain a first synthetic image corresponding to each frame;
[0084] A conversion unit 303, configured to convert the coordinates of the first synthesized image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor;
[0085] The second processing unit 304 is used to process the first synthetic image corresponding to each frame after the coordinate conversion to generate a target video picture.
[0086] Optionally, the acquiring unit includes:
[0087] A receiving subunit, used for receiving the host video screen and the image path of the live broadcast auxiliary associated image sent by the host end;
[0088] The first acquisition subunit is used to obtain a live broadcast auxiliary associated image corresponding to the anchor video screen of the current frame based on the image path; wherein the image path of the live broadcast auxiliary associated image is an image storage path generated after the anchor end converts the received live broadcast auxiliary associated image into a target image format.
[0089] Optionally, the first processing unit includes:
[0090] The first processing subunit is used to process the anchor video picture and the corresponding live broadcast auxiliary associated image of each frame respectively to obtain a first texture image of the anchor video picture and a second texture image of the live broadcast auxiliary associated image;
[0091] A first rendering subunit, used for rendering the first texture image to obtain a first layer;
[0092] A second rendering subunit, configured to render the second texture image on the first layer to obtain a second layer;
[0093] The synthesis subunit is used to synthesize the first layer and the second layer to obtain a first synthesized image corresponding to each frame.
[0094] Optionally, the first acquiring subunit is configured to:
[0095] Based on the image path, a live broadcast auxiliary associated coded image corresponding to the anchor video picture of the current frame is obtained;
[0096] The live broadcast auxiliary associated coded image is decoded to obtain a live broadcast auxiliary associated image.
[0097] Optionally, the conversion unit includes:
[0098] A second acquisition subunit is used to obtain a first resolution of the operating system screen and a second resolution corresponding to the coordinate system of the rendering processor;
[0099] The conversion subunit is used to convert the coordinates of the first composite image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor based on the first resolution and the second resolution.
[0100] Optionally, the conversion subunit is configured as:
[0101] If the first resolution and the second resolution are the same, obtaining the width and height of the first composite image corresponding to each frame on the operating system screen, and the width and height of the operating system;
[0102] Based on the width and height of the first composite image on the operating system screen and the width and height of the operating system, the coordinates of the first composite image in the coordinate system of the rendering processor are determined.
[0103] Optionally, the second processing unit includes:
[0104] A first processing subunit, configured to perform view processing on a first synthetic image corresponding to each frame after coordinate transformation to obtain a processed image;
[0105] The encapsulation subunit is used to encode and format the processed image to obtain the target video picture.
[0106] Optionally, it also includes:
[0107] The sending unit is used to send the target video picture to the anchor end and live broadcast the target video picture; wherein the target video picture is presented on the anchor end as a live preview picture of the anchor end.
[0108] Based on the foregoing embodiments, a storage medium is further provided in an embodiment of the present application, wherein the storage medium stores executable instructions, and when the instructions are executed by a processor, a live broadcast data processing method as described in any one of the foregoing embodiments is implemented.
[0109] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a live data processing method when executing the program.
[0110] It should be noted that the above-mentioned processor or CPU can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that the electronic device that realizes the function of the above-mentioned processor can also be other, and the embodiments of the present application are not specifically limited.
[0111] It should be noted that the above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.
[0112] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0113] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0114] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A live broadcast data processing method, characterized in that: Applied to streaming servers, including: Obtain the anchor video screen and live broadcast auxiliary associated images corresponding to the anchor end respectively; Performing texture processing on each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image to obtain a first synthetic image corresponding to each frame; Convert the coordinates of the first composite image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor; The first synthetic image corresponding to each frame after the coordinate conversion is processed to generate a target video screen.
2. The method according to claim 1, characterized in that The method of respectively obtaining the anchor video screen and the live broadcast auxiliary associated screen corresponding to the anchor terminal includes: Receive the host video screen and the image path of the live broadcast auxiliary associated image sent by the host end; Based on the image path, a live broadcast auxiliary associated image corresponding to the anchor video screen of the current frame is obtained; wherein the image path of the live broadcast auxiliary associated image is an image storage path generated after the anchor end converts the received live broadcast auxiliary associated image into a target image format.
3. The method according to claim 1, characterized in that The performing texture processing on each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image to obtain a first synthetic image corresponding to each frame includes: Processing each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image respectively to obtain a first texture image of the anchor video screen and a second texture image of the live broadcast auxiliary associated image; Rendering the first texture image to obtain a first layer; Rendering the second texture image on the first layer to obtain a second layer; The first layer and the second layer are synthesized to obtain a first synthesized image corresponding to each frame.
4. The method according to claim 2, characterized in that: The obtaining, based on the image path, a live broadcast auxiliary associated image corresponding to the anchor video picture of the current frame includes: Based on the image path, a live broadcast auxiliary associated coded image corresponding to the anchor video picture of the current frame is obtained; The live broadcast auxiliary associated coded image is decoded to obtain a live broadcast auxiliary associated image.
5. The method according to claim 1, characterized in that The step of converting the coordinates of the first synthesized image corresponding to each frame in the screen coordinate system of the operating system screen into the coordinates of the rendering processor includes: Obtaining a first resolution of the operating system screen and a second resolution corresponding to a coordinate system of a rendering processor; Based on the first resolution and the second resolution, the coordinates of the first synthesized image corresponding to each frame in the screen coordinate system of the operating system screen are converted into coordinates in the coordinate system of the rendering processor.
6. The method according to claim 5, characterized in that The step of converting the coordinates of the first composite image corresponding to each frame in the screen coordinate system of the operating system screen into the coordinates of the rendering processor based on the first resolution and the second resolution includes: If the first resolution and the second resolution are the same, obtaining the width and height of the first composite image corresponding to each frame on the operating system screen, and the width and height of the operating system; Based on the width and height of the first composite image on the operating system screen and the width and height of the operating system, the coordinates of the first composite image in the coordinate system of the rendering processor are determined.
7. The method according to claim 1, characterized in that The step of processing the first synthetic image corresponding to each frame after the coordinate conversion to generate a target video screen includes: Performing view processing on the first synthetic image corresponding to each frame after the coordinate transformation to obtain a processed image; The processed image is encoded and formatted to obtain the target video image.
8. The method according to claim 1, characterized in that Also includes: The target video screen is sent to the anchor end, and the target video screen is live-streamed; wherein the target video screen is presented on the anchor end as a live preview screen of the anchor end.
9. A live broadcast data processing device, characterized in that: Applied to streaming servers, including: An acquisition unit, used to respectively obtain a host video screen and a live broadcast auxiliary associated image corresponding to the host terminal; A first processing unit is used to perform texture processing on each frame of the anchor video screen and the corresponding live broadcast auxiliary associated image to obtain a first synthetic image corresponding to each frame; A conversion unit, used to convert the coordinates of the first composite image corresponding to each frame in the screen coordinate system of the operating system screen into coordinates in the coordinate system of the rendering processor; The second processing unit is used to process the first synthetic image corresponding to each frame after the coordinate conversion to generate a target video picture.
10. A storage medium, characterized in that: The storage medium stores executable instructions, and when the instructions are executed by the processor, the live broadcast data processing method as described in any one of claims 1-8 is implemented.
