Live video playback method and its device, equipment, and medium
Patent Information
- Application Number
- CN202211659152.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-12-22
Smart Images

Figure CN116170636B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network live broadcasting, and in particular to a live video playback method, and also to corresponding devices, equipment and computer-readable storage media of the method. Background Art
[0002] With the development of playback devices, some devices can play videos with high dynamic range image imaging specifications (HDR). Internet live broadcast platforms have also kept pace with the times and usually provide live video streams with high dynamic range image standards to the audience of the platform to improve the video viewing experience of the live video streams of each anchor end in the platform. However, the playback devices used by the audience users in the platform are different. The live video streams with high dynamic range image imaging specifications (HDR) occupy a huge amount of computing power resources during encoding and decoding, and have high configuration requirements for the anchor end, server and audience end of the platform. Therefore, the anchor end or audience end that pushes the stream may not be able to smoothly transmit the live video stream with high dynamic range, resulting in interruption of the live broadcast. For the audience end, it may cause the playback device on the audience end to freeze or crash, resulting in the inability to smoothly play the live video stream in the device, affecting the live viewing experience of the audience users.
[0003] In view of the problems existing in the existing playback of high dynamic range image live video streams, the applicant has made corresponding explorations in order to solve the problems. Summary of the invention
[0004] The purpose of this application is to provide a live video playback method to meet user needs, and also relates to corresponding devices, equipment, computer-readable storage media and computer program products of the method.
[0005] In order to achieve the purpose of this application, the following technical solutions are adopted:
[0006] In response to a live video stream push instruction, determine an interest region of a frame image in a live video stream of a high dynamic range image, and copy the interest region as a first high dynamic range image;
[0007] Filling the region of interest in the frame image with a solid color, and rendering the frame image as a standard dynamic range image;
[0008] Software encoding and decoding and hardware encoding and decoding are used for the first high dynamic range image and the standard dynamic range image respectively, and the standard dynamic range image is rendered into a second high dynamic range image, so that the first high dynamic range image and the second high dynamic range image are combined and output to a live video stream playback window for display.
[0009] In a further embodiment, the step of determining the region of interest of a frame image in the high dynamic range image live video stream comprises the following steps:
[0010] Obtaining a currently played frame image in a high dynamic range image live video stream, and performing image preprocessing on the frame image;
[0011] Extracting the image feature vector of the frame image after image preprocessing;
[0012] According to the image feature vector, a region of interest in the frame image is identified.
[0013] In a further embodiment, the step of determining the region of interest of a frame image in the high dynamic range image live video stream comprises the following steps:
[0014] In response to an interest region editing event acting on a live video stream playback window, obtaining interest region positioning information corresponding to the interest region editing event;
[0015] Acquire a currently played frame image in a high dynamic range image live video stream, and determine a center point of the region of interest corresponding to the frame image according to the region of interest positioning information including the region positioning data;
[0016] According to the region area data included in the region area positioning information, the region of interest corresponding to the region area data is determined from the center point of the region of interest in the frame image.
[0017] In a further embodiment, the step of copying the region of interest as a first high dynamic range image comprises the following steps:
[0018] Acquire image pixels contained in the region of interest in the frame image;
[0019] A first high dynamic range image of high dynamic range imaging specifications is generated, and pixels of the image are merged into the first high dynamic range image.
[0020] In a further embodiment, the step of filling the region of interest in the frame image with a solid color and rendering the frame image as a standard dynamic range image comprises the following steps:
[0021] Filling the image pixels corresponding to the region of interest in the frame image with pure color pixels;
[0022] The frame image is rendered from a high dynamic range imaging specification to a standard dynamic range image of a standard dynamic range imaging specification using a hardware rendering algorithm.
[0023] In a further embodiment, the step of using software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and rendering the standard dynamic range image into a second high dynamic range image, so as to combine the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display, comprises the following steps:
[0024] Acquire image positioning information and rendering formula information, wherein the image positioning information is determined according to the relative position of the region of interest in the frame image, and the rendering formula information is a conversion formula used to render the frame image into the standard dynamic range image;
[0025] Using software encoding and hardware encoding for the first high dynamic range image and the standard dynamic range image respectively to obtain high dynamic soft-coded data and standard dynamic hard-coded data;
[0026] The image positioning information, the rendering formula information, the high dynamic soft coding data and the standard dynamic hard coding data are pushed to the audience end to drive the audience end to perform software decoding and hardware decoding on the high dynamic coding data and the standard dynamic coding data respectively, and perform inverse conversion according to the rendering formula information to render the decoded standard dynamic range image into a second high dynamic range image, and then according to the image positioning information, merge the first dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display.
[0027] In a further embodiment, the step of using software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and rendering the standard dynamic range image into a second high dynamic range image, so as to combine the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display, comprises the following steps:
[0028] storing the frame numbers corresponding to the first high dynamic range image and the frame image in a first image queue;
[0029] storing the frame numbers corresponding to the second high dynamic range image and the frame image in a second image queue;
[0030] A first high dynamic range image and a second high dynamic range image with the same frame number are obtained from the first image queue and the second image queue, and the first high dynamic range image and the second high dynamic range image are merged and output to the live video stream playback window for display.
[0031] A live video playback device proposed for the purpose of this application includes:
[0032] An interest region copying module is used to respond to a live video stream push instruction, determine an interest region of a frame image in a live video stream of a high dynamic range image, and copy the interest region as a first high dynamic range image;
[0033] A standard image rendering module, used to fill the region of interest in the frame image with a solid color and render the frame image into a standard dynamic range image;
[0034] The merged image output module is used to use software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and render the standard dynamic range image into a second high dynamic range image, so as to merge the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display.
[0035] In a further embodiment, the interest region copy module includes:
[0036] An image preprocessing submodule is used to obtain a currently played frame image in a high dynamic range image live video stream and perform image preprocessing on the frame image;
[0037] A vector extraction submodule, used to extract the image feature vector of the frame image after image preprocessing;
[0038] The region of interest identification submodule is used to identify the region of interest in the frame image according to the image feature vector.
[0039] In a preferred embodiment, the interest region copy module further includes:
[0040] The interest region editing submodule is used to respond to an interest region editing event acting on the live video stream playback window and obtain interest region positioning information corresponding to the interest region editing event;
[0041] A region center point determination submodule is used to obtain a frame image currently being played in a high dynamic range image live video stream, and determine a center point of the region of interest corresponding to the frame image according to the region positioning data included in the region positioning information;
[0042] The interest region determination submodule is used to determine the interest region corresponding to the area data from the center point of the interest region in the frame image according to the area data included in the interest region positioning information.
[0043] In a preferred embodiment, the interest region copy module further includes:
[0044] An image pixel acquisition submodule, used to acquire image pixels contained in the region of interest in the frame image;
[0045] The image pixel merging submodule is used to generate a first high dynamic range image of high dynamic range imaging specifications, and merge the image pixels into the first high dynamic range image.
[0046] In a further embodiment, the standard image rendering module includes:
[0047] A pixel filling submodule, used to fill the image pixels corresponding to the region of interest in the frame image with pure color pixels;
[0048] The image rendering submodule is used to render the frame image from a high dynamic range imaging specification to a standard dynamic range image of a standard dynamic range imaging specification using a hardware rendering algorithm.
[0049] In a further embodiment, the merged image output module includes:
[0050] An image association information acquisition submodule, used to acquire image positioning information and rendering formula information, wherein the image positioning information is determined according to the relative position of the region of interest in the frame image, and the rendering formula information is a conversion formula used to render the frame image into the standard dynamic range image;
[0051] The coding data acquisition submodule is used to use software coding and hardware coding to respectively perform software coding and hardware coding on the first high dynamic range image and the standard dynamic range image to obtain high dynamic soft coding data and standard dynamic hard coding data;
[0052] The image merging and outputting submodule is used to push the image positioning information, the rendering formula information, the high dynamic soft coding data and the standard dynamic hard coding data to the audience end, so as to drive the audience end to perform software decoding and hardware decoding on the high dynamic coding data and the standard dynamic coding data respectively, and perform inverse conversion according to the rendering formula information to render the decoded standard dynamic range image into a second high dynamic range image, and then according to the image positioning information, merge the first dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display.
[0053] In a preferred embodiment, the merged image output module further includes:
[0054] A first high dynamic range image queueing submodule, configured to store the frame numbers corresponding to the first high dynamic range image and the frame image in a first image queue;
[0055] A second high dynamic range image queue entry submodule, used for storing the frame number sequence corresponding to the second high dynamic range image and the frame image in a second image queue;
[0056] The image merging and outputting submodule is used to obtain the first high dynamic range image and the second high dynamic range image with the same frame number from the first image queue and the second image queue, merge the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display.
[0057] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned live video playback method.
[0058] In order to solve the above technical problems, an embodiment of the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned live video playback method.
[0059] In order to solve the above technical problems, an embodiment of the present application also provides a computer program product, including a computer program and computer instructions. When the computer program and computer instructions are executed by a processor, the processor executes the steps of the above-mentioned live video playback method.
[0060] Compared with the prior art, the advantages of this application are as follows:
[0061] The present application distinguishes the interest area of the live video screen of the high dynamic range imaging specification (HDR), and performs different image processing on the image data of the interest area and the image data outside the interest area, so as to optimize the processing and transmission efficiency of the live video stream of the high dynamic range imaging specification. For the image in the interest area of the live video screen, the image data is generally less than the image outside the interest area. Therefore, the image in the interest area is software encoded and decoded running on the central processing unit (CPU), and the image outside the interest area is hardware encoded and decoded running on the graphics processing unit (GPU). This can effectively reduce the computing pressure of the central processing unit (CPU) of the device when the live video stream push end performs live video stream decoding, so that the computing resources saved by the central processing unit (CPU) of the server or anchor end can be applied to other services of the live online service, ensuring the stability of the live online service. The same effect is also achieved for the playback end that decodes and plays the live video stream, so that the central processing unit (CPU) of the audience end device can apply the saved computing resources to other functions of the live service, ensuring that the audience users can fully and smoothly participate in the live activities of the live broadcast room.
[0062] Secondly, after copying the image pixels of the area of interest of the frame image played in the live video stream, the present application fills the image pixels of the area of interest in the frame image with a solid color to reduce the resources occupied by the original image pixels in the area of interest, save the memory resources occupied by the output of the standard dynamic range image of the image pixels of the non-interest area, and save the computing and transmission resources required for the encoding, decoding and transmission of the image data.
[0063] In addition, the present application converts an image with more image data from the original high dynamic range imaging specification (HDR) into a standard dynamic range imaging specification (SDR) before encoding and decoding. The bit width occupied by the standard dynamic range imaging specification (SDR) during encoding and decoding will be much smaller than the bit width occupied by the high dynamic range imaging specification (HDR), so as to save the computing power resources occupied by the device when encoding and decoding the image, and after the audience end decodes and obtains the image of the standard dynamic range imaging specification (SDR), it will also perform high dynamic range rendering to ensure that the final live video stream can be played in the high dynamic range imaging specification (HDR). BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0065] Figure 1 A schematic diagram of a typical network deployment architecture related to implementing the technical solution of this application;
[0066] Figure 2 A flowchart of a typical embodiment of the live video playback method of the present application;
[0067] Figure 3 This is a schematic diagram of the interface for editing the interest area of the live video stream in the live studio interface of this application;
[0068] Figure 4 A schematic diagram of an image for copying an area of interest in a frame image to generate a first high dynamic range image in the present application;
[0069] Figure 5 This is a schematic diagram of filling a region of interest in a frame image with a solid color to generate an image with a standard dynamic range in this application;
[0070] Figure 6 A flowchart of a specific implementation method of using an image recognition model trained to convergence to identify a region of interest in a frame image in the present application;
[0071] Figure 7 A flowchart of a specific implementation method for determining an area of interest edited by a host user in this application;
[0072] Figure 8 A schematic diagram of a flow chart formed by a specific implementation method of copying an area of interest in a frame image to generate a first high dynamic range image in the present application;
[0073] Fig. 9 A flowchart of a specific implementation method of filling a region of interest in a frame image with a pure color to generate a standard dynamic range image in the present application;
[0074] Fig.10 An image schematic diagram of merging a first high dynamic range image and a second high dynamic range image to generate a new frame image in this application;
[0075] Fig.11 A flowchart of a specific implementation method of combining a first high dynamic range image and a standard dynamic range image to generate a new frame image and output it to a live video stream playback window for display in the present application;
[0076] Fig.12 A flowchart of a specific implementation method of synchronously outputting a first high dynamic range image and a second high dynamic range image to a live video stream playback window for display in the present application;
[0077] Fig.13 This is a principle block diagram of a typical embodiment of the live video playback device of the present application;
[0078] Fig.14 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION
[0079] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0080] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0081] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0082] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; traditional laptop and / or palmtop computers or other devices, which have and / or include radio frequency receivers. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.
[0083] The hardware referred to by the names such as "server", "client", and "work node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.
[0084] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.
[0085] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As an online server, it can be responsible for further connecting relevant data servers and other servers that provide relevant support, etc., so as to form a logically related service cluster to provide services for related terminal devices such as the smartphone 81 and personal computer 82 shown in the figure or a third-party server (not shown). The smartphone and personal computer can access the Internet through a well-known network access method and establish a data communication link with the server 80 in the cloud to run terminal applications related to the services provided by the server.
[0086] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.
[0087] The application described herein refers to an application running on a server or terminal device. This application implements the relevant technical solutions of the present application in a programming manner. Its program code can be stored in a computer-readable storage medium recognizable by a computer in the form of computer-executable instructions, and can be loaded into the memory by the central processing unit for execution. The relevant device of the present application is constructed by the operation of the application on the computer.
[0088] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.
[0089] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.
[0090] See also Figure 2 A live video playback method of the present application, in its typical embodiment, comprises the following steps:
[0091] Step S11, in response to a live video stream push event, determining an interest region of a frame image in a high dynamic range image live video stream, and copying the interest region as a first high dynamic range image:
[0092] The live video stream pushing event is an event triggered by broadcasting the live video stream of the anchor end to the audience end in the live broadcast room of the anchor end. After the audience end enters the live broadcast room of any anchor end, it will obtain the live video stream corresponding to the live broadcast room from the platform server. The live video stream possessed by the server is pushed by the anchor end to which the live broadcast room entered by the audience end belongs. Among them, the live video stream of some live broadcast rooms is the high dynamic range image live video stream with the high dynamic range image standard (HDR) as the specification.
[0093] Before the server broadcasts the high dynamic range image live video stream to the audience in the live broadcast room, or before the anchor end broadcasts its high dynamic range image live video stream to the audience in the live broadcast room, the server or the anchor end will determine the interest area in the frame image of the high dynamic range image live video stream. The interest area generally refers to the image area in the frame image that needs to be rendered and displayed according to the high dynamic range image standard (HDR). The interest area can generally be determined by a recognition algorithm or a neural network model. The interest area in the frame image can be determined by the anchor user at the anchor end by editing the corresponding custom edited interest area in the live video stream playback window to use the interest area as the interest area in the frame image.
[0094] Regarding the automatic determination of the interest region of the frame image through a recognition algorithm or a neural network model, the interest region generally refers to objects in the frame image such as the host's face, host's character or live broadcast products that the audience users are interested in when watching the live video. Therefore, when automatically determining the interest region of the frame image through a recognition algorithm or a neural network model, the image region of the host's face, host's character or live broadcast products that the audience users are interested in when watching the live video will generally be identified in the frame image, and the image region will be used as the interest region.
[0095] Regarding the implementation method of determining the region of interest in the frame image through an image recognition model constructed based on a neural network, specifically, a frame image currently being played in a high dynamic range image live video stream is obtained, image preprocessing is performed on the frame image, and then an image feature vector of the frame image after the image preprocessing is extracted, and based on the image feature vector, the result is obtained through a fully connected layer or a convolutional layer and then normalized to identify the region of interest in the frame image.
[0096] Please refer to Figure 3 The interest region can also be customized and edited by the anchor user at the anchor end. Specifically, in response to the interest region editing event acting on the live video stream playback window, the interest region positioning information corresponding to the interest region editing event is obtained, such as Figure 3 In the live studio interface shown, the audience user at the audience end can edit the corresponding interest area in the live video stream playback window 301 shown, such as the interest area editing control 302 in the live video stream playback window 301, and edit and frame the corresponding interest area with reference to the live video stream played in the live video stream playback window 301 through the interest area editing control 302, so as to generate the interest area positioning information corresponding to the interest area edited and framed by the interest area editing control 302. The anchor end obtains the frame image currently played in the high dynamic range image live video stream, determines the center point of the interest area corresponding to the frame image according to the area positioning data contained in the interest area positioning information, and then determines the interest area corresponding to the area area data from the center point of the interest area in the frame image according to the area positioning data contained in the interest area positioning information.
[0097] Please refer to Figure 4 After the server or the anchor terminal determines the region of interest in the frame image, the image pixels contained in the region of interest are merged into the first high dynamic range image in a high dynamic range imaging specification, and then the image pixels are merged into the first high dynamic range image. Specifically, Figure 4 As shown, Figure 4 As shown in Figure 4The image pixels included in the region of interest 402 in the frame image 401 shown in FIG. 4 are merged into the first high dynamic range image 402 shown in FIG.
[0098] Step S12, filling the region of interest in the frame image with a solid color, and rendering the frame image into a standard dynamic range image:
[0099] After the server or the anchor terminal determines the region of interest in the frame image, the region of interest in the frame image is filled with a solid color to reduce the resources consumed by the image pixels used in the region of interest. For details, please refer to Figure 5 , the client fills the image pixels corresponding to the region of interest in the frame image with pure color pixels, wherein the pure color pixels can be selected from black pixels, white pixels or green pixels, such as Figure 5 As shown, Figure 5 The region of interest 502 shown in the frame image 501 is filled with image pixels contained in the region of interest 502 as pure color pixels to convert the frame image 501 into the frame image 502, and the region of interest 503 in the frame image 502 is pure color pixels.
[0100] After the region of interest in the frame image is filled with a solid color, a hardware rendering algorithm is used to render the frame image from a high dynamic range imaging specification to a standard dynamic range image of a standard dynamic range imaging specification. The hardware rendering algorithm generally runs on the graphics processing unit (GPU) of the device to reduce the operating pressure of the central processing unit (CPU) of the device.
[0101] The hardware rendering algorithm generally has a conversion formula corresponding to the conversion of imaging specifications, so as to convert the frame image from high dynamic range imaging specification rendering to a standard dynamic range image of standard dynamic range imaging specifications through the conversion formula, and the conversion formula will be pushed to the audience end subsequently, so that the audience end can implement the inverse conversion of the imaging specification according to the conversion formula, and re-render the standard dynamic range image obtained after decoding into the second high dynamic range image of high dynamic range imaging specifications for output display.
[0102] Step S13, using software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and rendering the standard dynamic range image into a second high dynamic range image, so as to combine the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display:
[0103] After acquiring the first high dynamic range image and the standard dynamic range image, the software encoding is used for the first high dynamic range image to obtain corresponding high dynamic soft coding data, and the hardware encoding and decoding is used for the standard dynamic range image to obtain corresponding standard dynamic hard coding data, so as to push the high dynamic soft coding data and the standard dynamic hard coding data to the viewer end.
[0104] Since the first high dynamic range image generally only has image pixels included in the area of interest, compared with the image pixels of the standard dynamic range image, the first high dynamic range image has fewer image pixels, and the resolution of the first high dynamic range image is generally lower than that of the standard dynamic range image. Therefore, the first high dynamic range image is encoded and decoded using software running on the central processing unit (CPU) of the device, and the standard dynamic range image is encoded and decoded using hardware running on the graphics processing unit (GPU) of the device, which can save the computing resources of the central processing unit (CPU). For the server, the computing resources of the central processing unit (CPU) of the server can be saved to serve other live online services in the network live broadcast platform. For the anchor end, the computing power resources saved by the central processing unit (CPU) can ensure the stability of the device running the live broadcast service. For the audience end, the computing resources of the central processing unit (CPU) required for software decoding of the encoded data of the first high dynamic range image can be reduced to ensure that the live video stream in the audience end can be played smoothly, and the central processing unit (CPU) can run other functions of the live broadcast room online service, thereby improving the live broadcast room viewing experience of the audience end.
[0105] In addition to pushing the high dynamic soft coding data and the standard dynamic hard coding data to the audience end, the association information of the high dynamic soft coding data and the standard dynamic hard coding data will also be pushed to the audience end, the association information includes image positioning information and rendering formula information, the image positioning information is determined according to the relative position of the area where the interest area is located in the frame image, and the rendering formula information is the conversion formula used to render the frame image into the standard dynamic range image, so that after the audience end performs software decoding and hardware decoding on the high dynamic coding data and the standard dynamic coding data respectively, the rendering formula information is used to perform inverse conversion to render the decoded standard dynamic range image into a second high dynamic range image, and then according to the image positioning information, the first dynamic range image and the second high dynamic range image are merged and output to the live video stream playback window for display.
[0106] When the first high dynamic range image and the second high dynamic range image are combined and output to the live broadcast window for display, the first high dynamic range image will be positioned at the pure color area of the second high dynamic range image according to the image positioning information, and the image positioning information is determined according to the relative position of the area where the area of interest is located in the frame image.
[0107] In addition, in order to ensure that the first high dynamic range image and the second high dynamic range image can be displayed synchronously, the frame number sequence numbers corresponding to the first high dynamic range image and the frame image are stored in the first image queue, and correspondingly, the frame number sequence numbers corresponding to the second high dynamic range image and the frame image are stored in the second image queue. When the live video stream needs to output the image with the corresponding frame number, the first high dynamic range image and the second high dynamic range image corresponding to the frame number are obtained from the first image queue and the second image queue, and the first high dynamic range image and the second high dynamic range image are merged and synchronously output to the live video stream playback window for display:
[0108] Through the typical implementation of the method, it can be known that the method can output and play the image area of interest to the audience user in the live video stream with the high dynamic range graphics specification (HDR), and output and play the graphic area outside the interest with the standard dynamic range graphics specification (SDR), so that the live video content of interest to the user in the live video screen can be output and played with a better graphics specification than the live video content that the user is not interested in, thereby saving the computing resources of the device, and also allowing some viewers using devices with general configuration to watch the live video screen with the high dynamic range graphics specification (HDR); secondly, after copying the image pixels of the interest area of the frame image played in the live video stream, the method fills the image pixels of the interest area in the frame image with a solid color to reduce The resources occupied by the original image pixels in the area of interest are saved, and the memory resources occupied by the standard dynamic range image of the image pixels in the non-interest area are saved; in addition, the method uses software codec for images of the high dynamic range graphics specification (HDR), and uses hardware software codec for images of the standard dynamic range graphics specification (SDR). Software codec refers to the codec method running on the central processing unit (CPU), and hardware software codec refers to the codec method running on the display processing unit (GPU), thereby saving the computing resources of the central processing unit (CPU), so that the central processing unit (CPU) of the viewer-end device can apply the saved computing resources to other functions of the live broadcast service, ensuring that the audience users can fully and smoothly participate in the live broadcast activities of the live broadcast room.
[0109] The above typical embodiments and their variations fully disclose the implementation scheme of the live video playback method of the present application. However, various variations of the method can still be derived by changing and amplifying some technical means. Other embodiments are briefly described as follows:
[0110] In one embodiment, please refer to Figure 6 The step of determining the region of interest of the frame image in the high dynamic range image live video stream comprises the following steps:
[0111] Step S111, obtaining the currently played frame image in the high dynamic range image live video stream, and performing image preprocessing on the frame image:
[0112] After the frame image is acquired, image preprocessing is performed on the frame image, for example, preprocessing operations such as modifying the size of the frame image, dimension reduction or data expansion are performed.
[0113] Step S112, extracting the image feature vector of the frame image after image preprocessing:
[0114] After the image preprocessing of the frame image is completed, the image feature vector of the frame image is extracted.
[0115] Step S113, identifying the region of interest in the frame image according to the image feature vector:
[0116] After obtaining the image feature vector, the image feature vector is input into a fully connected layer or a convolutional layer of an image recognition model that has been trained to convergence, and normalized to identify the test pattern in the frame image.
[0117] In this embodiment, the image recognition model that has been trained to convergence is used to recognize the test pattern in the frame image through a neural network method, thereby improving the recognition efficiency of the test pattern.
[0118] In one embodiment, please refer to Figure 3 and Figure 7 The step of determining the region of interest of the frame image in the high dynamic range image live video stream comprises the following steps:
[0119] Step S111', in response to the interest region editing event acting on the live video stream playback window, obtaining the interest region positioning information corresponding to the interest region editing event:
[0120] The interest editing event generally refers to the event in which the anchor user at the anchor end edits and frames the interest area in the live video stream playback window in the live studio interface. Please refer to Figure 3The interest region can also be customized and edited by the anchor user at the anchor end. Specifically, in response to the interest region editing event acting on the live video stream playback window, the interest region positioning information corresponding to the interest region editing event is obtained, such as Figure 3 In the live broadcast room interface shown, the anchor user at the anchor end can edit the corresponding interest area in the live video stream playback window 301 shown, such as the interest area editing control 302 in the live video stream playback window 301, and edit and frame the corresponding interest area with reference to the live video stream played in the live video stream playback window 301 through the interest area editing control 302, so as to generate the interest area positioning information corresponding to the interest area edited and framed by the interest area editing control 302.
[0121] Step S112', obtaining the currently played frame image in the high dynamic range image live video stream, and determining the center point of the corresponding region of interest in the frame image according to the region of interest positioning information including the region positioning data:
[0122] The region positioning data included in the region of interest positioning information is used to characterize the position of the region of interest in the frame image. The region positioning data generally includes data of two vertical and horizontal coordinate axes, wherein the lower left corner of the frame image is generally the origin of the coordinate axis, and then the position of the center of the region of interest in the frame image is located according to the region positioning data, so as to use the position as the center point of the region of interest.
[0123] Step S113', according to the area data included in the area location information of the area of interest, determine the area of interest corresponding to the area data from the center point of the area of interest in the frame image:
[0124] The area area of the area of interest represented by the area area data contained in the area area positioning information is determined. After the center point of the area of interest where the area of interest in the frame image is located, the center point of the area of interest is used as the midpoint, and the area area represented by the area area data is determined with the center as the midpoint, and the image area contained in the area is used as the area of interest.
[0125] In this embodiment, the region of interest can be customized and edited by the anchor user, so that the region of interest that needs to be output and displayed in the live video stream with high dynamic range imaging specifications can be determined by the anchor user, allowing the anchor user to control the region of interest in the live video screen to assist its live broadcast business.
[0126] In one embodiment, please refer to Figure 4 and Figure 8 , the step of copying the region of interest as a first high dynamic range image comprises the following steps:
[0127] Step S111", obtaining the image pixels contained in the region of interest in the frame image:
[0128] Please refer to Figure 4 After determining the region of interest in the frame image, the viewer terminal will obtain the image pixels contained in the region of interest, such as Figure 4 As shown, Figure 4 As shown in Figure 4 The image pixels included in the region of interest 402 in the frame image 401 shown in FIG.
[0129] Step S112, generating a first high dynamic range image of high dynamic range imaging specifications, and merging the image pixels into the first high dynamic range image:
[0130] Please refer to Figure 3 , generating a high dynamic range image of the high dynamic range imaging specification, the newly generated first high dynamic range image generally does not include image pixels, and then merging the image pixels into the first high dynamic range image, such as Figure 4 The image pixels acquired from the region of interest 402 of the frame image 401 are merged into the first high dynamic range image 403 shown.
[0131] In this embodiment, the image pixels included in the region of interest are copied to an image generated with a high dynamic range imaging specification, so as to facilitate subsequent high dynamic range output display of the image pixels included in the region of interest.
[0132] In one embodiment, please refer to Figure 5 and Fig. 9 The step of filling the region of interest in the frame image with a solid color and rendering the frame image as a standard dynamic range image comprises the following steps:
[0133] Step S121, filling the image pixels corresponding to the region of interest in the frame image with pure color pixels:
[0134] Please refer to Figure 5 , the client fills the image pixels corresponding to the region of interest in the frame image with pure color pixels, wherein the pure color pixels can be selected from black pixels, white pixels or green pixels, such as Figure 5 As shown, Figure 5 The region of interest 502 shown in the frame image 501 is filled with image pixels contained in the region of interest 502 as pure color pixels to convert the frame image 501 into the frame image 502, and the region of interest 503 in the frame image 502 is pure color pixels.
[0135] Step S122, using a hardware rendering algorithm, rendering the frame image from a high dynamic range imaging specification to a standard dynamic range image of a standard dynamic range imaging specification:
[0136] After the region of interest in the frame image is filled with a solid color, a hardware rendering algorithm is used to render the frame image from a high dynamic range imaging specification to a standard dynamic range image of a standard dynamic range imaging specification. The hardware rendering algorithm generally runs on the graphics processing unit (GPU) of the device to reduce the operating pressure of the central processing unit (CPU) of the device.
[0137] The hardware rendering algorithm generally has a conversion formula corresponding to the conversion of imaging specifications, so as to convert the frame image from high dynamic range imaging specification rendering to a standard dynamic range image of standard dynamic range imaging specifications through the conversion formula, and the conversion formula will be pushed to the audience end subsequently, so that the audience end can implement the inverse conversion of the imaging specification according to the conversion formula, and re-render the standard dynamic range image obtained after decoding into the second high dynamic range image of high dynamic range imaging specifications for output display.
[0138] In this embodiment, the copied area of interest is filled with a solid color to reduce the resources occupied by the original image pixels in the area of interest and to reduce the bit rate of the image, thereby reducing the computing resources occupied by encoding and decoding and the transmission traffic occupied when the encoded data is transmitted, and rendering the original high dynamic range image into an image of standard dynamic range imaging specifications, which can also effectively reduce the computing resources occupied by the subsequent encoding and decoding of image pixels in non-interest areas.
[0139] In one embodiment, please refer to Fig.10 and Fig.11 The step of using software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and rendering the standard dynamic range image into a second high dynamic range image, so as to combine the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display, comprises the following steps:
[0140] Step S131, obtaining image positioning information and rendering formula information, wherein the image positioning information is determined according to the relative position of the region of interest in the frame image, and the rendering formula information is a conversion formula used to render the frame image into the standard dynamic range image:
[0141] The image positioning information is generally determined based on the relative position of the area where the interest area is located in the frame image, for example, based on the position of the regional midpoint of the interest area in the frame image, or when the interest area is edited and framed by the audience user through the live video stream playback window, the image positioning information is the interest area positioning information including regional positioning data.
[0142] The rendering formula information generally refers to the conversion formula used to convert the frame image from high dynamic range imaging specifications to standard dynamic range imaging specifications. By pushing the rendering formula information to the audience end, the audience end can reversely convert the standard dynamic range image into the second high dynamic range image of high dynamic range imaging specifications through the rendering formula information.
[0143] Step S132, using software encoding and hardware encoding for the first high dynamic range image and the standard dynamic range image respectively, to obtain high dynamic soft-coded data and standard dynamic hard-coded data:
[0144] The software encoding refers to an encoding method used for a central processing unit (CPU). Since the first high dynamic range image only has image pixels included in the area of interest in the frame image, compared to the standard dynamic range image that includes other image pixels outside the required area, the first high dynamic range image is encoded using the software encoding, while the standard dynamic range image is encoded using the hardware encoding running on a graphics processing unit (GPU), which can effectively save the computing resources consumed by the central processing unit (CPU).
[0145] Step S133, pushing the image positioning information, the rendering formula information, the high dynamic soft coding data and the standard dynamic hard coding data to the viewer end, so as to drive the viewer end to perform software decoding and hardware decoding on the high dynamic coding data and the standard dynamic coding data respectively, and perform inverse conversion according to the rendering formula information to render the decoded standard dynamic range image into a second high dynamic range image, and then combine the first dynamic range image and the second high dynamic range image according to the image positioning information, output them to the live video stream playback window for display:
[0146] After the audience receives the image positioning information, the rendering formula information, the high dynamic soft coding data and the standard dynamic hard coding data, the audience end will first perform the software decoding and the hardware decoding on the high dynamic coding data and the standard dynamic coding data respectively. The software decoding refers to a decoding method running on the central processing unit (CPU). Since the high dynamic soft coding data generally has less coding data than the standard dynamic hard coding, the high dynamic soft coding data is decoded using the software decoding, which can effectively reduce the computing pressure of the central processing unit (CPU). The standard dynamic hard coding data is decoded using the decoding method used for the graphics processing unit (GPU), so that the image processing unit (GPU) is responsible for decoding most of the coding data, thereby sharing most of the computing pressure for the central processing unit (CPU).
[0147] After the audience end decodes the high dynamic soft-coded data and the standard dynamic hard-coded data, it will obtain the first high dynamic range image and the standard dynamic range image, and then perform high dynamic range rendering on the standard dynamic range image. The audience end will reversely convert the standard dynamic range image with the imaging specifications of the standard dynamic range into the second high dynamic range image with the high standard dynamic range according to the rendering formula information.
[0148] Please refer to Fig.10 , get Fig.10 The first high dynamic range image 1001 and the second high dynamic range image 1002 shown will be merged with the standard dynamic range image 1002 according to the image positioning information to generate the new frame image 1003 shown, so as to output the new frame image 1003 to the live video stream playback window for display.
[0149] In this embodiment, software encoding and decoding is performed on high dynamic range images with less image data to reduce the computing pressure of the central processing unit of the decoding end (server or anchor end) and the encoding end (viewer end). Secondly, after the viewer end decodes and obtains the standard dynamic range image, the standard dynamic range image is rendered according to the high dynamic range imaging specification, and the first and second high dynamic range images are merged according to the image positioning information for output display to ensure the playback effect of the final live video stream.
[0150] In one embodiment, please refer to Fig.10 and Fig.12The step of using software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and rendering the standard dynamic range image into a second high dynamic range image, so as to combine the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display, comprises the following steps:
[0151] Step S131', storing the frame numbers corresponding to the first high dynamic range image and the frame image in a first image queue:
[0152] The first image queue stores a plurality of first high dynamic range images consisting of image pixels having regions of interest and their corresponding frame numbers.
[0153] Step S132', storing the frame numbers corresponding to the second high dynamic range image and the frame image in a second image queue:
[0154] The second image queue stores a plurality of second high dynamic range images consisting of image pixels other than image pixels of the area of interest and their corresponding frame numbers.
[0155] Step S133', obtaining the first high dynamic range image and the second high dynamic range image with the same frame number from the first image queue and the second image queue, merging the first high dynamic range image and the second high dynamic range image and outputting them to the live video stream playback window for display:
[0156] Please refer to Fig.10 , obtaining the target frame number sequence number of the image currently to be output and displayed, so as to obtain the first high dynamic range image stored corresponding to the target frame number sequence number from the first image queue, and obtain the second high dynamic range image stored corresponding to the target frame number sequence number from the second image queue, such as Fig.10 The first high dynamic range image 1001 and the second high dynamic range image 1002 have the same frame number sequence, so as to generate the new frame image 1003 shown by the first high dynamic range image 1001 and the second high dynamic range image 1002, so as to output the new frame image 1003 shown to the live video stream playback window for display, so as to synchronously output the first high dynamic range image 1001 and the second high dynamic range image 1002 shown to the live video stream playback window for display.
[0157] In this embodiment, the first high dynamic range image and the second high dynamic range image are stored respectively in two queues to ensure that the first high dynamic range image and the second high dynamic range image with the same frame number can be synchronously output to the live video stream playback window for display.
[0158] Furthermore, by functionalizing each step in the method disclosed in the above embodiments, a live video playback device of the present application can be constructed. Fig.13 In a typical embodiment, the device includes: an interest region copying module 11, which is used to respond to a live video stream push instruction, determine the interest region of a frame image in a high dynamic range image live video stream, and copy the interest region as a first high dynamic range image; a standard image rendering module 12, which is used to fill the interest region in the frame image with a solid color, and render the frame image as a standard dynamic range image; a merged image output module 13, which is used to use software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and render the standard dynamic range image into a second high dynamic range image, so as to merge the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display.
[0159] In one embodiment, the region of interest copy module 11 includes: an image preprocessing submodule, used to obtain a frame image currently being played in a high dynamic range image live video stream, and perform image preprocessing on the frame image; a vector extraction submodule, used to extract an image feature vector of the frame image that has completed image preprocessing; and a region of interest identification submodule, used to identify the region of interest in the frame image based on the image feature vector.
[0160] In another embodiment, the interest region copy module 11 also includes: an interest region editing submodule, which is used to respond to an interest region editing event acting on a live video stream playback window, and obtain the interest region positioning information corresponding to the interest region editing event; a region center point determination submodule, which is used to obtain a frame image currently being played in a high dynamic range image live video stream, and determine the corresponding interest region center point in the frame image based on the region positioning data included in the interest region positioning information; an interest region determination submodule, which is used to determine the interest region corresponding to the region area data from the interest region center point in the frame image based on the region area data included in the interest region positioning information.
[0161] In another embodiment, the region of interest copy module 11 also includes: an image pixel acquisition submodule, used to acquire image pixels contained in the region of interest in the frame image; an image pixel merging submodule, used to generate a first high dynamic range image of high dynamic range imaging specifications, and merge the image pixels into the first high dynamic range image.
[0162] In one embodiment, the standard image rendering module 12 includes: a pixel filling submodule, used to fill the image pixels corresponding to the area of interest in the frame image with pure color pixels; an image rendering submodule, used to use a hardware rendering algorithm to render the frame image from a high dynamic range imaging specification to a standard dynamic range image of a standard dynamic range imaging specification.
[0163] In one embodiment, the merged image output module 13 includes: an image association information acquisition submodule, used to acquire image positioning information and rendering formula information, wherein the image positioning information is determined according to the relative position of the region of interest in the frame image, and the rendering formula information is a conversion formula used to render the frame image into the standard dynamic range image; a coding data acquisition submodule, used to use software coding and hardware coding for the first high dynamic range image and the standard dynamic range image, respectively, to acquire high dynamic soft coding data and standard dynamic hard coding data; an image merging and outputting submodule, used to push the image positioning information, the rendering formula information, the high dynamic soft coding data and the standard dynamic hard coding data to the audience end, so as to drive the audience end to perform software decoding and hardware decoding on the high dynamic coding data and the standard dynamic coding data, respectively, and perform inverse conversion according to the rendering formula information to render the decoded standard dynamic range image into a second high dynamic range image, and then according to the image positioning information, merge the first dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display.
[0164] In another embodiment, the merged image output module 13 also includes: a first high dynamic image enqueue submodule, used to store the frame number serial numbers corresponding to the first high dynamic range image and the frame image in the first image queue; a second high dynamic image enqueue submodule, used to store the frame number serial numbers corresponding to the second high dynamic range image and the frame image in the second image queue; an image merging and outputting submodule, used to obtain the first high dynamic range image and the second high dynamic range image with the same frame number serial numbers from the first image queue and the second image queue, and merge the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display.
[0165] In order to solve the above technical problems, the embodiment of the present application also provides a computer device for running a computer program implemented according to the live video playback method. Fig.14 , Fig.14 This is a basic structural block diagram of the computer device in this embodiment.
[0166] like Fig.14As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a live video playback method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a live video playback method. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0167] In this embodiment, the processor is used to execute the specific functions of each module / submodule in the live video playback device of the present application, and the memory stores the program code and various data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the live video playback device, and the server can call the program code and data of the server to execute the functions of all submodules.
[0168] The present application also provides a computer-readable storage medium, and the live video playback method is written into a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is running in the computer, thereby enabling one or more processors to execute the steps of the live video playback method in any of the above-mentioned embodiments.
[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0170] In summary, the present application uses different image processing and encoding and decoding methods for the images of the areas of interest and non-interest to the user in the live video screen, thereby improving the viewing experience of the live screen while also ensuring the system operation stability of the live online service.
[0171] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0172] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.
[0173] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A live video playback method, characterized in that: The steps include: In response to a live video stream push instruction, determine an interest region of a frame image in a live video stream of a high dynamic range image, and copy the interest region as a first high dynamic range image; Filling the region of interest in the frame image with a solid color, and rendering the frame image as a standard dynamic range image; Software encoding and decoding and hardware encoding and decoding are respectively used for the first high dynamic range image and the standard dynamic range image, and the standard dynamic range image is rendered into a second high dynamic range image, so that the first high dynamic range image and the second high dynamic range image are merged and output to the live video stream playback window for display, wherein the first high dynamic range image is encoded by software to obtain corresponding high dynamic soft coding data, and the standard dynamic range image is encoded by hardware to obtain corresponding standard dynamic hard coding data, so that the high dynamic soft coding data and the standard dynamic hard coding data are pushed to the audience end, and after the audience end performs software decoding and hardware decoding on the high dynamic soft coding data and the standard dynamic hard coding data respectively, the standard dynamic range image obtained by decoding is rendered into a second high dynamic range image, and the first high dynamic range image and the second high dynamic range image are merged and output to the live video stream playback window for display.
2. The method according to claim 1, characterized in that The step of determining the region of interest of a frame image in a high dynamic range image live video stream comprises the following steps: Obtaining a currently played frame image in a high dynamic range image live video stream, and performing image preprocessing on the frame image; Extracting the image feature vector of the frame image after image preprocessing; According to the image feature vector, a region of interest in the frame image is identified.
3. The method according to claim 1, characterized in that The step of determining the region of interest of a frame image in a high dynamic range image live video stream comprises the following steps: In response to an interest region editing event acting on a live video stream playback window, obtaining interest region positioning information corresponding to the interest region editing event; Acquire a currently played frame image in a high dynamic range image live video stream, and determine a center point of the region of interest corresponding to the frame image according to the region of interest positioning information including the region positioning data; According to the region area data included in the region area positioning information, the region of interest corresponding to the region area data is determined from the center point of the region of interest in the frame image.
4. The method according to claim 1, characterized in that The step of copying the region of interest as a first high dynamic range image comprises the following steps: Acquire image pixels contained in the region of interest in the frame image; A first high dynamic range image of high dynamic range imaging specifications is generated, and pixels of the image are merged into the first high dynamic range image.
5. The method according to claim 1, characterized in that The step of filling the region of interest in the frame image with a solid color and rendering the frame image as a standard dynamic range image comprises the following steps: Filling the image pixels corresponding to the region of interest in the frame image with pure color pixels; The frame image is rendered from a high dynamic range imaging specification to a standard dynamic range image of a standard dynamic range imaging specification using a hardware rendering algorithm.
6. The method according to claim 1, characterized in that The step of using software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and rendering the standard dynamic range image into a second high dynamic range image, so as to combine the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display, comprises the following steps: Acquire image positioning information and rendering formula information, wherein the image positioning information is determined according to the relative position of the region of interest in the frame image, and the rendering formula information is a conversion formula used to render the frame image into the standard dynamic range image; Using software encoding and hardware encoding for the first high dynamic range image and the standard dynamic range image respectively to obtain high dynamic soft-coded data and standard dynamic hard-coded data; The image positioning information, the rendering formula information, the high dynamic soft coding data and the standard dynamic hard coding data are pushed to the audience end to drive the audience end to perform software decoding and hardware decoding on the high dynamic soft coding data and the standard dynamic hard coding data respectively, and perform inverse conversion according to the rendering formula information to render the decoded standard dynamic range image into a second high dynamic range image, and then according to the image positioning information, merge the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display.
7. The method according to claim 1, characterized in that The step of using software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and rendering the standard dynamic range image into a second high dynamic range image, so as to combine the first high dynamic range image and the second high dynamic range image and output them to the live video stream playback window for display, comprises the following steps: storing the frame numbers corresponding to the first high dynamic range image and the frame image in a first image queue; storing the frame numbers corresponding to the second high dynamic range image and the frame image in a second image queue; A first high dynamic range image and a second high dynamic range image with the same frame number are obtained from the first image queue and the second image queue, and the first high dynamic range image and the second high dynamic range image are merged and output to the live video stream playback window for display.
8. A live video playback device, characterized in that: include: An interest region copying module is used to respond to a live video stream push instruction, determine an interest region of a frame image in a live video stream of a high dynamic range image, and copy the interest region as a first high dynamic range image; A standard image rendering module, used to fill the region of interest in the frame image with a solid color and render the frame image into a standard dynamic range image; A merged image output module is used to use software encoding and decoding and hardware encoding and decoding for the first high dynamic range image and the standard dynamic range image respectively, and render the standard dynamic range image into a second high dynamic range image, so as to merge the first high dynamic range image and the second high dynamic range image and output them to a live video stream playback window for display, wherein software encoding is used for the first high dynamic range image to obtain corresponding high dynamic soft coding data, and hardware encoding is used for the standard dynamic range image to obtain corresponding standard dynamic hard coding data, so as to push the high dynamic soft coding data and the standard dynamic hard coding data to the audience end, and after the audience end performs software decoding and hardware decoding on the high dynamic soft coding data and the standard dynamic hard coding data respectively, the standard dynamic range image obtained by decoding is rendered into a second high dynamic range image, and the first high dynamic range image and the second high dynamic range image are merged and output to the live video stream playback window for display.
9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.
Citation Information
Patent Citations
Live broadcast processing method and device, storage medium and equipment
CN111010588A
Live streaming method and system, storage medium and equipment
CN111885393A
High dynamic range video generation method and device
CN112449120A
Video processing device and method, and computer storage medium
CN114554212A
System and method for controlling luminance during video production and broadcast
US20190253637A1
