Method and device for web client to play audio and video
By obtaining playback control parameters and hardware accelerated decoding processing, the performance problem of Web audio and video players in multiple split screens and H265 encoded videos is solved, and efficient audio and video playback is achieved, which avoids the screen loss caused by packet loss and improves the User experience.
Patent Information
- Application Number
- CN202410957955.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing Web audio and video players have insufficient performance in multi-split screen and H265 encoded video playback, which cannot meet actual needs, and fail to effectively handle packet loss, resulting in playback loss, affecting the user experience.
By obtaining playback control parameters, we can determine whether the machine hardware configuration supports hardware accelerated decoding, and render and frame extraction processing according to the CPU load to avoid packet loss detection, and directly realize playback based on the target audio and video output signal, eliminating the transcoding process and maximizing utilization Hardware resources.
It improves the audio and video playback performance of the web client, avoids the playback loss caused by packet loss, improves the user experience, reduces resource consumption and delays, and achieves smooth playback with multiple split screens.
Smart Images

Figure CN119011550A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of audio and video playback, and in particular to a method and device for playing audio and video on a web client. Background Art
[0002] As the economy and society develop rapidly, society has entered the era of digitalization and informatization. As video streaming sweeps the communication world, in order to maintain and increase user growth, content creators and streaming service providers need to provide high-quality user experience. To achieve this, they all need to embed Web audio and video players on their websites.
[0003] Web audio and video players are commonly used in browsers such as Chrome, Edge, and Firefox, and other platforms that support Web video playback. However, in some projects involving multi-screen video monitoring, pure Web audio and video players cannot meet the needs of smooth multi-screen playback. Moreover, with the popularity of H265 encoding cameras, most web browsers cannot support H265 code stream decoding and playback. Some versions of a few browsers (Firefox and Edge) support it, but the playback performance cannot meet the needs of actual projects.
[0004] Currently, existing technologies can solve the problem of client playing H265 format videos to a certain extent, but they all have the problem of large resource consumption; they cannot meet the multi-split screen (9-split screen 16-split screen) playback mode; they do not perform packet loss detection on the received video stream, and the screen will be distorted when the packet is lost. These problems have caused limitations in the use of web clients and affected the user experience. Therefore, improving the playback performance of web clients and avoiding distorted screens when the web client plays in the case of packet loss, and improving the user experience are problems that technicians in this field need to solve urgently. Summary of the invention
[0005] The present invention provides a method and device for playing audio and video on a web client, which are used to solve the defects of the prior art, improve the playing performance of the web client, avoid the web client playing distorted screen in the case of packet loss, and improve the user experience.
[0006] In a first aspect, the present invention provides a method for playing audio and video on a web client, comprising:
[0007] Get the playback control parameters of the target video point;
[0008] Based on the playback control parameters, obtaining audio and video output signals;
[0009] Determine whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding, and if so, perform hardware accelerated video decoding on the audio and video output signal; otherwise, perform CPU soft decoding on the audio and video output signal to obtain a decoded audio and video output signal;
[0010] Determine whether the CPU load of the web client is greater than a preset threshold, if so, render and extract frames for the decoded audio and video output signal to obtain a target audio and video output signal, otherwise, do not perform rendering and extract frames;
[0011] A playback window is created based on the playback control parameters, and audio and video playback is achieved based on the target audio and video output signal.
[0012] According to the method for playing audio and video on a web client, the play control parameters include: play point parameters, stream address parameters and play window information parameters.
[0013] According to the method for playing audio and video on a web client, the step of obtaining an audio and video output signal based on the playback control parameter includes:
[0014] Pull the media stream of the target video point according to the stream address parameter;
[0015] According to the playback point parameters, sequence number sorting, timestamp framing and decoding operations are performed on the media stream to obtain audio and video output signals.
[0016] According to the method for playing audio and video on a web client, after obtaining the audio and video output signal, the method further includes:
[0017] Based on the playback point parameters, it is determined whether the packet sequence numbers of the media stream are continuous. If they are not continuous, it is determined as packet loss and discarded. If they are continuous, no discard is performed.
[0018] According to the method for playing audio and video on a web client, before pulling the media stream of the target video point according to the stream address parameter, the method further includes:
[0019] Configuring the stream transmission mode based on the network status of the web client includes: judging whether the network status of the web client is good, and if so, configuring the stream transmission mode to the UDP mode, otherwise configuring it to the TCP mode.
[0020] According to the method for playing audio and video for a web client, after obtaining the target audio and video output signal, the method further includes:
[0021] It is determined whether the grayscale rate of the target audio and video output signal is greater than a preset threshold, and if so, it is discarded; otherwise, the decoded audio and video output signal is rendered by frame extraction to obtain it.
[0022] According to the method for playing audio and video on a web client, the step of determining whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding includes:
[0023] Call the system API of the web client to determine whether the machine has a graphics card installed. If a graphics card is installed, hardware accelerated decoding can be enabled.
[0024] In a second aspect, the present invention further provides an audio and video playing device for a web client, comprising:
[0025] The signaling protocol module is used to obtain the playback control parameters of the target video point;
[0026] A first processing module, used for obtaining an audio and video output signal based on the playback control parameter;
[0027] The second processing module determines whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding. If so, the audio and video output signals are subjected to hardware accelerated video decoding. Otherwise, the audio and video output signals are subjected to CPU soft decoding to obtain decoded audio and video output signals.
[0028] A rendering module is used to determine whether the CPU load of the web client is greater than a preset threshold. If so, the decoded audio and video output signal is rendered and framed to obtain a target audio and video output signal. Otherwise, rendering and framed are not performed.
[0029] The display module is used to create a playback window based on the playback control parameters and implement audio and video playback based on the target audio and video output signal.
[0030] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described methods for playing audio and video for a web client are implemented.
[0031] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for playing audio and video on a web client.
[0032] The method and device for playing audio and video on a web client provided by the present invention have the following better technical effects compared with the prior art.
[0033] The present invention obtains the playback control parameters of the target video point, and then obtains the audio and video output signal. It is judged whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding. If so, the audio and video output signal is subjected to hardware accelerated video decoding, otherwise the audio and video output signal is subjected to CPU soft decoding to obtain the decoded audio and video output signal; it is judged whether the CPU load of the web client is greater than the preset threshold value. If so, the decoded audio and video output signal is rendered and framed, and after obtaining the target audio and video output signal, a playback window is created based on the playback control parameters, and audio and video playback is realized based on the target audio and video output signal. The transcoding link is omitted in the overall architecture design, which greatly saves the resource consumption and playback delay caused by transcoding. Extra server resources are reduced. By automatically detecting the design of the machine hardware configuration, the resources of the machine itself are maximized to improve the decoding performance. The packet loss detection in the framing stage and the rendering stage avoids the occurrence of the playback screen phenomenon to the greatest extent, improves the audio and video playback quality and the adaptability of the audio and video playback plug-in, and thus greatly improves the performance of the web player. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 It is a schematic diagram of the principle of deploying a transcoding gateway provided by the present invention to realize web browser decoding and playback of H265;
[0036] Figure 2 It is a schematic diagram of the principle of installing a decoding plug-in provided by the present invention to realize web browser decoding and playing H265;
[0037] Figure 3 This is one of the flow charts of the method for playing audio and video by a web client provided by the present invention;
[0038] Figure 4 It is a schematic diagram of the principle of the audio and video playback plug-in provided by the present invention for playing audio and video for a web client;
[0039] Figure 5 This is a second flow chart of the method for playing audio and video on a web client provided by the present invention;
[0040] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0043] The following is a brief description of the prior art involved in this solution:
[0044] Web audio and video players are commonly used in browsers such as Chrome, Edge, and Firefox, and other platforms that support Web video playback. However, in some projects involving multi-screen video monitoring, pure Web audio and video players cannot meet the needs of smooth multi-screen playback. Moreover, with the popularity of H265 encoding cameras, most web browsers cannot support H265 code stream decoding and playback. Some versions of a few browsers (Firefox and Edge) support it, but the playback performance cannot meet the needs of actual projects.
[0045] To solve the problem of web browser decoding and playing H265, one of the existing technical solutions (Solution 1) is to deploy a transcoding gateway between the web client and the streaming media server. Figure 1 Schematic diagram of the principle of deploying a transcoding gateway to realize web browser decoding and playing H265 provided by the present invention, such as Figure 1As shown, the transcoding gateway can transcode H265 code stream into H264, or transcode high-bitrate video stream into low-bitrate video stream to solve the problem of browser unfriendly support for H265 decoding and insufficient decoding performance. The Web client applies for media stream from the streaming media service platform. When the video stream encoding format is H264, the streaming media service platform directly sends the video code stream to the Web client; when the video encoding format is H265, the streaming media service platform will first send the code stream to the transcoding gateway, and the transcoding gateway will transcode the code stream format from H265 to H264, and then the transcoding gateway will send the transcoded video stream to the Web client. The Web client decodes the received H264 format video and plays the video.
[0046] Another technique is to install a decoding plug-in (Solution 2). Figure 2 Schematic diagram of the principle of installing a decoding plug-in provided by the present invention to realize web browser decoding and playing H265, such as Figure 2 As shown in the figure, the plug-in is responsible for decoding and converting the decoded data into jpeg and sending it to the web end for rendering through web socket. The web client requests the media stream from the streaming service platform, and the streaming service platform sends the media stream to the decoding plug-in for processing. The decoding plug-in decodes the received stream and then encodes it into jpeg format. Finally, the jpeg video frame is sent to the web client through web socket. The web client directly renders the received jpeg video frame and plays the video.
[0047] Although Solution 1 can solve the problem that web clients cannot play H265 videos, deploying a transcoding gateway between the streaming media server and the web client will increase the playback delay of the web client, and the resources consumed by the transcoding gateway are also huge. Because in the process of transcoding the H265 format video stream into the H264 format video stream, it is necessary to decode the H265 code stream into the original video data (YUV format), and then encode the original video data in the YUV format into the H264 format, and both the decoding and encoding processes are time-consuming. In addition, video encoding and decoding itself consumes more CPU and memory resources. If the number of concurrent video streams to be transcoded is relatively large, the resource requirements for the server will be relatively high, increasing the construction cost.
[0048] Compared with Solution 1, Solution 2 mentioned above saves the need for a transcoding gateway. Instead, a decoding plug-in is installed on the web client, which decodes the data and then transcodes the decoded data into jpeg. Finally, the jpeg image data is sent to the web client through websocket, and finally rendered into the playback window by the web client. However, this introduces new problems. Since the jpeg format has a high degree of compression for video data, the displayed picture clarity is low when playing in full screen. On the other hand, re-encoding the decoded data into jpeg is also an additional consumption of resources on the client machine.
[0049] It can be seen that the two existing technical solutions have solved the problem of playing H265 format videos on the client to a certain extent, but both have the problem of large resource consumption; they cannot meet the multi-split screen (9-split screen 16-split screen) playback mode; they do not perform packet loss detection on the received video stream, and the screen will be distorted when the packet is lost. These problems have caused limitations in the use of web clients and affected the user experience. Therefore, improving the playback performance of web clients and avoiding distorted screens when the web client is playing in the case of packet loss, and improving the user experience are problems that need to be solved by technicians in this field.
[0050] The present invention provides a method and device for playing audio and video on a web client, so as to improve the playing performance and user experience of the web client. Figure 3-Figure 6 The present invention provides a method and device for playing audio and video on a web client.
[0051] Figure 3 FIG. 1 is one of the flow charts of the method for playing audio and video on a web client provided by the present invention. The following is an example of the method being applied to an audio and video playing plug-in. Figure 3 As shown, including but not limited to the following steps (STEP, referred to as S):
[0052] S1: Obtain the playback control parameters of the target video point.
[0053] Among them, the playback control parameters include playback point parameters, stream address parameters and playback window information parameters.
[0054] In this embodiment, Figure 4 Schematic diagram of the principle of the audio and video playback plug-in provided by the present invention for playing audio and video for a web client, such as Figure 4As shown in the figure, the web client selects a certain video point, applies for the audio and video stream of the point from the streaming media service platform, and obtains the rtsp address of the audio and video stream of the point. Then the web client calls the audio and video playback interface of the audio and video playback plug-in through the http protocol, carrying relevant playback control parameters including playback point parameters, stream address parameters, and playback window information parameters.
[0055] S2: Based on the playback control parameters, obtain audio and video output signals.
[0056] After receiving the rtsp audio and video stream address url of point P1 returned by the streaming media service platform, the web client calls the audio and video playback interface of the audio and video playback plug-in through the http protocol to play the audio and video stream of point P1 and specify the playback control parameters related to point P1, including the playback point id, stream address url, and playback window information window Info.
[0057] After receiving the http call message from the Web client, the audio and video playback plug-in parses the playback control parameters of point P1 carried in the http message, and gives each audio and video stream an identity id according to the playback point parameters. The identity id is used to detect whether it is repeated with other playback video streams to prevent repeated on-demand of the same point. Then, according to the stream address url, the rtsp audio and video stream of point P1 is pulled from the streaming media service platform through the rtsp protocol.
[0058] S3: Determine whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding. If so, perform hardware accelerated video decoding on the audio and video output signals. Otherwise, perform CPU soft decoding on the audio and video output signals to obtain decoded audio and video output signals.
[0059] In this embodiment, after obtaining the rtsp audio and video stream of point P1, it is detected whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding. If it supports, it is enabled to perform hardware accelerated video decoding. If it does not support, software decoding is performed. The detection of the machine hardware configuration can be performed by, for example, calling the system API to determine whether the machine is equipped with a graphics card. If a graphics card is installed, hardware accelerated decoding can be enabled.
[0060] S4: Determine whether the CPU load of the web client is greater than a preset threshold. If so, render and extract frames for the decoded audio and video output signal to obtain a target audio and video output signal. Otherwise, do not perform rendering and extract frames.
[0061] In this embodiment, when it is detected that the CPU load of the web client is large, for example, exceeding 90%, it can be set to trigger a rendering frame extraction strategy to determine whether the CPU load of the web client is greater than a preset threshold. If so, the decoded audio and video output signal is rendered and frame extracted to obtain the target audio and video output signal. Otherwise, rendering and frame extraction are not performed to reduce the resource consumption of each video channel and ensure smooth playback of multiple videos.
[0062] The present invention can improve the audio and video playback quality and the adaptability of the audio and video playback plug-in by judging whether the hardware configuration supports hardware accelerated decoding and then judging the CPU load of the web client, thereby greatly improving the performance of the web player.
[0063] When the CPU load is high, rendering and extracting frames of the decoded audio and video output signals can reduce the amount of calculation in the rendering process, thereby reducing the CPU load, reducing the resource consumption of each video channel, and ensuring smooth playback of multiple videos.
[0064] Hardware accelerated decoding can increase the decoding speed and reduce errors in the decoding process, thereby improving the quality of audio and video playback. At the same time, rendering frame extraction can reduce image freezes and delays, improving user experience.
[0065] By dynamically adjusting the decoding and rendering strategies according to the client's hardware configuration and CPU load, the audio and video playback plug-in has strong adaptability and can achieve better audio and video playback effects in different devices and network environments.
[0066] S5: Creating a playback window based on the playback control parameters, and implementing audio and video playback based on the target audio and video output signals.
[0067] In this embodiment, a play window is created based on the play control parameters, and audio and video play is achieved based on the target audio and video output signal. There are two modes for use with the Web client: one is that the Web client provides the window coordinates, and the audio and video plug-in device embeds the rendering window in the Web client window according to the window coordinates; the other is that the Web client provides the window coordinate position, and the audio and video plug-in device creates its own rendering window floating on the Web client according to the corresponding call parameters.
[0068] Based on the contents of the above embodiments, as an optional embodiment, the step of obtaining the audio and video output signal based on the playback control parameters in the audio and video playback method for a web client provided by the present invention includes: pulling the media stream of the target video point according to the stream address parameters; performing serial number sorting, timestamp framing and decoding operations on the media stream according to the playback point parameters to obtain the audio and video output signal.
[0069] In this embodiment, after the audio and video playback plug-in receives the http call message from the Web client, it parses the point P1 playback parameters carried in the http message, and gives each audio and video stream an identity id according to the playback point parameters. The identity id is used to detect whether it is repeated with other playback video streams to prevent repeated on-demand viewing of the same point; according to the stream address parameter url, the rtsp audio and video stream of point P1 is pulled from the streaming media service platform through the rtsp protocol; according to the playback window information parameter windowInfo, a playback window is created and its display style is determined.
[0070] Then, after the streaming media service platform receives the rtsp audio and video stream pulling message of point P1 from the audio and video playback plug-in device, the rtsp stream of point P1 is sent to the receiving port of the audio and video playback plug-in device. The audio and video playback plug-in sorts the received media stream according to the size of the rtp packet number, frames it according to the timestamp, and then decodes the assembled audio and video frame data.
[0071] Furthermore, after obtaining the audio and video output signal in the audio and video playing method for a web client provided by the present invention, it also includes: judging whether the packet sequence number of the media stream is continuous based on the playback point parameter, if not continuous, determining it as packet loss and discarding it, and if continuous, not discarding it.
[0072] In this embodiment, if the packet sequence number is discontinuous, it can be determined as packet loss. In this case, the frame containing the packet loss is discarded and not decoded to avoid decoding distortion (if a frame of data is incompletely lost, the picture generated by forced decoding will be distorted).
[0073] Based on the content of the above embodiment, as an optional embodiment, the method for playing audio and video for a web client provided by the present invention further includes before pulling the media stream of the target video point according to the stream address parameter:
[0074] Configuring the stream transmission mode based on the network status of the web client includes: judging whether the network status of the web client is good, and if so, configuring the stream transmission mode to the UDP mode, otherwise configuring it to the TCP mode.
[0075] In this embodiment, Figure 5 FIG. 2 is a flow chart of the method for playing audio and video by a web client provided by the present invention. Figure 5As shown in the figure, the web client selects the P1 video point and applies for the audio and video stream of the point from the streaming service platform. The streaming service platform returns the rtsp audio and video stream address of the P1 point to the web client. An example of the rtsp stream address is: "rtsp: / / 127.0.0.1:554 / rtp / 41010500001110000002_34020000001320000003".
[0076] The Web client can call the audio and video playback interface of the audio and video playback plug-in through the http protocol, carrying parameters including the playback point parameter id, the stream address parameter url, and the playback window information parameter windowInfo. The id example is as follows: 34020000001320000003, the stream address url is as described above, and the windowInfo example is as follows: {width=352; height=288; x=900; y=100; handle=HWND;}, wherein width represents the width of the playback window, height represents the height of the playback window, x represents the horizontal coordinate position of the upper left vertex of the playback window on the display screen, y represents the vertical coordinate position of the upper left vertex of the playback window on the display screen, and handle represents the method of window creation. If the handle value is not empty, the audio and video plug-in device creates a window according to the value provided by the handle and is nested in the Web client. If the handle value is empty, the audio and video plug-in device automatically creates a playback window floating on the Web client.
[0077] After the http call message from the web side, parse the playback window information windowInfo, create a playback window at the specified coordinate position of the web client, determine the window display style, and pull the rtsp media stream at point P1 from the streaming media service platform through the given rtsp stream address url. The streaming mode can be configured as, for example, UDP or TCP mode. When the network condition is good, it is configured to UDP mode, and when the network condition is poor, it is configured to TCP mode. The streaming media service platform negotiates the streaming transmission mode and the receiving IP and port according to the rtsp pull request sent by the audio and video playback plug-in device, and sends the rtsp audio and video stream at point P1 to the specified IP and port. For example, if the negotiated stream transmission mode is UDP, the receiving IP is: 127.0.0.1 and port 5000; the stream transmission mode is TCP, and the receiving IP is: 127.0.0.1 and port 5002
[0078] The audio and video playback plug-in sorts the received media stream according to the size of the RTP packet sequence number, groups the frames according to the timestamp, and then sends the grouped audio and video frame data to the decoding module for decoding. If the packet sequence number is not continuous, it can be determined as packet loss. In this case, the frame containing the packet loss will be discarded and not decoded. For example, the audio and video playback plug-in device receives the following RTP packet in the UDP stream transmission mode:
[0079] rtp_101, rtp_96, rtp_97, rtp_98, rtp_99, the numbers in the names of this example indicate the order in which the streaming media service platform sends them, and the order in which they are listed indicates the order in which the audio and video playback plug-in device receives the packets. It can be seen that due to network fluctuations, the rtp101 packet is the last one sent by the streaming media service platform, but it is the first one received by the audio and video playback plug-in device. At this time, the audio and video plug-in device will sort them, and then parse the audio and video data therein, and combine them into a complete frame of data frame1 according to the timestamp. In some cases, for example, the audio and video playback plug-in device receives the following rtp packets: rtp_101, rtp_96, rtp_97, rtp_99. At this time, after they are sorted and framed, it will be found that the rtp packet with sequence number 98 has not been received. At this time, it can be determined that frame1 is an incomplete frame. If forced decoding and rendering are performed, a distorted screen will appear, so frame1 will be discarded and not decoded and displayed.
[0080] Based on the content of the above embodiment, as an optional embodiment, after obtaining the target audio and video output signal provided by the present invention, the method further includes:
[0081] It is determined whether the grayscale rate of the target audio and video output signal is greater than a preset threshold, and if so, it is discarded; otherwise, the decoded audio and video output signal is rendered by frame extraction to obtain it.
[0082] In this embodiment, after obtaining the target audio and video output signal, it is determined whether the grayscale rate of the target audio and video output signal is greater than a preset threshold, that is, the decoded video data is subjected to grayscale detection. If the grayscale rate is greater than or equal to the set threshold, it is determined that the content of the frame of video data is missing, and it is discarded and not rendered. If the grayscale rate is less than the set threshold, rendering is performed. If the grayscale rate is less than the set threshold, rendering is performed. The above-mentioned grayscale detection may refer to the detection of the grayscale channel of the decoded YUV data. The basis for the detection is that if the video source data is lost, the ffmpeg decoding will fill the grayscale channel of the decoded YUV data with 128, and the effect of displaying it on the player is a flower screen effect.
[0083] Therefore, based on this feature, we can determine whether the frame data is missing by calculating the percentage of the number of pixel grayscale channel values of 128 in the entire data. For example, if the grayscale threshold is set to 80%, if the decoded frame data has a grayscale channel value of 128 of not less than 80%, then it can be determined that the frame is a defective frame and it will be discarded and not rendered.
[0084] In summary, the overall architecture design of the present invention eliminates the transcoding link, greatly saving resource consumption and playback delay caused by transcoding. It reduces additional server resources. By automatically detecting the hardware configuration of the machine, the machine's own resources are maximized to improve decoding performance. Packet loss detection in the framing stage and rendering stage can avoid the occurrence of screen distortion to the greatest extent.
[0085] The present invention also provides an audio and video playing device for a web client, comprising:
[0086] The signaling protocol module is used to obtain the playback control parameters of the target video point;
[0087] A first processing module, used for obtaining an audio and video output signal based on the playback control parameter;
[0088] The second processing module determines whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding. If so, the audio and video output signals are subjected to hardware accelerated video decoding. Otherwise, the audio and video output signals are subjected to CPU soft decoding to obtain decoded audio and video output signals.
[0089] A rendering module is used to determine whether the CPU load of the web client is greater than a preset threshold. If so, the decoded audio and video output signal is rendered and framed to obtain a target audio and video output signal. Otherwise, rendering and framed are not performed.
[0090] The display module is used to create a playback window based on the playback control parameters and implement audio and video playback based on the target audio and video output signal.
[0091] It should be noted that the apparatus for playing audio and video for a web client provided in an embodiment of the present invention can execute the method for playing audio and video for a web client described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.
[0092] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the audio and video playing method for the web client, including: obtaining the playback control parameters of the target video point; obtaining the audio and video output signal based on the playback control parameters; judging whether the hardware configuration of the machine where the web client is located supports video decoding hardware acceleration, if so, performing hardware accelerated video decoding on the audio and video output signal, otherwise performing CPU soft decoding on the audio and video output signal to obtain the decoded audio and video output signal; judging whether the CPU load of the web client is greater than a preset threshold, if so, performing rendering and frame extraction on the decoded audio and video output signal to obtain the target audio and video output signal, otherwise rendering and frame extraction is not performed; creating a playback window based on the playback control parameters, and realizing audio and video playback based on the target audio and video output signal.
[0093] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the audio and video playback method for a web client provided in the above-mentioned embodiments, including: obtaining playback control parameters of a target video point; based on the playback control parameters, obtaining an audio and video output signal; determining whether the hardware configuration of a machine where the web client is located supports hardware acceleration of video decoding, and if so, performing hardware accelerated video decoding on the audio and video output signal, otherwise performing CPU soft decoding on the audio and video output signal to obtain a decoded audio and video output signal; determining whether the CPU load of the web client is greater than a preset threshold, and if so, performing rendering and frame extraction on the decoded audio and video output signal to obtain a target audio and video output signal, otherwise no rendering and frame extraction is performed; creating a playback window based on the playback control parameters, and implementing audio and video playback based on the target audio and video output signal.
[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the audio and video playback method for a web client provided in the above-mentioned embodiments, including: obtaining playback control parameters of a target video point; obtaining an audio and video output signal based on the playback control parameters; determining whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding, and if so, performing hardware accelerated video decoding on the audio and video output signal, otherwise performing CPU soft decoding on the audio and video output signal to obtain a decoded audio and video output signal; determining whether the CPU load of the web client is greater than a preset threshold, and if so, performing rendering and frame extraction on the decoded audio and video output signal to obtain a target audio and video output signal, otherwise no rendering and frame extraction is performed; creating a playback window based on the playback control parameters, and implementing audio and video playback based on the target audio and video output signal.
[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for playing audio and video for a web client, characterized in that: include: Get the playback control parameters of the target video point; Based on the playback control parameters, obtaining audio and video output signals; Determine whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding, and if so, perform hardware accelerated video decoding on the audio and video output signal; otherwise, perform CPU soft decoding on the audio and video output signal to obtain a decoded audio and video output signal; Determine whether the CPU load of the web client is greater than a preset threshold, if so, render and extract frames for the decoded audio and video output signal to obtain a target audio and video output signal, otherwise, do not perform rendering and extract frames; A playback window is created based on the playback control parameters, and audio and video playback is achieved based on the target audio and video output signal.
2. The method for playing audio and video for a web client according to claim 1, characterized in that: The playback control parameters include: playback point parameters, stream address parameters and playback window information parameters.
3. The method for playing audio and video for a web client according to claim 2, characterized in that: The step of obtaining the audio and video output signal based on the playback control parameter comprises: Pull the media stream of the target video point according to the stream address parameter; According to the playback point parameters, sequence number sorting, timestamp framing and decoding operations are performed on the media stream to obtain audio and video output signals.
4. The method for playing audio and video for a web client according to claim 3, characterized in that: After obtaining the audio and video output signals, the method further includes: Based on the playback point parameters, it is determined whether the packet sequence numbers of the media stream are continuous. If they are not continuous, it is determined as packet loss and discarded. If they are continuous, no discard is performed.
5. The method for playing audio and video for a web client according to claim 3, characterized in that: Before pulling the media stream of the target video point according to the stream address parameter, the method further includes: Configuring the stream transmission mode based on the network status of the web client includes: judging whether the network status of the web client is good, and if so, configuring the stream transmission mode to the UDP mode, otherwise configuring it to the TCP mode.
6. The method for playing audio and video for a web client according to claim 1, characterized in that: After obtaining the target audio and video output signal, the method further includes: It is determined whether the grayscale rate of the target audio and video output signal is greater than a preset threshold, and if so, it is discarded; otherwise, the decoded audio and video output signal is rendered by frame extraction to obtain it.
7. The method for playing audio and video for a web client according to claim 1, characterized in that: The step of determining whether the hardware configuration of the machine where the web client is located supports video decoding hardware acceleration includes: Call the system API of the web client to determine whether the machine has a graphics card installed. If a graphics card is installed, hardware accelerated decoding can be enabled.
8. An audio and video playing device for a web client, characterized in that: include: The signaling protocol module is used to obtain the playback control parameters of the target video point; A first processing module, used for obtaining an audio and video output signal based on the playback control parameter; The second processing module determines whether the hardware configuration of the machine where the web client is located supports hardware acceleration of video decoding. If so, the audio and video output signals are subjected to hardware accelerated video decoding. Otherwise, the audio and video output signals are subjected to CPU soft decoding to obtain decoded audio and video output signals. A rendering module is used to determine whether the CPU load of the web client is greater than a preset threshold. If so, the decoded audio and video output signal is rendered and framed to obtain a target audio and video output signal. Otherwise, rendering and framed are not performed. The display module is used to create a playback window based on the playback control parameters and implement audio and video playback based on the target audio and video output signal.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for playing audio and video for a web client as described in any one of claims 1 to 7 are implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for playing audio and video for a web client as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Picture playing method and device as well as set-top box
CN104661059A
Video playing control method and device as well as terminal equipment
CN106713937A
Data processing method and device, electronic equipment, system and storage medium
CN110620959A
Frame rate adjusting method and device
CN112506653A
Video playing method, device and system based on HTML5, medium and browser
CN112788431A
Cited By
Multi-graphics card spliced screen control method and device and computer program product
CN120295593A
A multi-graphics card splicing screen control method, device and computer program product
CN120295593B
