Video encoding method, device, apparatus and computer readable storage medium

Abstract

The present invention relates to the field of video processing technology, and discloses a video encoding method, equipment, device and computer-readable storage medium. The method comprises: obtaining a frame interval between a frame to be encoded and a previous I frame, and judging whether the frame interval is equal to a preset I frame interval, monitoring whether an I frame request fed back by a terminal device according to a live video stream is received, and judging whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result; because the present invention encodes the frame to be encoded as an I frame only when the frame interval between the frame to be encoded and the previous I frame is equal to the preset I frame interval, or when an I frame request fed back by a terminal device according to a live video stream is received, the interval between the I frames is increased, and the I frame can be dynamically inserted based on a user request, thereby reducing the amount of transmission data of the video stream, greatly reducing or even avoiding the freeze and screen distortion of the live screen, and improving the user experience.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video encoding method, device, apparatus and computer-readable storage medium. Background Art

[0002] In the encoding process of existing live broadcast director solutions, in order to decode new video data sources in time through IDR / I frames when the program director switches to a new program, the IDR / I frame (GOP) interval is often set very small, such as 1s to 2s. However, the IDR / I frame is a complete image, so compared with the P frame / B frame in the GOP, the data volume of the IDR / I frame is very large, which can easily cause network congestion or packet loss, and the receiving end takes time or delays to process a large amount of data, resulting in the video receiving end being prone to screen freezes and distorted screens.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a video encoding method, device, apparatus and computer-readable storage medium, aiming to solve the technical problem in the prior art that the IDR / I frame (GOP) interval is set very small, resulting in the video receiving end being prone to screen freeze and screen distortion under low network speed conditions.

[0005] To achieve the above object, the present invention provides a video encoding method, which comprises the following steps:

[0006] Obtaining a frame interval between a frame to be encoded and a previous I frame, and determining whether the frame interval is equal to a preset I frame interval;

[0007] Monitor whether an I frame request based on the live video stream feedback from the terminal device is received;

[0008] It is determined whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result.

[0009] Optionally, the step of obtaining a frame interval between a frame to be encoded and a previous I frame, and determining whether the frame interval is equal to a preset I frame interval, includes:

[0010] Acquire a previous I frame, and detect whether the previous I frame is an inserted I frame, where the inserted I frame is an I frame inserted based on a preset I frame interval;

[0011] If so, the frame interval between the frame to be encoded and the previous I frame is obtained, and it is determined whether the frame interval is equal to the preset I frame interval.

[0012] Optionally, after the step of obtaining the previous I frame and detecting whether the previous I frame is an inserted I frame, the method further includes:

[0013] If not, get the previous inserted I frame;

[0014] A frame interval between a frame to be encoded and a previous inserted I frame is obtained, and it is determined whether the frame interval is equal to a preset I frame interval.

[0015] Optionally, before the step of obtaining the frame interval between the frame to be encoded and the previous I frame, and determining whether the frame interval is equal to the preset I frame interval, the step further includes:

[0016] Obtaining an initial I frame interval, and determining whether the initial I frame interval is within a preset interval;

[0017] The initial threshold is adjusted according to the judgment result to obtain a preset I frame interval.

[0018] Optionally, the step of monitoring whether an I frame request fed back by the terminal device according to the live video stream is received includes:

[0019] When the video data source of the live video stream is a single channel, the receiving terminal device feeds back the playback failure information according to the live video stream;

[0020] According to the playback failure information, monitor whether an I frame request fed back by the terminal device according to the live video stream is received.

[0021] Optionally, the step of monitoring whether an I frame request fed back by the terminal device according to the live video stream is received includes:

[0022] When the video data source of the live video stream is multiple, the receiving terminal device switches the source information fed back by the live video stream;

[0023] Monitor whether an I frame request fed back by the terminal device according to the live video stream is received according to the source switching information.

[0024] Optionally, the step of judging whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result includes:

[0025] When the frame interval is equal to the preset I frame interval, or when an I frame request is received from the terminal device according to the live video stream feedback, it is determined that the frame to be encoded is encoded as an I frame;

[0026] When the frame interval is not equal to the preset I frame interval and no I frame request is received from the terminal device according to the live video stream feedback, it is determined not to encode the frame to be encoded as an I frame.

[0027] In addition, to achieve the above-mentioned purpose, the present invention also proposes a video encoding device, which includes: a central processing unit and a graphics processing unit; the video encoding device includes a memory, a processor, and a video encoding program stored in the memory and executable on the processor, and the video encoding program is configured to implement the video encoding method as described above.

[0028] In addition, to achieve the above-mentioned purpose, the present invention further proposes a computer-readable storage medium, on which a video encoding program is stored, and when the video encoding program is executed by a processor, the video encoding method as described above is implemented.

[0029] In addition, to achieve the above-mentioned purpose, the present invention further proposes a video encoding device, the video encoding device comprising: the video encoding device comprising: an interval determination module, a request monitoring module and a frame encoding module;

[0030] The interval determination module is used to obtain a frame interval between a frame to be encoded and a previous I frame, and determine whether the frame interval is equal to a preset I frame interval;

[0031] The request monitoring module is used to monitor whether an I frame request fed back by the terminal device according to the live video stream is received;

[0032] The frame encoding module is used to determine whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result.

[0033] The present invention discloses obtaining the frame interval between the frame to be encoded and the previous I frame, judging whether the frame interval is equal to the preset I frame interval, monitoring whether an I frame request according to the feedback of the terminal device based on the live video stream is received, and judging whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result; because the present invention encodes the frame to be encoded as an I frame only when the frame interval between the frame to be encoded and the previous I frame is equal to the preset I frame interval, or when the I frame request according to the feedback of the terminal device based on the live video stream is received, the interval between the I frames is increased, and the I frame can be dynamically inserted based on the user request, thereby reducing the transmission data volume of the video stream, greatly reducing or even avoiding the freeze and screen distortion of the live screen, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a structural diagram of a video encoding device in a hardware operating environment involved in an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a flow chart of a first embodiment of a video encoding method according to the present invention;

[0036] Figure 3A schematic diagram of the location of a live broadcast acquisition device according to an embodiment of a video encoding method of the present invention;

[0037] Figure 4 is a schematic diagram of a flow chart of a second embodiment of a video encoding method according to the present invention;

[0038] Figure 5 is a schematic diagram of a flow chart of a third embodiment of a video encoding method according to the present invention;

[0039] Figure 6 FIG. 1 is a schematic diagram of image block division according to an embodiment of a video encoding method of the present invention.

[0040] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0042] Reference Figure 1 , Figure 1 The present invention is a schematic diagram of the structure of a video encoding device in a hardware operating environment according to an embodiment of the present invention.

[0043] like Figure 1 As shown, the video encoding device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), and the user interface 1003 may also include a standard wired interface and a wireless interface. The wired interface of the user interface 1003 may be a USB interface in the present invention. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable memory (Non-volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0044] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the video encoding device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0045] like Figure 1 As shown, the memory 1005 identified as a computer storage medium may include an operating system, a network communication module, a user interface module, and a video encoding program.

[0046] exist Figure 1 In the video encoding device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the user device; the video encoding device calls the video encoding program stored in the memory 1005 through the processor 1001, and executes the video encoding method provided by the embodiment of the present invention.

[0047] Based on the above hardware structure, an embodiment of the video encoding method of the present invention is proposed.

[0048] Reference Figure 2 , Figure 2 The first embodiment of the video encoding method of the present invention is provided.

[0049] Step S10: obtaining a frame interval between the frame to be encoded and the previous I frame, and determining whether the frame interval is equal to a preset I frame interval.

[0050] It should be noted that the executor of the method of this embodiment can be a live broadcast acquisition device with data processing, network communication and program running functions, such as a camera, etc., or other electronic devices that can achieve the same or similar functions, and this embodiment is not limited to this.

[0051] It should be understood that the method can be applied to scenarios where video data needs to be transmitted in the network, such as live broadcasting, video conferencing, and online classes, and this embodiment is not limited to this.

[0052] It is understandable that with the development of science and technology, digital video technology has been increasingly widely used in the fields of communication, live broadcasting, and remote conferencing. Especially since the 1990s, with the rapid development of the Internet and mobile communications, the processing and transmission of video information and multimedia information in the Internet network and mobile network has become a hot technology in my country's current informatization. Video information has a series of advantages, such as intuitiveness, accuracy, efficiency, and extensiveness. However, the amount of video information is too large. If video is to be effectively used, the problem of video compression encoding must be solved first, followed by the problem of ensuring the quality of compressed video, while ensuring the reliability and stability of network transmission of video data.

[0053] In existing professional live broadcast director applications, video is generally collected using proprietary data acquisition equipment (such as cameras). The direct-output video data is usually in YUV420 / 422 / 444, YUYV, NV12, NV21 and other video formats. The data is compressed through the encoding and compression function of the device into new data with a very small data volume, such as H264 / HEVC / H266, etc., and then packaged and encapsulated according to the transmission protocol, such as RTP, and transmitted to the media server. Here, the media server is a general server that includes functions such as clustering or transcoding enhanced distribution. After the director completes the requirements through the director operation, the user can access and browse the video according to the specified URL.

[0054] With the help of modern encoding, decoding and network communication technologies, the current live broadcast director has also achieved good development and experience. In the encoding link of the existing live broadcast director solution, in order to decode the new video data source in time through the IDR / I frame when the program director switches to a new program, the IDR / I frame (GOP) interval is often set to be very small, such as 1s to 2s. However, IDR / I is a complete frame of image. Compared with P frame and B frame, the data volume is very large. Even if the modern network is very developed, due to the differences in network equipment conditions in different regions and unpredictable influences in the network, the data transmission of large amounts of data is more likely to cause packet loss, and the user's storage device is required to be higher. Whether it is the sending end or the receiving end, there are requirements for whether the processor can process large amounts of data in time. The processor with insufficient performance may not be able to process large amounts of data in time, resulting in data loss, and image delay. Moreover, with the development trend of higher and higher video quality, the adaptability is getting worse and worse. Even if various remediation mechanisms such as FEC / packet loss retransmission / NACK are used at the end, the large amount of data is easy to cause the remediation of the data itself, and the remediation effect is not good, especially under the future 4K / 8K and other large resolution data conditions, it is more difficult to remedy. In addition, the picture is prone to visible convergence and jumping phenomena, which makes the user experience very poor.

[0055] In order to overcome the above defects, in this embodiment, the frame to be encoded is encoded as an I frame only when the frame interval between the frame to be encoded and the previous I frame is equal to the preset I frame interval, or when an I frame request is received from the terminal device based on the live video stream feedback, thereby increasing the interval between I frames, greatly reducing the data transmission volume, reducing or even avoiding the freeze and screen distortion of the live screen caused by packet loss or a large number of data processing errors, and at the same time, it can dynamically insert I frames based on user requests, solving the problem of the video source not decoding or switching the IDR / I frames required by the video source, thereby improving the user experience.

[0056] It should be noted that the frame to be encoded may be a video frame to be encoded in a video source.

[0057] For ease of understanding, the following examples are given, but this solution is not limited. The video source acquisition process steps are as follows:

[0058] (1) Arrange the location of the live broadcast acquisition equipment. The location needs to have a good field of view and picture composition for the live broadcast scene, and fix the equipment;

[0059] (2) Open the device to collect data;

[0060] Usually, Windows uses VFW or DirectShow to collect real-time video, Linux uses the v4l2 standard interface to collect, Android uses the Android API to collect, and iOS or other systems have corresponding platform collection methods, which will not be described in detail here.

[0061] The video data after acquisition is usually in the form of YUV420 / 422 / 444, YUYV, NV12, NV21 and other video formats, and the data volume is very large.

[0062] For ease of understanding, refer to Figure 3 This is for illustration only, but does not limit this solution. Figure 3 This is a schematic diagram of the location of live broadcast collection equipment. In the figure, eight live broadcast collection devices Cam1 to 8 are arranged around the basketball court, and the layout positions are shown in the figure.

[0063] It should be noted that the frame interval between the frame to be encoded and the previous I frame may be the frame interval between the frame to be encoded and the previous inserted I frame, or the frame interval between the frame to be encoded and the previous requested inserted I frame. The inserted I frame may be an I frame inserted based on a preset I frame interval, and the requested inserted I frame may be an I frame inserted based on an I frame request fed back by a terminal device.

[0064] The frame interval may be a frame number interval; or it may be a coding time interval. In this embodiment and other embodiments, the coding time interval between the frame to be coded and the previous I frame is taken as an example for description.

[0065] The preset I frame interval may be the time interval between two I frames. In this embodiment and other embodiments, the preset I frame interval is represented by IFTimeValue.

[0066] It is understandable that the preset I frame interval can be preset. For example, in order to increase the interval of IDR / I frames, the preset I frame interval can be preset to 120s.

[0067] The preset I frame interval may also be determined according to the number of frames of the live video stream, for example, IFTimeValue=Fps*2, wherein IFTimeValue represents the preset I frame interval, and Fps represents the number of frames of the live video stream.

[0068] It should be understood that in order to increase the interval of IDR / I frames, in this embodiment, it can be determined whether the frame interval is equal to the preset I frame interval. When the frame interval is equal to the preset I frame interval, the frame to be encoded is encoded as an I frame.

[0069] Step S20: monitoring whether an I frame request fed back by the terminal device according to the live video stream is received.

[0070] It should be understood that increasing the interval between I frames may result in the risk of packet loss and freeze. Therefore, in order to overcome the above defects, in this embodiment, the I frame request fed back by the terminal device according to the live video stream is also monitored, so as to perform I frame encoding when the terminal device feeds back freeze or switches the live source.

[0071] It should be noted that the terminal device can be a receiving end of the live video stream, for example, a client, a director, and a server.

[0072] It is understandable that when the terminal device detects that the live video stream is stuck or switches the live source, it can send an I frame request to the live capture device so that the live capture device can perform I frame encoding.

[0073] Step S30: judging whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result.

[0074] It should be understood that judging whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result may be to encode the frame to be encoded as an I frame when the frame interval is equal to the preset I frame interval, or when an I frame request is received according to the feedback of the live video stream by the terminal device;

[0075] When the frame interval is equal to the preset I frame interval and an I frame request is received from the terminal device according to the feedback of the live video stream, the frame to be encoded is encoded as an I frame.

[0076] When the frame interval is not equal to the preset I frame interval and no I frame request is received from the terminal device according to the live video stream feedback, the frame to be encoded is encoded as a P frame or a B frame.

[0077] For ease of understanding, the following examples are given, but this solution is not limited. The steps for encoding the frame to be encoded into a P frame are as follows: using the spatial or temporal correlation between adjacent pixels in the video image, use the transmitted pixels to predict the pixels currently being encoded, and calculate the difference between the predicted value and the authenticity. At the same time, the image described in the spatial domain is transformed (such as discrete cosine, sine, Hadamard, etc.) to form data (coefficients) in the transform domain, so as to achieve the purpose of changing the data distribution and reducing the amount of effective data. Combine motion estimation and motion compensation, quantization scanning and entropy coding, and loop filtering to complete the encoding of the next P frame.

[0078] It is understandable that after the encoding is completed, the video stream can be processed according to a preset transmission method (http, rtp / rtcp, etc.) and then packaged and sent to a media server or a terminal device.

[0079] It should be understood that the video streaming transmission method can greatly shorten the startup delay, reduce the buffering requirements, and realize the real-time media data transmission in the form of live broadcast.

[0080] It should be noted that the media server can decode, encode, transcode, enhance and distribute the video stream.

[0081] It is understandable that after the live broadcast acquisition device successfully transmits the video stream to the server, the director can pull the stream from the media server to obtain the video and play it. Similarly, the user client can also browse and play according to the URL provided by the media server.

[0082] In a specific implementation, for example, there is a sports game in a certain place A, and it is necessary to open the exciting process of the game to the public through live broadcast. Eight cameras Camera1 to Camera8 are respectively arranged at eight locations with good viewing angles in the venue, and the IDR / I frame interval IFtimeValue is set to 120s.

[0083] After the game starts, all eight cameras start streaming, push the game video data to the media server MediaA through the network, and start the IDR / I frame request monitoring service. The server requests IDR / I frames.

[0084] Before the game, the backend administrator ManagerA logs in to the Nanjing backend server ServiceA and first previews the effect through the pre-monitor or main monitor. If the player plays the stream incorrectly or an invalid stream cannot be decoded, ServiceA requests IDR / I frames from Camera1 to Camera8. Once it can be decoded normally, it stops requesting IDR / I frames. At the same time, before switching the lens Camera1 to Camera8 during the game, it first requests IDR / I frames from Camera1 to Camera8, and then executes the source switching and directing functions.

[0085] When a user watches a live broadcast through a browser or client, if decoding cannot be performed, the browser or client requests IDR / I frames from the backend server ServiceA, and ServiceA then requests IDR / I frames from Camera1 to Camera8.

[0086] During the encoding process, the video data source greatly increases the IDR / I frame interval, while dynamically requesting IDR / I frames to solve the complete image required for decoding, so that the transmission is mainly in the form of P frames / B frames with a small amount of data.

[0087] The same image quality effect, without screen jumps and breathing convergence, reduces the bandwidth by more than 2 to 15 times. Assuming a 4K image as an example, a complete 4K image is about 8M. If there is an IDR frame every 2 seconds, 120s will generate 8M*60+(P frame / B frame)*60*29>480M. The P frame / B frame is generated by the difference between the two frames. In live broadcasting, the difference is actually very small, often only a few KB. So if there is only one IDR frame in 120s, the data size is 8M*1+59*30*(P frame / B frame). If the average P frame is 10KB, the data size before using this proposal is: 8M*60+10KB*60*29→about 497M, and after using this proposal: 8M*1+59*60*(P frame / B frame)->about 42.57M. The modified data size is nearly one-tenth of the size before the modification.

[0088] In the first embodiment, it is disclosed to obtain the frame interval between the frame to be encoded and the previous I frame, determine whether the frame interval is equal to the preset I frame interval, monitor whether an I frame request fed back by the terminal device according to the live video stream is received, and determine whether to encode the frame to be encoded as an I frame according to the determination result and the monitoring result; because in this embodiment, the frame to be encoded is encoded as an I frame only when the frame interval between the frame to be encoded and the previous I frame is equal to the preset I frame interval, or an I frame request fed back by the terminal device according to the live video stream is received, thereby increasing the interval between I frames, greatly reducing the data transmission volume, reducing or even avoiding the freeze and screen distortion of the live screen caused by packet loss or a large number of data processing errors and delays, and being able to dynamically insert I frames based on user requests, solving the problem of the video source not decoding or switching the IDR / I frames required by the video source, thereby improving the user experience.

[0089] Reference Figure 4 , Figure 4 This is a flow chart of a second embodiment of a video encoding method according to the present invention. Figure 2 The first embodiment shown provides a second embodiment of the video encoding method of the present invention.

[0090] In the second embodiment, the step S10 includes:

[0091] Step S101: obtaining a previous I frame, and detecting whether the previous I frame is an inserted I frame, wherein the inserted I frame is an I frame inserted based on a preset I frame interval.

[0092] It should be understood that in practical applications, it is necessary to distinguish between inserting an I frame and requesting to insert an I frame to improve the accuracy of the frame interval. The inserting I frame may be an I frame inserted based on a preset I frame interval, and the requesting to insert an I frame may be an I frame inserted based on an I frame request fed back by a terminal device.

[0093] In the specific implementation, for example, if the current video data source is C, IFtimeValue=120s, the current time is T1, and the first IDR / I frame is generated, then the spontaneously generated IDR / I frame is: T1+120s*n, but the packet loss at 45 and 255 seconds after T1 makes it impossible to decode and generates a dynamic request IDR / I frame respectively. When n is 3, the actual IDR / I frame time points are T1, T1+45s, T1+120s, T1+240s, T1+255s, T1+360s, a total of 6, not 3.

[0094] Therefore, in order to overcome the above defects, in this embodiment, it is first detected whether the previous I frame is an inserted I frame. When the previous I frame is an inserted I frame, the frame interval between the frame to be encoded and the previous I frame is directly obtained.

[0095] Furthermore, in order to obtain a correct frame interval when the previous I frame is not an inserted I frame, after step S101, the method further includes:

[0096] If not, get the previous inserted I frame;

[0097] A frame interval between a frame to be encoded and a previous inserted I frame is obtained, and it is determined whether the frame interval is equal to a preset I frame interval.

[0098] It is understandable that if not, it means that the previous I frame cannot be used to determine the frame interval. Therefore, in this embodiment, the previous inserted I frame is obtained first, and then the frame interval between the frame to be encoded and the previous inserted I frame is obtained.

[0099] Step S102: If yes, obtain the frame interval between the frame to be encoded and the previous I frame, and determine whether the frame interval is equal to the preset I frame interval.

[0100] It should be understood that if so, it means that the previous I frame can be used to determine the frame interval. Therefore, in this embodiment, the frame interval between the frame to be encoded and the previous I frame is directly obtained, and it is determined whether the frame interval is equal to the preset I frame interval.

[0101] In the second embodiment, it is disclosed to obtain the previous I frame and detect whether the previous I frame is an inserted I frame, where the inserted I frame is an I frame inserted based on a preset I frame interval. If so, the frame interval between the frame to be encoded and the previous I frame is obtained, and it is determined whether the frame interval is equal to the preset I frame interval. Because this embodiment verifies whether the frame interval is reliable before making a judgment based on the frame interval, thereby ensuring the accuracy of the frame interval judgment and further ensuring the reliability of video encoding.

[0102] In the second embodiment, the step S20 includes:

[0103] Step S201: When the video data source of the live video stream is a single channel, receiving playback failure information fed back by the terminal device according to the live video stream.

[0104] It should be understood that when there is only one video data source, before playing, the director or user client first sends an IDR / I frame request with "request idr" in the content to the live acquisition device, and continuously requests every 1 to 2 seconds, then click play, and stop sending IDR / I frame requests after the playback is successful. If packet loss occurs during playback, an IDR / I frame request is sent.

[0105] Step S202: monitoring whether an I frame request fed back by the terminal device according to the live video stream is received according to the playback failure information.

[0106] It can be understood that monitoring whether an I frame request fed back by the terminal device according to the live video stream is received based on the playback failure information can be determined when the playback failure information is playback packet loss, and whether an I frame request fed back by the terminal device according to the live video stream is received.

[0107] In the second embodiment, it is disclosed that when the video data source of the live video stream is single-channel, the receiving terminal device monitors whether an I frame request fed back by the terminal device based on the live video stream is received according to the playback failure information, so that when the live video stream is erroneous or an invalid stream cannot be decoded, the I frame can be encoded in time to ensure the normal playback of the live video.

[0108] In the second embodiment, the step S20 further includes:

[0109] Step S201': when there are multiple video data sources for the live video stream, the receiving terminal device feeds back source switching information according to the live video stream.

[0110] It should be understood that when there are multiple video data sources, the director sends an IDR / I frame request to the live acquisition device before playing, and then clicks to play. When the director needs to switch different sources, he can send an IDR / I frame request first, and then execute the switch and play. If the switch is continuous, the IDR / I frame request will be continuously sent. When the switching of the video data source is completed, as long as the playback is normal, the IDR / I frame request will be stopped.

[0111] Step S202': monitoring whether an I frame request fed back by the terminal device according to the live video stream is received according to the source switching information.

[0112] It can be understood that monitoring whether an I frame request is received from the terminal device based on the live video stream feedback according to the source switching information can be determined when the source switching information is for switching the video data source, and an I frame request is received from the terminal device based on the live video stream feedback.

[0113] In the second embodiment, it is disclosed that when there are multiple video data sources for a live video stream, a receiving terminal device monitors whether an I frame request fed back by the terminal device based on the live video stream is received according to the source switching information based on the source switching information when the video data source is multiplexed. Due to this embodiment, when the terminal device switches the video data source, normal playback of the data source switching can be ensured.

[0114] Reference Figure 5 , Figure 5 This is a schematic diagram of a flow chart of a third embodiment of a video encoding method according to the present invention. Figure 2 The first embodiment shown is a third embodiment of the video encoding method of the present invention.

[0115] In the third embodiment, before step S10, the method further includes:

[0116] Step S01: Obtain an initial I frame interval, and determine whether the initial I frame interval is within a preset interval.

[0117] It should be understood that in order to avoid the preset I frame interval being too large or too small, in this embodiment, the initial I frame interval input by the user can be obtained first, and then the initial I frame interval can be adjusted based on the preset interval to obtain the preset I frame interval.

[0118] It should be noted that the preset interval can be composed of a minimum interval value and a maximum interval value, wherein the minimum interval value and the maximum interval value can be pre-set. In this embodiment and other embodiments, MinIFTimeValue represents the minimum interval value, and MaxIFTimeValue represents the maximum interval value. The preset interval is [MinIFTimeValue, MaxIFTimeValue].

[0119] In a specific implementation, for example, the MinIFTimeValue value is preset to 3s, the MaxIFTimeValue is preset to 240s, and the preset interval is [3s, 240s].

[0120] Step S02: adjusting the initial threshold according to the judgment result to obtain a preset I frame interval.

[0121] It can be understood that the initial threshold is adjusted according to the judgment result, and the preset I frame interval can be obtained by using the initial I frame interval as the preset I frame interval when the initial I frame interval is in the preset interval; using the minimum interval value as the preset I frame interval when the initial I frame interval is less than the minimum interval value; and using the maximum interval value as the preset I frame interval when the initial I frame interval is greater than the maximum interval value.

[0122] In the third embodiment, it is disclosed to obtain the initial I frame interval, and to determine whether the initial I frame interval is in a preset interval, and to adjust the initial threshold value according to the determination result to obtain the preset I frame interval; since the value range of the preset I frame interval is limited in this embodiment, the rationality of the value of the preset I frame interval is ensured, thereby increasing the interval between I frames.

[0123] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a video encoding program is stored. When the video encoding program is executed by a processor, the video encoding method as described above is implemented.

[0124] In addition, refer to Figure 6 , an embodiment of the present invention further provides a video encoding device, the video encoding device comprising: an interval determination module 10, a request monitoring module 20 and a frame encoding module 30;

[0125] The interval determination module 10 is used to obtain a frame interval between a frame to be encoded and a previous I frame, and determine whether the frame interval is equal to a preset I frame interval.

[0126] It should be understood that the method can be applied to scenarios where video data needs to be transmitted in the network, such as live broadcasting, video conferencing, and online classes, and this embodiment is not limited to this.

[0127] It is understandable that with the development of science and technology, digital video technology has been increasingly widely used in the fields of communication, live broadcasting, and remote conferencing. Especially since the 1990s, with the rapid development of the Internet and mobile communications, the processing and transmission of video information and multimedia information in the Internet network and mobile network has become a hot technology in my country's current informatization. Video information has a series of advantages, such as intuitiveness, accuracy, efficiency, and extensiveness. However, the amount of video information is too large. If video is to be effectively used, the problem of video compression encoding must be solved first, followed by the problem of ensuring the quality of compressed video, while ensuring the reliability and stability of network transmission of video data.

[0128] In existing professional live broadcast director applications, video is generally collected using proprietary data acquisition equipment (such as cameras). The direct-output video data is usually in YUV420 / 422 / 444, YUYV, NV12, NV21 and other video formats. The data is compressed through the encoding and compression function of the device into new data with a very small data volume, such as H264 / HEVC / H266, etc., and then packaged and encapsulated according to the transmission protocol, such as RTP, and transmitted to the media server. Here, the media server is a general server that includes functions such as clustering or transcoding enhanced distribution. After the director completes the requirements through the director operation, the user can access and browse the video according to the specified URL.

[0129] With the help of modern encoding, decoding and network communication technologies, the current live broadcast director has also achieved good development and experience. In the encoding link of the existing live broadcast director solution, in order to decode the new video data source in time through the IDR / I frame when the program director switches to a new program, the IDR / I frame (GOP) interval is often set to be very small, such as 1s to 2s. However, IDR / I is a complete frame of image. Compared with P frame and B frame, the data volume is very large. Even if the modern network is very developed, due to the differences in network equipment conditions in different regions and unpredictable influences in the network, the data transmission of large amounts of data is more likely to cause packet loss, and the user's storage device is required to be higher. Whether it is the sending end or the receiving end, there are requirements for whether the processor can process large amounts of data in time. The processor with insufficient performance may not be able to process large amounts of data in time, resulting in data loss, and image delay. Moreover, with the development trend of higher and higher video quality, the adaptability is getting worse and worse. Even if various remediation mechanisms such as FEC / packet loss retransmission / NACK are used at the end, the large amount of data is easy to cause the remediation of the data itself, and the remediation effect is not good, especially under the future 4K / 8K and other large resolution data conditions, it is more difficult to remedy. In addition, the picture is prone to visible convergence and jumping phenomena, which makes the user experience very poor.

[0130] In order to overcome the above defects, in this embodiment, the frame to be encoded is encoded as an I frame only when the frame interval between the frame to be encoded and the previous I frame is equal to the preset I frame interval, or when an I frame request is received from the terminal device based on the live video stream feedback, thereby increasing the interval between I frames and being able to dynamically insert I frames based on user requests, thereby reducing the amount of transmission data of the video stream, avoiding freezes and screen distortion in the live broadcast, and improving the user experience.

[0131] It should be noted that the frame to be encoded may be a video frame to be encoded in a video source.

[0132] For ease of understanding, the following examples are given, but this solution is not limited. The video source acquisition process steps are as follows:

[0133] (1) Arrange the location of the live broadcast acquisition equipment. The location needs to have a good field of view and picture composition for the live broadcast scene, and fix the equipment;

[0134] (2) Open the device to collect data;

[0135] Usually, Windows uses VFW or DirectShow to collect real-time video, Linux uses the v4l2 standard interface to collect, Android uses the Android API to collect, and iOS or other systems have corresponding platform collection methods, which will not be described in detail here.

[0136] The video data after acquisition is usually in the form of YUV420 / 422 / 444, YUYV, NV12, NV21 and other video formats, and the data volume is very large.

[0137] For ease of understanding, refer to Figure 3 This is for illustration only, but does not limit this solution. Figure 3 This is a schematic diagram of the location of live broadcast collection equipment. In the figure, eight live broadcast collection devices Cam1 to 8 are arranged around the basketball court, and the layout positions are shown in the figure.

[0138] It should be noted that the frame interval between the frame to be encoded and the previous I frame may be the frame interval between the frame to be encoded and the previous inserted I frame, or the frame interval between the frame to be encoded and the previous requested inserted I frame. The inserted I frame may be an I frame inserted based on a preset I frame interval, and the requested inserted I frame may be an I frame inserted based on an I frame request fed back by a terminal device.

[0139] The frame interval may be a frame number interval; or it may be a coding time interval. In this embodiment and other embodiments, the coding time interval between the frame to be coded and the previous I frame is taken as an example for description.

[0140] The preset I frame interval may be the time interval between two I frames. In this embodiment and other embodiments, the preset I frame interval is represented by IFTimeValue.

[0141] It can be understood that the preset I frame interval can be preset, for example, the preset I frame interval can be preset to 120s.

[0142] The preset I frame interval may also be determined according to the number of frames of the live video stream, for example, IFTimeValue=Fps*2, wherein IFTimeValue represents the preset I frame interval, and Fps represents the number of frames of the live video stream.

[0143] It should be understood that in order to increase the interval of IDR / I frames, in this embodiment, it can be determined whether the frame interval is equal to the preset I frame interval. When the frame interval is equal to the preset I frame interval, the frame to be encoded is encoded as an I frame.

[0144] It should be understood that in order to increase the interval of IDR / I frames, in this embodiment, it can be determined whether the frame interval is equal to the preset I frame interval. When the frame interval is equal to the preset I frame interval, the frame to be encoded is encoded as an I frame.

[0145] The request monitoring module 20 is used to monitor whether an I frame request fed back by the terminal device according to the live video stream is received.

[0146] It should be understood that increasing the interval between I frames may result in the risk of packet loss and freeze. Therefore, in order to overcome the above defects, in this embodiment, the I frame request fed back by the terminal device according to the live video stream is also monitored, so as to perform I frame encoding when the terminal device feeds back freeze or switches the live source.

[0147] It should be noted that the terminal device can be a receiving end of the live video stream, for example, a client, a director, and a server.

[0148] It is understandable that when the terminal device detects that the live video stream is stuck or switches the live source, it can send an I frame request to the live capture device so that the live capture device can perform I frame encoding.

[0149] The frame encoding module 30 is used to determine whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result.

[0150] It should be understood that judging whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result may be to encode the frame to be encoded as an I frame when the frame interval is equal to the preset I frame interval, or when an I frame request is received according to the feedback of the live video stream by the terminal device;

[0151] When the frame interval is equal to the preset I frame interval and an I frame request is received from the terminal device according to the feedback of the live video stream, the frame to be encoded is encoded as an I frame.

[0152] When the frame interval is not equal to the preset I frame interval and no I frame request is received from the terminal device according to the live video stream feedback, the frame to be encoded is encoded as a P frame or a B frame.

[0153] For ease of understanding, the following examples are given, but this solution is not limited. The steps for encoding the frame to be encoded into a P frame are as follows: using the spatial or temporal correlation between adjacent pixels in the video image, use the transmitted pixels to predict the pixels currently being encoded, and calculate the difference between the predicted value and the authenticity. At the same time, the image described in the spatial domain is transformed (such as discrete cosine, sine, Hadamard, etc.) to form data (coefficients) in the transform domain, so as to achieve the purpose of changing the data distribution and reducing the amount of effective data. Combine motion estimation and motion compensation, quantization scanning and entropy coding, and loop filtering to complete the encoding of the next P frame.

[0154] It is understandable that after the encoding is completed, the video stream can be processed according to a preset transmission method (http, rtp / rtcp, etc.) and then packaged and sent to a media server or a terminal device.

[0155] It should be understood that the video streaming transmission method can greatly shorten the startup delay, reduce the buffering requirements, and realize the real-time media data transmission in the form of live broadcast.

[0156] It should be noted that the media server can decode, encode, transcode, enhance and distribute the video stream.

[0157] It is understandable that after the live broadcast acquisition device successfully transmits the video stream to the server, the director can pull the stream from the media server to obtain the video and play it. Similarly, the user client can also browse and play according to the URL provided by the media server.

[0158] In a specific implementation, for example, there is a sports game in a certain place A, and it is necessary to open the exciting process of the game to the public through live broadcast. Eight cameras Camera1 to Camera8 are respectively arranged at eight locations with good viewing angles in the venue, and the IDR / I frame interval IFtimeValue is set to 120s.

[0159] After the game starts, all eight cameras start streaming, push the game video data to the media server MediaA through the network, and start the IDR / I frame request monitoring service. The server requests IDR / I frames.

[0160] Before the game, the backend administrator ManagerA logs in to the Nanjing backend server ServiceA and first previews the effect through the pre-monitor or main monitor. If the player plays the stream incorrectly or an invalid stream cannot be decoded, ServiceA requests IDR / I frames from Camera1 to Camera8. Once it can be decoded normally, it stops requesting IDR / I frames. At the same time, before switching the lens Camera1 to Camera8 during the game, it first requests IDR / I frames from Camera1 to Camera8, and then executes the source switching and directing functions.

[0161] When a user watches a live broadcast through a browser or client, if decoding cannot be performed, the browser or client requests IDR / I frames from the backend server ServiceA, and ServiceA then requests IDR / I frames from Camera1 to Camera8.

[0162] During the encoding process, the video data source greatly increases the IDR / I frame interval, while dynamically requesting IDR / I frames to solve the complete image required for decoding, so that the transmission is mainly in the form of P frames / B frames with a small amount of data.

[0163] The same image quality effect, without screen jumps and breathing convergence, reduces the bandwidth by more than 2 to 15 times. Assuming a 4K image as an example, a complete 4K image is about 8M. If there is an IDR frame every 2 seconds, 120s will generate 8M*60+(P frame / B frame)*60*29>480M. The P frame / B frame is generated by the difference between the two frames. In live broadcasting, the difference is actually very small, often only a few KB. So if there is only one IDR frame in 120s, the data size is 8M*1+59*30*(P frame / B frame). If the average P frame is 10KB, the data size before using this proposal is: 8M*60+10KB*60*29→about 497M, and after using this proposal: 8M*1+59*60*(P frame / B frame)->about 42.57M. The modified data size is nearly one-tenth of the size before the modification.

[0164] In the first embodiment, it is disclosed to obtain the frame interval between the frame to be encoded and the previous I frame, determine whether the frame interval is equal to the preset I frame interval, monitor whether an I frame request fed back by the terminal device according to the live video stream is received, and determine whether to encode the frame to be encoded as an I frame according to the determination result and the monitoring result; because in this embodiment, the frame to be encoded is encoded as an I frame only when the frame interval between the frame to be encoded and the previous I frame is equal to the preset I frame interval, or an I frame request fed back by the terminal device according to the live video stream is received, thereby increasing the interval between I frames, greatly reducing the data transmission volume, reducing or even avoiding the freeze and screen distortion of the live screen caused by packet loss or a large number of data processing errors and delays, and being able to dynamically insert I frames based on user requests, solving the problem of the video source not decoding or switching the IDR / I frames required by the video source, thereby improving the user experience.

[0165] In one embodiment, the interval determination module 10 is further used to obtain a previous I frame and detect whether the previous I frame is an inserted I frame, wherein the inserted I frame is an I frame inserted based on a preset I frame interval;

[0166] The interval determination module 10 is further configured to obtain a frame interval between the frame to be encoded and the previous I frame, and determine whether the frame interval is equal to a preset I frame interval.

[0167] In one embodiment, the interval determination module 10 is further configured to obtain the previous inserted I frame if no;

[0168] The interval determination module 10 is further used to obtain a frame interval between a frame to be encoded and a previous inserted I frame, and determine whether the frame interval is equal to a preset I frame interval.

[0169] In one embodiment, the video encoding device further includes: an interval adjustment module;

[0170] The interval adjustment module is used to obtain an initial I frame interval and determine whether the initial I frame interval is within a preset interval;

[0171] The interval adjustment module is further used to adjust the initial threshold according to the judgment result to obtain a preset I frame interval.

[0172] In one embodiment, the request monitoring module 20 is further used to receive playback failure information fed back by the terminal device according to the live video stream when the video data source of the live video stream is a single channel;

[0173] The request monitoring module 20 is further used to monitor whether an I frame request fed back by the terminal device according to the live video stream is received according to the playback failure information.

[0174] In one embodiment, the request monitoring module 20 is further used to receive playback failure information fed back by the terminal device according to the live video stream when the video data source of the live video stream is a single channel;

[0175] The request monitoring module 20 is further used to monitor whether an I frame request fed back by the terminal device according to the live video stream is received according to the playback failure information.

[0176] In one embodiment, the frame encoding module 30 is further configured to determine whether to encode the frame to be encoded as an I frame when the frame interval is equal to a preset I frame interval or when an I frame request is received from a terminal device according to feedback of a live video stream;

[0177] The frame encoding module 30 is further used to determine not to encode the frame to be encoded as an I frame when the frame interval is not equal to the preset I frame interval and no I frame request is received from the terminal device based on the live video stream feedback.

[0178] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0179] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as a read-only memory image (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for enabling an access device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0181] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video encoding method, characterized in that: The video encoding method comprises the following steps: Obtaining a frame interval between a frame to be encoded and a previous I frame, and determining whether the frame interval is equal to a preset I frame interval; Monitor whether an I frame request based on the live video stream feedback from the terminal device is received; Determine whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result; The step of obtaining a frame interval between a frame to be encoded and a previous I frame, and determining whether the frame interval is equal to a preset I frame interval, comprises: Acquire a previous I frame, and detect whether the previous I frame is an inserted I frame, where the inserted I frame is an I frame inserted based on a preset I frame interval; If so, the frame interval between the frame to be encoded and the previous I frame is obtained, and it is determined whether the frame interval is equal to the preset I frame interval.

2. The video encoding method according to claim 1, characterized in that: After the step of obtaining the previous I frame and detecting whether the previous I frame is an inserted I frame, the method further includes: If not, get the previous inserted I frame; A frame interval between a frame to be encoded and a previous inserted I frame is obtained, and it is determined whether the frame interval is equal to a preset I frame interval.

3. The video encoding method according to claim 1 or 2, characterized in that: Before the step of obtaining the frame interval between the frame to be encoded and the previous I frame, and determining whether the frame interval is equal to the preset I frame interval, the step further includes: Obtaining an initial I frame interval, and determining whether the initial I frame interval is within a preset interval; The initial threshold is adjusted according to the judgment result to obtain a preset I frame interval.

4. The video encoding method according to claim 1 or 2, characterized in that: The step of monitoring whether an I frame request fed back by the terminal device according to the live video stream is received includes: When the video data source of the live video stream is a single channel, the receiving terminal device feeds back the playback failure information according to the live video stream; According to the playback failure information, monitor whether an I frame request fed back by the terminal device according to the live video stream is received.

5. The video encoding method according to claim 1 or 2, characterized in that: The step of monitoring whether an I frame request fed back by the terminal device according to the live video stream is received includes: When the video data source of the live video stream is multiple, the receiving terminal device switches the source information fed back by the live video stream; Monitor whether an I frame request fed back by the terminal device according to the live video stream is received according to the source switching information.

6. The video encoding method according to claim 1 or 2, characterized in that: The step of judging whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result includes: When the frame interval is equal to the preset I frame interval, or when an I frame request is received from the terminal device according to the live video stream feedback, it is determined that the frame to be encoded is encoded as an I frame; When the frame interval is not equal to the preset I frame interval and no I frame request is received from the terminal device according to the live video stream feedback, it is determined not to encode the frame to be encoded as an I frame.

7. A video encoding device, characterized in that: The video encoding device comprises: a memory, a processor, and a video encoding program stored in the memory and executable on the processor, wherein the video encoding program implements the video encoding method according to any one of claims 1 to 6 when executed by the processor.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a video encoding program, and when the video encoding program is executed by a processor, the video encoding method according to any one of claims 1 to 6 is implemented.

9. A video encoding device, characterized in that: The video encoding device comprises: an interval determination module, a request monitoring module and a frame encoding module; The interval determination module is used to obtain a frame interval between a frame to be encoded and a previous I frame, and determine whether the frame interval is equal to a preset I frame interval; The request monitoring module is used to monitor whether an I frame request fed back by the terminal device according to the live video stream is received; The frame encoding module is used to determine whether to encode the frame to be encoded as an I frame according to the judgment result and the monitoring result; Among them, the interval judgment module is also used to obtain the previous I frame and detect whether the previous I frame is an inserted I frame, and the inserted I frame is an I frame inserted based on a preset I frame interval; if so, the frame interval between the frame to be encoded and the previous I frame is obtained, and it is determined whether the frame interval is equal to the preset I frame interval.

Images