Video processing method and device based on webpage instant messaging, equipment and medium

Abstract

The embodiment of the invention discloses a video processing method and device based on webpage instant messaging, equipment and a medium, which can be applied to various scenes such as intelligent traffic, auxiliary driving, cloud technology and artificial intelligence. The video processing method based on webpage instant messaging comprises the following steps: a stream pushing terminal side selectively performs down-sampling operation on a video in combination with the current real-time network quality, and a stream pulling terminal side selectively performs super-resolution processing on the video in combination with the current real-time network quality and the image quality condition of an image frame. According to the technical scheme, the video quality in the video processing process based on the webpage instant messaging is ensured, and the video processing scheme based on the webpage instant messaging is greatly optimized.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a video processing method based on web page instant messaging, a video processing device based on web page instant messaging, an electronic device, and a computer-readable medium. Background Art

[0002] Web Real-Time Communications (WebRTC) is a real-time communication technology that allows network applications or sites to establish peer-to-peer connections between browsers without the help of an intermediary (i.e., without installing any plug-ins or third-party software) to achieve the transmission of video streams or other arbitrary data. It plays an extremely important role in the development of video communication software.

[0003] Currently, in the video processing process based on web instant messaging, it is difficult to ensure the video quality due to many factors (such as network, hardware configuration, etc.).

[0004] Therefore, how to improve video processing based on web instant messaging to ensure video quality is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a video processing method, device, equipment, and medium based on web instant messaging, which improves the video processing based on web instant messaging, thereby ensuring the video quality.

[0006] In a first aspect, an embodiment of the present application provides a video processing method based on web instant messaging, the method comprising: obtaining a streaming video sent by a streaming terminal via web instant messaging, the streaming video comprising multiple image frames, and performing image quality evaluation on each image frame in the streaming video; performing super-resolution processing on image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation results to obtain super-resolution image frames; and generating a target video based on the super-resolution image frames and other image frames among the multiple image frames except those subjected to super-resolution processing.

[0007] In the second aspect, an embodiment of the present application provides a video processing method based on web instant messaging, the method comprising: obtaining network quality information of the current network environment of the streaming terminal; if the network quality information is less than a preset network quality threshold, obtaining a target resolution that matches the network quality information; wherein the target resolution is positively correlated with the network quality information; encoding the image frames in the captured video according to the target resolution to obtain the encoded video; streaming the encoded video to the streaming terminal through web instant messaging, so that the streaming terminal performs super-resolution processing on the image frames in the received video whose image quality parameters are less than the preset image quality threshold to generate the target video.

[0008] In a third aspect, an embodiment of the present application provides a video processing device based on web instant messaging, the device comprising: an acquisition and evaluation module, configured to acquire a streaming video sent by a streaming terminal via web instant messaging, the streaming video comprising multiple image frames, and perform image quality evaluation on each image frame in the streaming video; a super-resolution module, configured to perform super-resolution processing on image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation results to obtain super-resolution image frames; a generation module, configured to generate a target video based on the super-resolution image frames and other image frames among the multiple image frames except those subjected to super-resolution processing.

[0009] In one embodiment of the present application, based on the aforementioned scheme, the acquisition and evaluation module is specifically configured as follows: for each image frame in the pushed video, the waiting time for the image frame to be decoded is obtained; if the waiting time is greater than a preset waiting time threshold, an image quality evaluation result is obtained that is less than the preset image quality threshold; if the waiting time is less than or equal to the preset waiting time threshold, an image quality evaluation result is obtained that is greater than or equal to the preset image quality threshold.

[0010] In one embodiment of the present application, based on the aforementioned scheme, the acquisition and evaluation module is further specifically configured to: store the image frame in a jitter buffer area, and record the starting time when the image frame is stored in the jitter buffer area; start timing from the starting time, and statistically obtain the waiting time for the image frame to wait for decoding processing.

[0011] In one embodiment of the present application, based on the aforementioned scheme, the super-resolution module is specifically configured as follows: marking the image frames among the multiple image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation results; and performing super-resolution processing on the marked image frames to obtain super-resolution image frames.

[0012] In one embodiment of the present application, based on the aforementioned scheme, the super-resolution module is specifically configured as follows: according to the image quality evaluation result, a target image frame whose image quality is less than a preset image quality threshold is selected from the multiple image frames as the target image frame, and adjacent image frames of the target image frame are selected; based on the feature information of the target image frame and the feature information of the adjacent image frames, the target image frame is super-resolution processed to obtain an image frame after super-resolution processing.

[0013] In one embodiment of the present application, based on the aforementioned scheme, the super-resolution module is further specifically configured as follows: performing feature extraction on the target image frame and the adjacent image frame respectively, and fusing the extracted features to obtain fused features; performing bidirectional propagation on the fused features to obtain reconstructed features; obtaining a super-resolution image frame corresponding to the target image frame based on the reconstructed features, and using the super-resolution image frame as the image frame after super-resolution processing.

[0014] In one embodiment of the present application, based on the aforementioned scheme, the super-resolution module is specifically configured as follows: according to the image quality evaluation result, an image frame whose image quality is less than a preset image quality threshold is selected from the multiple image frames as a target image frame; based on the image quality of the target image frame, a target resolution of the target image frame is obtained; wherein different image qualities correspond to different target resolutions; super-resolution is performed on the target image frame according to the target resolution of the target image frame to obtain a super-resolution image frame corresponding to the target image frame, and the super-resolution image frame is used as the image frame after super-resolution processing.

[0015] In one embodiment of the present application, based on the aforementioned scheme, a generation module is specifically configured as follows: decoding the image frames other than the super-resolution processed image frames among the multiple image frames according to a specified resolution to obtain decoded image frames; combining the super-resolution processed image frames and the decoded image frames to generate a target video.

[0016] In one embodiment of the present application, based on the aforementioned scheme, the acquisition and evaluation module is specifically configured as follows: receiving a real-time transport protocol data packet sent by a streaming terminal through web instant messaging; decapsulating the data packet based on the real-time transport protocol to obtain decapsulated video data; and decrypting the video data to obtain the streaming video.

[0017] In a fourth aspect, an embodiment of the present application provides a video processing device based on web instant messaging, the device comprising: a first acquisition module, configured to obtain network quality information of the current network environment of the streaming terminal; a second acquisition module, configured to obtain a target resolution matching the network quality information if the network quality information is less than a preset network quality threshold; wherein the target resolution is positively correlated with the network quality information; an encoding module, configured to encode image frames in the captured video according to the target resolution to obtain an encoded video; a streaming module, configured to stream the encoded video to the streaming terminal through web instant messaging, so that the streaming terminal performs super-resolution processing on image frames in the received video whose image quality parameters are less than the preset image quality threshold to generate a target video.

[0018] In one embodiment of the present application, based on the aforementioned scheme, the streaming module is specifically configured as follows: obtaining a target streaming bit rate corresponding to the network quality information; wherein the target streaming bit rate is positively correlated with the network quality information; and streaming the encoded video to the streaming terminal through web instant messaging according to the target streaming bit rate.

[0019] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the video processing method based on web instant messaging as described above.

[0020] In a sixth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the video processing method based on web instant messaging as described above is implemented.

[0021] In a seventh aspect, an embodiment of the present application provides a computer program product, including computer instructions, which, when executed by a processor, implement the video processing method based on web instant messaging as described above.

[0022] In the technical solution provided in the embodiments of the present application:

[0023] In the video processing process based on web instant messaging, the sending end (i.e., the streaming terminal side) selectively downsamples the video in combination with the current real-time network quality, so that the video quality can be guaranteed even when the network quality is poor or the network condition is poor (e.g., insufficient available network bandwidth). The receiving end (i.e., the streaming terminal side) selectively performs super-resolution processing on the video in combination with the current real-time network quality and the image quality of the image frame, so that the video quality can be guaranteed even when the network quality is poor or the network condition is poor (e.g., insufficient available network bandwidth), that is, the video quality is maximized while ensuring system stability and performance.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1-1 and Figure 1-2 is a schematic diagram of an exemplary implementation environment to which the technical solution of the embodiments of the present application can be applied;

[0026] Figure 2 is a flowchart of a video processing method based on web page instant messaging shown in an exemplary embodiment of the present application;

[0027] Figure 3 is a flowchart of a video processing method based on web page instant messaging shown in another exemplary embodiment of the present application;

[0028] Figure 4 is a flowchart of a video processing method based on web page instant messaging shown in another exemplary embodiment of the present application;

[0029] Figure 5 is a flowchart of a video processing method based on web page instant messaging shown in another exemplary embodiment of the present application;

[0030] Figure 6 is a flowchart of a video processing method based on web page instant messaging shown in another exemplary embodiment of the present application;

[0031] Figure 7 is a flowchart of a video processing method based on web page instant messaging shown in another exemplary embodiment of the present application;

[0032] Figure 8 is a flowchart of a video processing method based on web page instant messaging shown in an exemplary embodiment of the present application;

[0033] Fig. 9 is a flowchart of a video processing method based on web page instant messaging shown in another exemplary embodiment of the present application;

[0034] Fig.10 is a schematic diagram of a video processing method based on web page instant messaging, shown in another exemplary embodiment of the present application;

[0035] Fig.11 is a schematic diagram of a video processing method based on web page instant messaging, shown in another exemplary embodiment of the present application;

[0036] Fig.12 is a block diagram of a video processing device based on web page instant messaging shown in an exemplary embodiment of the present application;

[0037] Fig.13 is a block diagram of a video processing device based on web page instant messaging shown in an exemplary embodiment of the present application;

[0038] Fig.14 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0039] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments that are the same as the present application. Instead, they are only examples of devices and methods that are the same as some aspects of the present application as detailed in the attached claims.

[0040] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0042] It should be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0043] It is understandable that web instant messaging is a technology that realizes real-time communication directly based on the web page without downloading software. At present, most video communication software also relies on it for development. However, in the video processing process based on web instant messaging, it is difficult to guarantee the video quality due to many factors (such as network, hardware device configuration, etc.).

[0044] Therefore, in order to improve the video processing based on web instant messaging and ensure the video quality, this application provides a video processing solution based on web instant messaging. Figure 1-1 , Figure 1-1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment mainly includes a streaming terminal 101 and a streaming terminal 102. It is understandable that the streaming terminal 101 and the streaming terminal 102 can communicate with each other via a wired or wireless network.

[0045] The streaming terminal refers to the party that sends the video; it includes but is not limited to smartphones, tablets, laptops, computers, intelligent voice interaction devices, smart home appliances, car terminals, aircraft, etc.

[0046] The streaming terminal refers to the party that receives the video; it includes but is not limited to smartphones, tablets, laptops, computers, intelligent voice interaction devices, smart home appliances, car terminals, aircraft, etc.

[0047] Among them, push-stream terminal and pull-stream terminal are relative concepts. For the same device, it can be a push-stream terminal or a pull-stream terminal. In actual applications, the specific application scenario shall prevail.

[0048] It is understandable that the streaming terminal and / or the streaming terminal are usually mobile devices. For example, the smartphones, tablets, laptops, computers, intelligent voice interaction devices, etc. mentioned above are all mobile devices. Mobile devices usually have limited hardware configurations, so they will not do too much processing on the video during the video processing process based on web instant messaging, resulting in poor video quality.

[0049] Therefore, in the process of video processing based on web instant messaging, the present application selectively downsamples the video on the one hand at the sending end (i.e., the streaming terminal side) in combination with the current real-time network quality; on the other hand, at the receiving end (i.e., the streaming terminal side), by leveraging the limited hardware configuration of mobile devices and super-resolution technology, selectively super-resolution the image frames with poor image quality in the video, thereby improving the video quality.

[0050] It is understandable that, considering the limited hardware configuration of mobile devices, using the limited hardware configuration of mobile devices to perform video downsampling and super-resolution processing will put a lot of computing pressure on mobile devices and their computing resources will be tight.

[0051] Therefore, in order to reduce the computing pressure of mobile devices and alleviate the phenomenon of tight computing resources, in one embodiment of the present application, refer to Figure 1-2 , Figure 1-2 1 is a schematic diagram of another implementation environment involved in the present application. The implementation environment mainly includes a streaming terminal 101, a streaming terminal 102, and an edge node 103. It can be understood that the streaming terminal 101, the streaming terminal 102, and the edge node 103 ( Figure 1-2 Multiple edge nodes are shown in FIG. 4 ) can communicate with each other through a wired or wireless network. Among them:

[0052] The push stream terminal and pull stream terminal are the same as above. Figure 1-1 , I will not go into details here.

[0053] Edge nodes refer to the party that can at least perform video downsampling and super-resolution processing, thereby reducing the computing pressure of the push and pull terminals. Edge nodes can be servers that provide various services. They can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This section does not restrict this.

[0054] It should be noted that Figure 1-1 and Figure 1-2 The push stream terminal 101 and the pull stream terminal 102, and Figure 1-2 The number of edge nodes 103 in the figure is only for illustration, and any number of streaming terminals 101, streaming terminals 102 and edge nodes 103 may be provided according to actual needs.

[0055] In one embodiment of the present application, the video processing method based on webpage instant messaging can be interactively executed by the streaming terminal 101 and the streaming terminal 102 .

[0056] Exemplarily, the streaming terminal obtains network quality information of the network environment in which the streaming terminal is currently located; if the network quality information is less than a preset network quality threshold, a target resolution matching the network quality information is obtained, wherein the target resolution is positively correlated with the network quality information; then, the image frames in the captured video are encoded according to the target resolution to obtain the encoded video; then, the encoded video is streamed to the streaming terminal via web instant messaging.

[0057] Correspondingly, the pull stream terminal obtains the push stream video sent by the push stream terminal through web instant messaging, the push stream video includes multiple image frames, and performs image quality evaluation on each image frame in the push stream video; then, according to the image quality evaluation result, super-resolution processing is performed on the image frames whose image quality is less than a preset image quality threshold to obtain super-resolution processed image frames; then, based on the super-resolution processed image frames and other image frames among the multiple image frames except the ones subjected to super-resolution processing, a target video is generated.

[0058] In one embodiment of the present application, the video processing method based on web instant messaging can be interactively executed by the streaming terminal 101, the streaming terminal 102 and the edge node 103.

[0059] Exemplarily, the streaming terminal obtains network quality information of the network environment in which the streaming terminal is currently located; if the network quality information is less than a preset network quality threshold, a target resolution matching the network quality information is obtained, wherein the target resolution is positively correlated with the network quality information; and then the target resolution is sent to the edge node corresponding to the streaming terminal.

[0060] Accordingly, the edge node corresponding to the streaming terminal (can be Figure 1-2 The video downsampling module shown in the figure encodes the image frames in the collected video according to the target resolution to obtain the encoded video; and then returns the encoded video to the streaming terminal.

[0061] Correspondingly, the streaming terminal streams the encoded video to the streaming terminal through web instant messaging.

[0062] Furthermore, the stream pulling terminal obtains the streaming video sent by the streaming terminal through web instant messaging, where the streaming video includes multiple image frames, and performs image quality evaluation on each image frame in the streaming video; then, the image quality evaluation of each image frame is sent to the edge node corresponding to the stream pulling terminal.

[0063] Accordingly, the edge node corresponding to the streaming terminal (can be Figure 1-2 The video super-resolution module shown in the figure performs super-resolution processing on image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation result to obtain super-resolution image frames; and then returns the super-resolution image frames to the streaming terminal.

[0064] Accordingly, the streaming terminal generates a target video based on the image frame after the super-resolution processing and other image frames among the multiple image frames except the image frames after the super-resolution processing.

[0065] Figure 1-1 and Figure 1-2The technical solutions of the illustrated embodiments can be applied to various scenarios, including but not limited to smart transportation, assisted driving, cloud technology, artificial intelligence, etc.; in actual applications, corresponding adjustments can be made according to specific application scenarios.

[0066] For example, if applied to smart transportation or assisted driving scenarios, the streaming terminal 101 may be a vehicle-mounted terminal, a navigation terminal, etc., and the streaming terminal 102 may be a computer, a smart phone, a tablet, a laptop, etc. By implementing the present application solution, the video quality of the relevant videos in the Internet of Vehicles service can be guaranteed.

[0067] For example, if applied in cloud technology or artificial intelligence scenarios, the edge node 103 may be a cloud server, and the streaming terminal 101 and the streaming terminal 102 may be a computer, a smart phone, etc. By implementing the solution of this application, the video quality of relevant videos in payment services, entertainment services (such as game services), etc. can be guaranteed.

[0068] It should be noted that in the specific implementation of this application, user-related data is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0069] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0070] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of a video processing method based on web page instant messaging according to an embodiment of the present application. The video processing method based on web page instant messaging can be executed by the streaming terminal 102. Figure 2 As shown, the video processing method based on webpage instant messaging at least includes S201 to S203, which are described in detail as follows:

[0071] S201, obtaining a streaming video sent by a streaming terminal through web instant messaging, wherein the streaming video includes a plurality of image frames, and performing image quality evaluation on each image frame in the streaming video.

[0072] In the embodiment of the present application, the push stream video refers to the video received from the push stream terminal after the pull stream terminal has processed the video accordingly.

[0073] In one embodiment of the present application, the process of obtaining the streaming video sent by the streaming terminal through web instant messaging in S201 may include:

[0074] Receiving a real-time transport protocol data packet sent by a streaming terminal via web instant messaging;

[0075] Decapsulating the data packet based on the real-time transport protocol to obtain decapsulated video data;

[0076] Decrypt the video data to get the streaming video.

[0077] That is, in an optional embodiment, the streaming terminal may first encrypt the collected video data, and then encapsulate the encrypted video data based on the Real-time Transport Protocol (RTP) to obtain encapsulated video data (i.e., real-time transport protocol data packet), and then send the real-time transport protocol data packet to the streaming terminal via web instant messaging. Correspondingly, the streaming terminal receives the real-time transport protocol data packet sent by the streaming terminal via web instant messaging, and then first decapsulates the data packet based on the real-time transport protocol to obtain decapsulated video data, and then decrypts the decapsulated video data to obtain decrypted video data, which is the streaming video.

[0078] Among them, in the optional embodiment, the encryption of the push terminal and the decryption of the pull terminal are determined by negotiation between the two parties, and can specifically be a pair of matching public and private keys, that is, the push terminal uses the public key for encryption, and the pull terminal uses the private key matching the public key for decryption, etc. In actual applications, it can be flexibly adjusted according to the specific application scenario.

[0079] By implementing the optional embodiment, the security of the video in the video processing process based on web instant messaging is guaranteed through encryption at the push terminal and decryption at the pull terminal, which is particularly suitable for scenarios with high video security.

[0080] In the embodiment of the present application, the pull stream terminal obtains the push stream video sent by the push stream terminal through web instant messaging, and can perform image quality evaluation on each image frame in the push stream video. Image quality evaluation refers to judging the quality of the image frame, and the quality can be reflected by the clarity of the image frame. Generally, the higher the clarity of the image frame, the better the quality of the image frame. Conversely, the lower the clarity of the image frame, the worse the quality of the image frame.

[0081] S202, performing super-resolution processing on image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation result to obtain super-resolution image frames.

[0082] In the embodiment of the present application, the stream pulling terminal performs image quality evaluation on each image frame in the push stream video, and can obtain the image quality evaluation result of each image frame. Then, it can be determined whether to perform super-resolution processing on the image frame based on the image quality evaluation result of each image frame.

[0083] In the embodiment of the present application, whether to perform super-resolution processing on an image frame is determined according to the image quality evaluation result of each image frame, including the following two situations:

[0084] Case 1: if the image quality assessment result of the image frame indicates that the image quality of the image frame is less than a preset image quality threshold, then the image quality of the image frame is poor at this time. Therefore, the image frame needs to be super-resolution processed to improve the image quality of the image frame.

[0085] Case 2: if the image quality assessment result of the image frame indicates that the image quality of the image frame is greater than or equal to a preset image quality threshold, then the image quality of the image frame is relatively good at this time, and therefore, there is no need to perform super-resolution processing on the image frame.

[0086] S203, generating a target video based on the image frame after the super-resolution processing and other image frames among the multiple image frames except the image frames after the super-resolution processing.

[0087] In the embodiment of the present application, the streaming terminal performs super-resolution processing on image frames whose image quality is less than a preset image quality threshold to obtain super-resolution image frames. Thereafter, a target video can be generated based on the super-resolution image frames and other image frames among the multiple image frames except those that have been super-resolution processed. In the generated target video, the quality of the target video is improved because the image frames with poor original image quality have been super-resolution processed.

[0088] In the embodiment of the present application, other image frames refer to image frames other than the image frames that have been super-resolution processed among the multiple image frames.

[0089] For example, there are 200 image frames, represented by [I0, I1...I198, I199]. Suppose the image frames after super-resolution are [I100, I101...I149, I150], and the other image frames are [I0, I1...I98, I99], [I151, I152...I198, I199]. Accordingly, the target video is generated based on the image frames after super-resolution [I100, I101...I149, I150] and the other image frames [I0, I1...I98, I99], [I151, I152...I198, I199].

[0090] In the embodiment of the present application, the streaming terminal side selectively performs super-resolution processing on the video in combination with the current real-time network quality and the image quality of the image frame, thereby ensuring the video quality in the video processing process based on web instant messaging even when the network quality or network conditions are poor (for example, insufficient available network bandwidth). That is, while ensuring the stability and performance of the system, the video quality is maximized, thereby also improving the user's viewing experience to a certain extent.

[0091] In one embodiment of the present application, another video processing method based on web page instant messaging is provided, and the video processing method based on web page instant messaging can be executed by the stream pulling terminal 102. Figure 3 As shown, the video processing method based on web instant messaging may include S301 to S303 and S202 to S203.

[0092] As described in the above-mentioned embodiment, the stream pulling terminal will perform image quality evaluation on each image frame in the pushed stream video. The embodiment of the present application will describe in detail the process of the stream pulling terminal performing image quality evaluation on each image frame in the pushed stream video.

[0093] S301 to S303 are described in detail as follows:

[0094] S301, for each image frame in the pushed video, obtaining the waiting time for the image frame to be decoded.

[0095] In the embodiment of the present application, the streaming terminal can obtain the waiting time for each image frame to be decoded, and then evaluate the image quality of each image frame based on the waiting time for each image frame.

[0096] In one embodiment of the present application, the process of obtaining the waiting time for the image frame to be decoded in S301 may include:

[0097] The image frame is stored in the jitter buffer area, and the start time when the image frame is stored in the jitter buffer area is recorded;

[0098] The timing starts from the start time, and the waiting time of the image frame waiting for decoding processing is obtained by counting.

[0099] That is, in an optional embodiment, the streaming terminal obtains a streaming video containing multiple image frames, and then stores the multiple image frames in a jitter buffer area. In the process of storing the multiple image frames in the jitter buffer area, the streaming terminal records the starting time when each image frame is stored in the jitter buffer area, and starts timing from the starting time corresponding to each image frame, thereby obtaining the waiting time for each image frame to be decoded.

[0100] In the optional embodiment, the jitter buffer area is a buffer mechanism in the web instant messaging transmission architecture, which aims to combat the instability of the network environment, accumulate the video in the jitter buffer area for a period of time before decoding and playing, thereby ensuring the smoothness of the video playback. In the optional embodiment, the size of the jitter buffer area can be flexibly adjusted to adapt to the changing network environment.

[0101] Among them, in an optional embodiment, a timer can be matched for each image frame (optionally, one timer corresponds to one thread), so that the timer corresponding to each image frame starts timing from the starting time corresponding to each image frame to obtain the waiting time of each image frame waiting for decoding processing, thereby ensuring the accuracy of the waiting time of each image frame waiting for decoding processing obtained by counting.

[0102] S302: If the waiting time is greater than a preset waiting time threshold, an image quality evaluation result indicating that the image quality is less than a preset image quality threshold is obtained.

[0103] S303: If the waiting time is less than or equal to a preset waiting time threshold, an image quality evaluation result is obtained for indicating that the image quality is greater than or equal to a preset image quality threshold.

[0104] It is understandable that the image quality of an image frame is usually related to the network quality of the network environment; therefore, in the embodiment of the present application, the image quality of each image frame is evaluated based on the waiting time of each image frame, including the following two cases:

[0105] Case 1: if the waiting time is greater than the preset waiting time threshold, the network quality of the current network environment is poor, and therefore, an image quality evaluation result representing an image quality less than the preset image quality threshold is obtained.

[0106] Case 2: If the waiting time is less than or equal to the preset waiting time threshold, the network quality of the current network environment is better at this time. Therefore, the image quality evaluation result obtained is used to characterize an image quality greater than or equal to the preset image quality threshold.

[0107] It should be noted that the above embodiments are mainly based on the perspective of image frames to reflect the network quality of the current network environment from the side, thereby achieving image quality evaluation of image frames.

[0108] In one embodiment of the present application, network indicator information of the current network environment may also be obtained to positively reflect the network quality of the current network environment, thereby achieving image quality evaluation of the image frame.

[0109] Among them, in an optional embodiment, network indicator information of the current network environment is obtained in real time, and image frames contained in the streaming video received in the current network environment are recorded; then, when it is detected that the network quality of the network environment represented by the network indicator information is poor, the image quality evaluation result of the image frames recorded during the same period is obtained as the image quality is less than a preset image quality threshold, or when it is detected that the network quality of the network environment represented by the network indicator information is good, the image quality evaluation result of the image frames recorded during the same period is obtained as the image quality is greater than or equal to the preset image quality threshold.

[0110] Among them, in the optional embodiment, the network indicator information is information related to network indicators, wherein the network indicators are indicators that can reflect the quality of the network, including but not limited to latency, packet loss rate, throughput and freeze rate.

[0111] By implementing an optional embodiment, the network indicator information of the current network environment is used to implement image quality assessment of the image frame, so that the image quality assessment result of the image frame can be obtained quickly and easily, providing strong support for the judgment of subsequent image frame super-resolution processing.

[0112] It should be noted that Figure 3 For detailed description of S202 to S203, please refer to Figure 2 S202 to S203 shown are not described in detail here.

[0113] In the embodiment of the present application, the streaming terminal side implements image quality assessment of the image frame by utilizing the waiting time of the image frame waiting for decoding processing, so that the image quality assessment result of the image frame can be obtained quickly and easily, providing strong support for the judgment of subsequent image frame super-resolution processing.

[0114] In one embodiment of the present application, another video processing method based on web page instant messaging is provided, and the video processing method based on web page instant messaging can be executed by the stream pulling terminal 102. Figure 4 As shown, the video processing method based on web instant messaging may include S401 to S402, S201, and S203.

[0115] As described in the above embodiments, the streaming terminal will make a judgment on the image frame super-resolution processing according to the image quality evaluation result. The embodiments of the present application will describe in detail the judgment process of the streaming terminal on the image frame super-resolution processing according to the image quality evaluation result.

[0116] S401 to S402 are described in detail as follows:

[0117] S401, marking image frames whose image quality is less than a preset image quality threshold among multiple image frames according to the image quality evaluation result.

[0118] It can be understood that since the pushed streaming video includes multiple image frames, in order to ensure the accuracy and convenience of super-resolution processing, the streaming terminal in the embodiment of the present application can first mark the image frames that need to be super-resolution processed according to the image quality evaluation results, so as to have a basis for super-resolution processing of the image frames.

[0119] In the embodiment of the present application, the image frames that need to be super-resolution are first marked according to the image quality assessment result, including the following two situations:

[0120] Case 1: if the image quality assessment result of the image frame indicates that the image quality of the image frame is less than a preset image quality threshold, then the image quality of the image frame is poor at this time. Therefore, the image frame needs to be marked so that the image frame can be super-resolution processed later.

[0121] Case 2: if the image quality assessment result of the image frame indicates that the image quality of the image frame is greater than or equal to a preset image quality threshold, then the image quality of the image frame is relatively good at this time, and therefore, there is no need to mark the image frame.

[0122] S402, performing super-resolution processing on the marked image frame to obtain a super-resolution image frame.

[0123] In the embodiment of the present application, the stream pulling terminal may perform super-resolution processing on the marked image frames to obtain super-resolution image frames.

[0124] For example, following the above example, the marked image frames are [I100, I101...I149, I150], then super-resolution is performed on the marked image frames [I100, I101...I149, I150] to obtain super-resolution image frames.

[0125] It should be noted that Figure 4 For detailed description of S201 and S203, please refer to Figure 2 S201 and S203 shown are not described in detail here.

[0126] In the embodiment of the present application, the streaming terminal side marks the image frames that need to be super-resolution processed according to the image quality evaluation results, so that when the image frames are super-resolution processed, the marked image frames can be super-resolution processed quickly and easily, and the super-resolution processing efficiency is high.

[0127] In one embodiment of the present application, another video processing method based on web page instant messaging is provided, and the video processing method based on web page instant messaging can be executed by the stream pulling terminal 102. Figure 5 As shown, the video processing method based on web instant messaging may include S501 to S502, S201, and S203.

[0128] As described in the above-mentioned embodiments, the streaming terminal will judge and process the image frame super-resolution processing according to the image quality evaluation results. The embodiments of the present application will describe in detail the judgment and processing process of the streaming terminal for performing the image frame super-resolution processing according to the image quality evaluation results.

[0129] S501 to S502 are described in detail as follows:

[0130] S501 , selecting a target image frame whose image quality is less than a preset image quality threshold from multiple image frames as a target image frame according to an image quality evaluation result, and selecting adjacent image frames of the target image frame.

[0131] In the embodiment of the present application, the streaming terminal can first select a target image frame whose image quality is less than a preset image quality threshold from multiple image frames as the target image frame according to the image quality evaluation result, and then select adjacent image frames of the target image frame.

[0132] In an embodiment of the present application, a target image frame refers to an image frame among multiple image frames whose image quality is less than a preset image quality threshold; an adjacent image frame refers to an image frame adjacent to the image position of the target image frame, wherein the image position refers to the position of the image frame in the streaming video. Exemplarily, the number of adjacent image frames may be one frame or multiple frames, for example, in the case of one frame, it may be the subsequent image frame or the previous image frame of the target image frame, etc., and in the case of multiple frames, it may be the subsequent image frame (which may be multiple subsequent image frames) and the previous image frame (which may be multiple subsequent image frames) of the target image frame, etc.

[0133] S502 , performing super-resolution processing on the target image frame based on feature information of the target image frame and feature information of adjacent image frames to obtain a super-resolution image frame.

[0134] In the embodiment of the present application, the streaming terminal selects the target image frame and the adjacent image frames of the target image frame, and then performs super-resolution processing on the target image frame based on the feature information of the target image frame and the feature information of the adjacent image frames to obtain the image frame after super-resolution processing. That is, in the embodiment of the present application, the super-resolution processing of the target image frame is realized by using the feature information of the target image frame and the adjacent image frames, so as to obtain the target image frame after super-resolution processing.

[0135] In one embodiment of the present application, the process of performing super-resolution processing on the target image frame based on the feature information of the target image frame and the feature information of the adjacent image frames in S502 to obtain the super-resolution image frame may include:

[0136] Extract features from the target image frame and the adjacent image frames respectively, and fuse the extracted features to obtain fused features;

[0137] Perform bidirectional propagation on the fused features to obtain the reconstructed features;

[0138] A super-resolution image frame corresponding to the target image frame is obtained based on the reconstructed features, and the super-resolution image frame is used as an image frame after super-resolution processing.

[0139] That is, in the optional embodiment, the streaming terminal first extracts and fuses features of the target image frame and adjacent image frames respectively to obtain fused features, then bidirectionally propagates the fused features to obtain reconstructed features, and then obtains the super-resolution image frame corresponding to the target image frame based on the reconstructed features, and uses the super-resolution image frame as the target image frame after super-resolution processing.

[0140] By implementing an optional embodiment, the target image frame is super-resolution processed by combining the feature information of the target image frame and image frames adjacent to the target image frame, taking the feature information between the image frames into consideration, thereby improving the image quality of the target image frame after super-resolution processing.

[0141] In one embodiment of the present application, the target image frame may be input into a video super-resolution network to determine the adjacent image frames of the target image frame through the video super-resolution network, and then the target image frame is super-resolved based on the feature information of the target image frame and the feature information of the adjacent image frames to obtain the super-resolved target image frame, and the super-resolved target image frame is output, thereby realizing the super-resolved processing of the target image frame. The specific processing process of the video super-resolution network may be the processing process described above, which will not be repeated here.

[0142] Among them, in the optional embodiment, the video super-resolution network can use EVSRNet, which is a lightweight and high-speed network, especially suitable for mobile devices with limited hardware configuration. In practical applications, the video super-resolution network can be flexibly selected according to the specific application scenario.

[0143] By implementing the optional embodiment, the super-resolution processing of the image frame is completed using the video super-resolution network, and the super-resolution efficiency and accuracy are higher. If EVSRNet is selected, it is very suitable for mobile devices with limited hardware configuration due to its lightweight and high-resolution characteristics, which greatly reduces the computing pressure of mobile devices.

[0144] It is understandable that the super-resolution processing process introduced above can be executed by the edge node, and in practical applications, it can be flexibly adjusted according to specific application scenarios.

[0145] It should be noted that Figure 5 For detailed description of S201 and S203, please refer to Figure 2 S201 and S203 shown are not described in detail here.

[0146] In the embodiment of the present application, the streaming terminal side performs super-resolution processing on the target image frame through a video super-resolution algorithm combined with the feature information of the target image frame and the image frames adjacent to the target image frame, taking into account the feature information between the image frames, so that the image quality of the target image frame after super-resolution processing is better and the super-resolution processing effect is good.

[0147] In one embodiment of the present application, another video processing method based on web page instant messaging is provided, and the video processing method based on web page instant messaging can be executed by the stream pulling terminal 102. Figure 6 As shown, the video processing method based on web instant messaging may include S601 to S603, S201, and S203.

[0148] As described in the above-mentioned embodiments, the streaming terminal will judge and process the image frame super-resolution processing according to the image quality evaluation results. The embodiments of the present application will describe in detail the judgment and processing process of the streaming terminal for performing the image frame super-resolution processing according to the image quality evaluation results.

[0149] S601 to S603 are described in detail as follows:

[0150] S601 , selecting an image frame whose image quality is less than a preset image quality threshold from multiple image frames as a target image frame according to an image quality evaluation result.

[0151] In the embodiment of the present application, the streaming terminal may first select a target image frame whose image quality is less than a preset image quality threshold from multiple image frames as the target image frame according to the image quality evaluation result.

[0152] In the embodiment of the present application, the target image frame refers to an image frame whose image quality is less than a preset image quality threshold among multiple image frames.

[0153] S602, based on the image quality of the target image frame, obtaining a target resolution of the target image frame; wherein different image qualities correspond to different target resolutions.

[0154] It is understandable that usually there are differences in the image quality of multiple target image frames.

[0155] To illustrate, for example, continuing with the previous example, the target image frames are [I100, I101...I149, I150], wherein the image qualities corresponding to the target image frames [I100, I101...I108, I109] are almost the same, for example, they can all be approximated by Iq1, and the image qualities corresponding to the target image frames [I110, I111...I149, I150] are almost the same, for example, they can all be approximated by Iq2, wherein Iq1≠Iq2.

[0156] Therefore, in the embodiment of the present application, the stream pulling terminal can obtain the target resolution of the target image frame based on the image quality of the target image frame.

[0157] In one embodiment of the present application, the process of acquiring the target resolution of the target image frame based on the image quality of the target image frame in S602 may include:

[0158] Finding a resolution matching the image quality of the target image frame from a preset image quality and resolution mapping relationship table;

[0159] The found resolution is used as the target resolution.

[0160] That is, in an optional embodiment, an image quality and resolution mapping relationship table is pre-set, and the target resolution of the target image frame can be quickly and easily determined by looking up the preset image quality and resolution mapping relationship table. Please refer to Table 1 below, which is an example of a preset image quality and resolution mapping relationship table.

[0161] Image Quality Resolution Iq1 R1 Iq2 R2 …… ……

[0162] Table 1

[0163] As shown in Table 1, following the previous example, the target image frames [I100, I101...I108, I109] correspond to image quality Iq1, and accordingly, the resolution R1 corresponding to the target image frames [I100, I101...I108, I109] is obtained; the target image frames [I110, I111...I149, I150] correspond to image quality Iq2, and accordingly, the resolution R2 corresponding to the target image frames [I110, I111...I149, I150] is obtained.

[0164] In the embodiment of the present application, there are differences in image quality between different target image frames, which may be determined by the waiting time for different target image frames to be decoded in the aforementioned embodiment.

[0165] For example, suppose the waiting time corresponding to the target image frames [I100, I101...I108, I109] is [t1, t2], where t1<t2, then the image quality corresponding to the target image frames [I100, I101...I108, I109] is determined to be Iq1; suppose the waiting time corresponding to the target image frames [I110, I111...I149, I150] is [t3, t4], where t3<t4, t1≠t3, t2≠t4, then the image quality corresponding to the target image frames [I110, I111...I149, I150] is determined to be Iq2.

[0166] Exemplarily, if t2<t3, then Iq1>Iq2, that is, the image quality corresponding to the target image frames [I100, I101…I108, I109] is better than the image quality corresponding to the target image frames [I110, I111…I149, I150]; accordingly, R1<R2, that is, the resolution corresponding to the target image frames [I100, I101…I108, I109] is smaller than the resolution corresponding to the target image frames [I110, I111…I149, I150].

[0167] In one embodiment of the present application, the image quality of the target image frame may be positively correlated with the resolution of the target image frame, that is, the better the image quality of the target image frame, the higher the resolution of the target image frame, and conversely, the worse the image quality of the target image frame, the lower the resolution of the target image frame.

[0168] In one embodiment of the present application, the image quality of the target image frame and the resolution of the target image frame may be negatively correlated, that is, the better the image quality of the target image frame, the lower the resolution of the target image frame, and conversely, the worse the image quality of the target image frame, the higher the resolution of the target image frame.

[0169] S603 , super-resolution processing is performed on the target image frame according to the target resolution of the target image frame to obtain a super-resolution image frame corresponding to the target image frame, and the super-resolution image frame is used as the image frame after super-resolution processing.

[0170] In the embodiment of the present application, the streaming terminal obtains the target resolution of the target image frame, and then can super-resolution the target image frame according to the target resolution of the target image frame to obtain the super-resolution image frame corresponding to the target image frame, and use the super-resolution image frame as the image frame after super-resolution processing.

[0171] To illustrate, following the previous example, for the target image frames [I100, I101...I108, I109], super-resolution processing is performed according to the resolution R1, and for the target image frames [I110, I111...I149, I150], super-resolution processing is performed according to the resolution R2, thereby obtaining the super-resolution image frames corresponding to the target image frames [I100, I101...I149, I150], and completing the corresponding super-resolution processing.

[0172] It is understandable that the super-resolution processing process introduced above can be executed by the edge node, and in practical applications, it can be flexibly adjusted according to specific application scenarios.

[0173] It should be noted that Figure 6 For detailed description of S201 and S203, please refer to Figure 2 S201 and S203 shown are not described in detail here.

[0174] In the embodiment of the present application, the streaming terminal side performs super-resolution processing of target image frames with different image qualities at different target resolutions, which is highly targeted and flexible and applicable to a wide range of scenarios.

[0175] In one embodiment of the present application, another video processing method based on web page instant messaging is provided, and the video processing method based on web page instant messaging can be executed by the stream pulling terminal 102. Figure 7As shown, the video processing method based on web instant messaging may include S701 to S702 and S201 to S202.

[0176] As introduced in the above-mentioned embodiment, the streaming terminal will generate a target video based on the image frames after super-resolution processing and other image frames among multiple image frames except those that have been super-resolution processed. The embodiment of the present application will describe in detail the process of the streaming terminal generating a target video based on the image frames after super-resolution processing and other image frames among multiple image frames except those that have been super-resolution processed.

[0177] S701 to S702 are described in detail as follows:

[0178] S701 , decoding the other image frames among the plurality of image frames except the image frames subjected to super-resolution processing according to a specified resolution to obtain decoded image frames.

[0179] In the embodiment of the present application, the streaming terminal can decode the other image frames among the multiple image frames except the image frames subjected to super-resolution processing according to the specified resolution to obtain the decoded image frames.

[0180] In the embodiment of the present application, the specified resolution refers to the resolution set when decoding the pushed stream video. Prior to the present application, the related art decoded all image frames in the pushed stream video according to the specified resolution, while the present application proposes to perform super-resolution decoding on some image frames in the pushed stream video and to perform decoding on some image frames at the specified resolution.

[0181] For example, following the above example, suppose the image frames after super-resolution processing are [I100, I101...I149, I150], and the other image frames are [I0, I1...I98, I99], [I151, I152...I198, I199]. Accordingly, the other image frames [I0, I1...I98, I99], [I151, I152...I198, I199] are decoded according to the specified resolution.

[0182] S702, combining the image frame after super-resolution processing and the image frame after decoding processing to generate a target video.

[0183] In the embodiment of the present application, the streaming terminal obtains the image frames after super-resolution processing (which are actually also the image frames after decoding processing, but here they are used to distinguish the different resolutions used during decoding) and the image frames after decoding processing. Then, the image frames after super-resolution processing and the image frames after decoding processing can be combined to generate the target video.

[0184] It should be noted that Figure 7 For detailed description of S201 to S202, please refer to Figure 2S201 to S202 shown are not described in detail here.

[0185] In the embodiment of the present application, the streaming terminal side combines the image frames after super-resolution processing and the image frames after decoding processing, which can quickly generate the target video and provide strong support for the rapid display of the target video.

[0186] It should be noted that Figures 2 to 7 The embodiment shown is explained from the perspective of the streaming terminal. Figures 8 to 9 The implementation details of the technical solution of the embodiment of the present application are elaborated in detail from the perspective of the streaming terminal:

[0187] See also Figure 8 , Figure 8 FIG. 1 is a flow chart of a video processing method based on web page instant messaging according to an embodiment of the present application. The video processing method based on web page instant messaging can be executed by the streaming terminal 101. Figure 8 As shown, the video processing method based on webpage instant messaging at least includes S801 to S804, which are described in detail as follows:

[0188] S801, obtaining network quality information of the current network environment of the streaming terminal.

[0189] In the embodiment of the present application, the streaming terminal will obtain the network quality information of the current network environment in real time. The network quality information is information used to characterize the quality of the network environment.

[0190] S802: If the network quality information is less than a preset network quality threshold, a target resolution matching the network quality information is obtained; wherein the target resolution is positively correlated with the network quality information.

[0191] In the embodiment of the present application, if the network quality information is less than a preset network quality threshold, the network quality of the network environment is poor at this time, and therefore, a target resolution matching the network quality information can be obtained.

[0192] In the embodiment of the present application, the target resolution is positively correlated with the network quality information, that is, the better the network quality, the higher the target resolution, and conversely, the worse the network quality, the lower the target resolution. It is understandable that in the embodiment of the present application, when the network quality is poor, a lower target resolution will be obtained.

[0193] S803, encoding the image frames in the collected video according to the target resolution to obtain an encoded video.

[0194] In the embodiment of the present application, the streaming terminal obtains a lower target resolution, and then can encode the image frames in the captured video according to the lower target resolution to obtain the encoded video.

[0195] S804, pushing the encoded video to the streaming terminal through web instant messaging, so that the streaming terminal performs super-resolution processing on the image frames in the received video whose image quality parameters are less than a preset image quality threshold to generate a target video.

[0196] In the embodiment of the present application, the push stream terminal obtains the encoded video, and then pushes the encoded video to the pull stream terminal through web instant messaging; accordingly, the pull stream terminal receives the encoded video sent by the push stream terminal, and performs corresponding processing on the encoded video to generate the target video. The process of the pull stream terminal generating the target video can be found in the introduction of the aforementioned embodiment, which will not be repeated here.

[0197] In the embodiment of the present application, the streaming terminal side selectively downsamples the video in combination with the current real-time network quality, thereby ensuring the video quality during the video processing process based on web instant messaging even when the network quality or network conditions are poor (for example, insufficient available network bandwidth).

[0198] In one embodiment of the present application, another video processing method based on web page instant messaging is provided, and the video processing method based on web page instant messaging can be executed by the streaming terminal 101. Fig. 9 As shown, the video processing method based on web instant messaging may include S901 to S902 and S801 to S803.

[0199] S901 to S902 are described in detail as follows:

[0200] S901, obtaining a target streaming bitrate corresponding to network quality information; wherein the target streaming bitrate is positively correlated with the network quality information.

[0201] In the embodiment of the present application, if the network quality information is less than a preset network quality threshold, the network quality of the network environment is poor at this time. Therefore, a target streaming bit rate matching the network quality information can also be obtained.

[0202] In the embodiment of the present application, the target streaming bitrate is positively correlated with the network quality information, that is, the better the network quality, the higher the target streaming bitrate, and conversely, the worse the network quality, the lower the target streaming bitrate. It can be understood that in the embodiment of the present application, when the network quality is poor, a lower target streaming bitrate will be obtained.

[0203] S902: Push the encoded video to the streaming terminal through web instant messaging according to the target streaming bit rate.

[0204] In the embodiment of the present application, the streaming terminal obtains a lower target streaming bit rate, and then can stream the encoded video to the streaming terminal through web instant messaging according to the lower target streaming bit rate.

[0205] It should be noted that Fig. 9 For detailed description of S801 to S803, please refer to Figure 8 S801 to S803 shown are not described in detail here.

[0206] In the embodiment of the present application, the streaming terminal side selectively downsamples the video in combination with the current real-time network quality and flexibly selects the streaming bit rate, thereby ensuring the video quality in the video processing process based on web instant messaging even when the network quality or network conditions are poor (for example, insufficient available network bandwidth).

[0207] The following is a detailed description of a specific scenario of an embodiment of the present application:

[0208] See also Fig.10 , which is a schematic diagram of the composition and streaming of the streaming terminal (i.e., the video sending end, also called the WebRTC sending end), mainly including the video data acquisition module, video downsampling module, video encoding module, RTP packaging / sending module, bitstream control module, etc.; among which:

[0209] Video data acquisition module: mainly used to acquire video through camera, screen capture, etc., wherein the acquired video data is referred to as original video data in the embodiment of the present application.

[0210] Video downsampling module: mainly used to selectively downsample the original video data, that is, reduce the resolution of the original video data, according to the network quality / condition of the network environment (such as the bandwidth and congestion of the network).

[0211] Video encoding module: mainly used to compress the original video data (usually the bit rate of the original video data is relatively high) using the video encoding algorithm, and compress it into a transmittable bit stream. Optionally, considering the deployment scenario of the mobile terminal, the VP9 video encoder can be used in the embodiment of the present application. It can be understood that the computational consumption of the VP9 video encoder is relatively small.

[0212] Stream control module: mainly used to dynamically adjust the target encoding bit rate and target resolution of the original video data according to the network quality / condition of the network environment (such as the network bandwidth and congestion) so as to adapt to the network quality / condition of the current network environment, thereby ensuring the smoothness of video transmission.

[0213] RTP packaging / sending module: It can be understood that WebRTC uses RTP for data transmission. Therefore, the RTP packaging / sending module is mainly used to package the encoded video data into RTP packets and send them to the streaming terminal. Optionally, the packaged RTP packets in the embodiment of the present application can select the optimal transmission path through the ICE (Interactive Connectivity Establishment) protocol. Optionally, the forward error correction (FEC) redundant packet technology can be used in the embodiment of the present application to add redundant data during the transmission process, thereby ensuring the reliability of video transmission.

[0214] The embodiment of the present application improves the video quality by adding a video downsampling module. Among them, before video encoding, the video downsampling module selectively downsamples the original video data according to the network quality / condition of the current network environment to ensure that the video quality can still be guaranteed to a certain extent when the network quality is poor or the network condition is poor (for example, insufficient available network bandwidth). That is, before the bitstream control module adjusts the target bitstream according to the network quality / condition of the current network environment, the video downsampling module will first reduce the resolution of the video to avoid the phenomenon of information loss and video quality loss caused by compression as much as possible, thereby improving the video quality to a certain extent.

[0215] In short, when the network quality / condition of the current network environment deteriorates, the original bitrate control module will reduce the target bitrate input to the video encoding module to adapt to the changeable network environment; however, after adding the video downsampling module, the bitrate control module will not only reduce the target bitrate, but also control the video downsampling module to downsample the original video data first, and then by reducing the resolution, it can avoid the information loss and video quality loss caused by compression, thereby improving the video quality to a certain extent.

[0216] See also Fig.11 , which is a schematic diagram of the composition and streaming of the streaming terminal (i.e., the receiving end of the video, also called the WebRTC receiving end), mainly including the RTP receiving module, the frame quality assessment module, the video super-resolution module, the decoding module, the jitterbuffer, the video display module, etc.; among which:

[0217] RTP receiving module: mainly used to receive video data in the form of RTP packets from the sender, and decapsulate and decrypt the received video data to obtain the original video data. Specifically, the receiver first decapsulates the video data using the RTP protocol, and then decrypts the data using SRTP (Secure Real-time Transport Protocol).

[0218] Jitter buffer: It is understandable that the WebRTC receiver uses a jitter buffer to handle network jitter. The main function of the jitter buffer mechanism is to cache the original video data and ensure the continuity of the original video data playback by controlling the size of the jitter buffer area and the time to clear the cache.

[0219] Frame quality assessment module: mainly used to assess the image quality of the image frames in the jitter buffer area. It can be understood that if the image quality of an image frame is very good, it can be directly decoded and played, and if the quality of an image frame is poor, the image frame can be marked for subsequent super-resolution processing.

[0220] Optionally, the frame quality assessment module can assess the image quality of the image frame by the waiting time (wait_time) of the image frame waiting for decoding in the jitter buffer area. When the network quality is good, the waiting time of the image frame in the jitter buffer area is generally short, and the image frame will be sent to the decoder contained in the decoding module for decoding after reaching the head of the queue; when the network condition deteriorates, the waiting time of the image frame is relatively long. It can be understood that if the waiting time of an image frame is short, it means that the network condition is good, the quality of the image frame is high, and thus super-resolution processing is not required. Conversely, if the waiting time of an image frame is long, it means that the network condition is poor, the quality of the image frame may be low, and thus super-resolution processing is required to improve the video quality.

[0221] Video super-resolution module: mainly used to convert low-resolution image frames into high-resolution image frames, that is, to perform super-resolution processing on the aforementioned marked image frames. Optionally, considering the deployment scenario of the mobile terminal, the embodiment of the present application can select a lightweight and high-speed EVSRNet super-resolution algorithm suitable for the mobile terminal, wherein the EVSRNet super-resolution algorithm has a very outstanding performance in lightweight and real-time performance. Specifically, it has been proved that its processing speed on the Snapdragon 865Soc is 11.3ms / f, which is very suitable for the mobile terminal deployment scenario of the embodiment of the present application.

[0222] Decoding module: mainly used to decode the image frames in the jitter buffer area when they are ready. Optionally, considering the deployment scenario of the mobile terminal, the VP9 video decoder can be used in the embodiment of the present application. It can be understood that the computational consumption of the VP9 video decoder is relatively small.

[0223] Video display module: mainly used to display and play the decoded video.

[0224] In the embodiment of the present application, the video quality is improved by adding a frame quality assessment module and a video super-resolution module. Since the super-resolution processing consumes a lot of computational consumption, it is necessary to consider the computing power and network conditions of the mobile terminal to ensure the stability and performance of the mobile terminal. To this end, in the embodiment of the present application, with the help of the frame quality assessment module, the super-resolution module dynamically decides whether to perform super-resolution processing based on the real-time network conditions and the image quality of the image frame. In this way, under the joint action of the frame quality assessment module and the video super-resolution module, the purpose of balancing the video quality and the performance of the mobile terminal is achieved, that is, the video quality is maximized while ensuring the stability and performance of the mobile terminal.

[0225] Fig.12 FIG. 1 is a block diagram of a video processing device based on web page instant messaging, shown in one embodiment of the present application. Fig.12 As shown, the video processing device based on web page instant messaging includes:

[0226] The acquisition and evaluation module 1201 is configured to acquire a streaming video sent by a streaming terminal via web instant messaging, wherein the streaming video includes a plurality of image frames, and perform image quality evaluation on each image frame in the streaming video;

[0227] The super-resolution module 1202 is configured to perform super-resolution processing on the image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation result to obtain the image frames after super-resolution processing;

[0228] The generating module 1203 is configured to generate a target video based on the image frame after the super-resolution processing and other image frames among the multiple image frames except the image frames after the super-resolution processing.

[0229] In one embodiment of the present application, the acquisition and evaluation module 1201 is specifically configured as follows:

[0230] For each image frame in the pushed video, obtaining a waiting time for the image frame to be decoded;

[0231] If the waiting time is greater than a preset waiting time threshold, an image quality evaluation result is obtained for indicating that the image quality is less than the preset image quality threshold;

[0232] If the waiting time is less than or equal to the preset waiting time threshold, an image quality evaluation result representing an image quality greater than or equal to the preset image quality threshold is obtained.

[0233] In one embodiment of the present application, the acquisition and evaluation module 1201 is further specifically configured as follows:

[0234] storing the image frame in a jitter buffer area, and recording a start time when the image frame is stored in the jitter buffer area;

[0235] The timing is started from the starting time, and the waiting time of the image frame waiting for decoding processing is obtained by counting.

[0236] In one embodiment of the present application, the super-resolution module 1202 is specifically configured as follows:

[0237] Marking the image frames whose image quality is less than a preset image quality threshold among the multiple image frames according to the image quality evaluation result;

[0238] Perform super-resolution processing on the marked image frame to obtain a super-resolution image frame.

[0239] In one embodiment of the present application, the super-resolution module 1202 is specifically configured as follows:

[0240] Selecting a target image frame whose image quality is less than a preset image quality threshold from the multiple image frames as a target image frame according to the image quality evaluation result, and selecting adjacent image frames of the target image frame;

[0241] The target image frame is super-resolved based on the feature information of the target image frame and the feature information of the adjacent image frames to obtain a super-resolved image frame.

[0242] In one embodiment of the present application, the super-resolution module 1202 is further specifically configured as follows:

[0243] Extracting features from the target image frame and the adjacent image frame respectively, and fusing the extracted features to obtain fused features;

[0244] Performing bidirectional propagation on the fused features to obtain reconstructed features;

[0245] A super-resolution image frame corresponding to the target image frame is obtained based on the reconstructed features, and the super-resolution image frame is used as the image frame after super-resolution processing.

[0246] In one embodiment of the present application, the super-resolution module 1202 is specifically configured as follows:

[0247] Selecting, from the plurality of image frames, an image frame whose image quality is less than a preset image quality threshold as a target image frame according to the image quality evaluation result;

[0248] Based on the image quality of the target image frame, obtaining a target resolution of the target image frame; wherein different image qualities correspond to different target resolutions;

[0249] The target image frame is super-resolved according to the target resolution of the target image frame to obtain a super-resolved image frame corresponding to the target image frame, and the super-resolved image frame is used as the image frame after super-resolved processing.

[0250] In one embodiment of the present application, the generation module 1203 is specifically configured as follows:

[0251] Decoding the other image frames among the plurality of image frames except the image frames subjected to super-resolution processing according to the specified resolution to obtain decoded image frames;

[0252] The image frame after the super-resolution processing and the image frame after the decoding processing are combined to generate a target video.

[0253] In one embodiment of the present application, the acquisition and evaluation module 1201 is specifically configured as follows:

[0254] Receiving a real-time transport protocol data packet sent by a streaming terminal via web instant messaging;

[0255] Decapsulating the data packet based on the real-time transport protocol to obtain decapsulated video data;

[0256] The video data is decrypted to obtain the streaming video.

[0257] Fig.13 FIG. 1 is a block diagram of a video processing device based on web page instant messaging, shown in one embodiment of the present application. Fig.13 As shown, the video processing device based on web page instant messaging includes:

[0258] The first acquisition module 1301 is configured to acquire network quality information of the network environment in which the streaming terminal is currently located;

[0259] The second acquisition module 1302 is configured to acquire a target resolution matching the network quality information if the network quality information is less than a preset network quality threshold; wherein the target resolution is positively correlated with the network quality information;

[0260] The encoding module 1303 is configured to encode the image frames in the collected video according to the target resolution to obtain an encoded video;

[0261] The streaming module 1304 is configured to stream the encoded video to the streaming terminal via web instant messaging, so that the streaming terminal performs super-resolution processing on image frames in the received video whose image quality parameters are less than a preset image quality threshold to generate a target video.

[0262] In one embodiment of the present application, the streaming module 1304 is specifically configured as follows:

[0263] Obtaining a target streaming bitrate corresponding to the network quality information; wherein the target streaming bitrate is positively correlated with the network quality information;

[0264] The encoded video is pushed to the streaming terminal through web instant messaging according to the target streaming bit rate.

[0265] It should be noted that the apparatus provided in the aforementioned embodiment and the method provided in the aforementioned embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment.

[0266] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the video processing method based on web instant messaging as described above.

[0267] Fig.14 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application.

[0268] It should be noted that Fig.14 The computer system 1400 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0269] like Fig.14As shown, the computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1402 or the program loaded from the storage part 1408 to the random access memory (RAM) 1403, such as executing the method in the above embodiment. In the RAM 1403, various programs and data required for system operation are also stored. The CPU 1401, the ROM 1402 and the RAM 1403 are connected to each other through the bus 1404. The input / output (I / O) interface 1405 is also connected to the bus 1404.

[0270] The following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, a mouse, etc.; an output section 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed so that a computer program read therefrom is installed into the storage section 1408 as needed.

[0271] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1409, and / or installed from a removable medium 1411. When the computer program is executed by a central processing unit (CPU) 1401, various functions defined in the system of the present application are executed.

[0272] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable media may be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0273] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0274] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0275] Another aspect of the present application also provides a computer-readable medium on which a computer program is stored, and when the computer program is executed by a processor, the video processing method based on web page instant messaging is implemented as described above. The computer-readable medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0276] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable medium. A processor of a computer device reads the computer instruction from the computer-readable medium, and the processor executes the computer instruction, so that the computer device executes the video processing method based on web page instant messaging provided in each of the above embodiments.

[0277] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A video processing method based on web page instant messaging, characterized in that: include: Acquire a streaming video sent by a streaming terminal through webpage instant messaging, wherein the streaming video includes a plurality of image frames, and perform image quality evaluation on each image frame in the streaming video; According to the image quality evaluation result, super-resolution processing is performed on the image frames whose image quality is less than a preset image quality threshold value to obtain the image frames after super-resolution processing; A target video is generated based on the image frame after the super-resolution processing and other image frames among the multiple image frames except the image frames after the super-resolution processing.

2. The method according to claim 1, characterized in that The performing image quality assessment on each image frame in the streaming video includes: For each image frame in the pushed video, obtaining a waiting time for the image frame to be decoded; If the waiting time is greater than a preset waiting time threshold, an image quality evaluation result is obtained for indicating that the image quality is less than the preset image quality threshold; If the waiting time is less than or equal to the preset waiting time threshold, an image quality evaluation result representing an image quality greater than or equal to the preset image quality threshold is obtained.

3. The method according to claim 2, characterized in that The obtaining of the waiting time for the image frame to be decoded includes: storing the image frame in a jitter buffer area, and recording a start time when the image frame is stored in the jitter buffer area; The timing is started from the starting time, and the waiting time of the image frame waiting for decoding processing is obtained by counting.

4. The method according to claim 1, characterized in that: The step of performing super-resolution processing on image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation result to obtain image frames after super-resolution processing comprises: Marking the image frames whose image quality is less than a preset image quality threshold among the multiple image frames according to the image quality evaluation result; Perform super-resolution processing on the marked image frame to obtain a super-resolution image frame.

5. The method according to claim 1, characterized in that The step of performing super-resolution processing on image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation result to obtain image frames after super-resolution processing comprises: Selecting a target image frame whose image quality is less than a preset image quality threshold from the multiple image frames as a target image frame according to the image quality evaluation result, and selecting adjacent image frames of the target image frame; The target image frame is super-resolved based on the feature information of the target image frame and the feature information of the adjacent image frames to obtain a super-resolved image frame.

6. The method according to claim 5, characterized in that The super-resolution processing of the target image frame based on the feature information of the target image frame and the feature information of the adjacent image frames to obtain the image frame after the super-resolution processing includes: Extracting features from the target image frame and the adjacent image frame respectively, and fusing the extracted features to obtain fused features; Performing bidirectional propagation on the fused features to obtain reconstructed features; A super-resolution image frame corresponding to the target image frame is obtained based on the reconstructed features, and the super-resolution image frame is used as the image frame after super-resolution processing.

7. The method according to claim 1, characterized in that The step of performing super-resolution processing on image frames whose image quality is less than a preset image quality threshold according to the image quality evaluation result to obtain image frames after super-resolution processing comprises: Selecting, from the plurality of image frames, an image frame whose image quality is less than a preset image quality threshold as a target image frame according to the image quality evaluation result; Based on the image quality of the target image frame, obtaining a target resolution of the target image frame; wherein different image qualities correspond to different target resolutions; The target image frame is super-resolved according to the target resolution of the target image frame to obtain a super-resolved image frame corresponding to the target image frame, and the super-resolved image frame is used as the image frame after super-resolved processing.

8. The method according to claim 1, characterized in that The generating a target video based on the image frame after the super-resolution processing and other image frames among the multiple image frames except the image frames after the super-resolution processing comprises: Decoding the other image frames among the plurality of image frames except the image frames subjected to super-resolution processing according to the specified resolution to obtain decoded image frames; The image frame after the super-resolution processing and the image frame after the decoding processing are combined to generate a target video.

9. The method according to any one of claims 1 to 8, characterized in that The step of obtaining the streaming video sent by the streaming terminal via web instant messaging includes: Receiving a real-time transport protocol data packet sent by a streaming terminal via web instant messaging; Decapsulating the data packet based on the real-time transport protocol to obtain decapsulated video data; The video data is decrypted to obtain the streaming video.

10. A video processing method based on webpage instant messaging, characterized in that: include: Obtain the network quality information of the current network environment of the streaming terminal; If the network quality information is less than a preset network quality threshold, obtaining a target resolution matching the network quality information; wherein the target resolution is positively correlated with the network quality information; Encoding the image frames in the captured video according to the target resolution to obtain an encoded video; The encoded video is pushed to the streaming terminal through web instant messaging, so that the streaming terminal performs super-resolution processing on image frames in the received video whose image quality parameters are less than a preset image quality threshold to generate a target video.

11. The method according to claim 10, characterized in that The step of pushing the encoded video to the streaming terminal via web instant messaging includes: Obtaining a target streaming bitrate corresponding to the network quality information; wherein the target streaming bitrate is positively correlated with the network quality information; The encoded video is pushed to the streaming terminal through web instant messaging according to the target streaming bit rate.

12. A video processing device based on webpage instant messaging, characterized in that: include: An acquisition and evaluation module is configured to acquire a streaming video sent by a streaming terminal via web instant messaging, wherein the streaming video includes a plurality of image frames, and perform image quality evaluation on each image frame in the streaming video; A super-resolution module is configured to perform super-resolution processing on image frames whose image quality is less than a preset image quality threshold according to an image quality evaluation result, to obtain image frames after super-resolution processing; The generating module is configured to generate a target video based on the image frame after the super-resolution processing and other image frames among the multiple image frames except the image frames after the super-resolution processing.

13. A video processing device based on web page instant messaging, characterized in that: include: A first acquisition module is configured to acquire network quality information of the network environment in which the streaming terminal is currently located; A second acquisition module is configured to acquire a target resolution matching the network quality information if the network quality information is less than a preset network quality threshold; wherein the target resolution is positively correlated with the network quality information; An encoding module configured to encode the image frames in the collected video according to the target resolution to obtain an encoded video; The streaming module is configured to stream the encoded video to the streaming terminal through web instant messaging, so that the streaming terminal performs super-resolution processing on image frames in the received video whose image quality parameters are less than a preset image quality threshold to generate a target video.

14. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the electronic device, enables the electronic device to implement the video processing method based on web instant messaging as described in any one of claims 1 to 11.

15. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video processing method based on web instant messaging according to any one of claims 1 to 11 is implemented.

Images