Video processing method, device, storage medium and electronic device
By receiving video frames and keyframe information in video data packets, judging frame drop situations and sending adjustment instructions, dynamically adjusting video encoding strategies, the problem of lack of flexibility in video encoding in the prior art is solved, and encoding efficiency and adaptability are improved .
Patent Information
- Application Number
- CN202211337948.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-10-28
AI Technical Summary
Existing electronic devices lack flexibility in video encoding, resulting in coding according to a fixed strategy every time, unable to adapt to changes in network conditions, resulting in video packet loss and low encoding efficiency.
By receiving video data packets sent by external devices, obtaining video frames and keyframe information, judging frame drop situations, and sending adjustment instructions to external devices when preset conditions are met, adjusting video encoding strategies to improve encoding flexibility.
It realizes dynamic adjustment of video encoding strategy according to network conditions, reduces the loss of video frames, and improves the flexibility and transmission efficiency of video encoding.
Smart Images

Figure CN115866335B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of electronic technology, and in particular, relates to a video processing method, device, storage medium and electronic device. Background Art
[0002] With the continuous development of the Internet, people hope to transmit more multimedia information on the Internet, such as video information or audio information. Taking video information as an example, in the related art, the electronic device encodes the video to be encoded according to a fixed video encoding strategy each time to obtain a video frame, and encapsulates it into a video data packet, and then sends the video data packet to an external device through the network. Since the electronic device encodes the video to be encoded according to a fixed video encoding strategy each time, the flexibility of video encoding is poor. Summary of the invention
[0003] The embodiments of the present application provide a video processing method, device, storage medium and electronic device, which can make the flexibility of video encoding relatively good.
[0004] In a first aspect, an embodiment of the present application provides a video processing method, which is applied to an electronic device, including:
[0005] Receiving a video data packet sent by an external device, the video data packet including a video frame and key frame information obtained by encoding the video to be encoded by the external device, the key frame information being used to indicate a key frame in the video frame;
[0006] According to the video frames and the key frame information in the video data packet, acquiring the frame loss information of the video data packet, wherein the frame loss information is used to indicate the loss status of the key frame;
[0007] If the frame loss information meets a preset condition, adjustment indication information is sent to the external device, where the adjustment indication information is used to instruct the external device to adjust a video encoding strategy.
[0008] In a second aspect, an embodiment of the present application provides a video processing method, which is applied to an electronic device, including:
[0009] Encode the video to be encoded according to the video encoding strategy to obtain a video frame, and obtain key frame information, where the key frame information is used to indicate a key frame in the video frame;
[0010] Encapsulating the video frame and the key frame information into a video data packet and sending it to an external device;
[0011] receiving adjustment indication information sent by an external device, the adjustment indication information being used to instruct the electronic device to adjust the video encoding strategy, the adjustment indication information being sent by the external device when frame loss information obtained according to video frames and key frame information present in a received video data packet meets a preset condition, the frame loss information being used to indicate a loss status of the key frame;
[0012] The video encoding strategy is adjusted according to the adjustment indication information.
[0013] In a third aspect, an embodiment of the present application provides a video processing device, applied to an electronic device, including:
[0014] A data packet receiving module, used to receive a video data packet sent by an external device, wherein the video data packet includes a video frame and key frame information obtained by encoding the video to be encoded by the external device, and the key frame information is used to indicate a key frame in the video frame;
[0015] An information acquisition module, used to acquire frame loss information of the video data packet according to the video frame and the key frame information in the video data packet, wherein the frame loss information is used to indicate the loss status of the key frame;
[0016] The information sending module is used to send adjustment indication information to the external device if the frame loss information meets a preset condition, wherein the adjustment indication information is used to instruct the external device to adjust the video encoding strategy.
[0017] In a fourth aspect, an embodiment of the present application provides a video processing device, which is applied to an electronic device, including:
[0018] The video encoding module is used to encode the video to be encoded according to the video encoding strategy, obtain the video frame, and obtain the key frame information;
[0019] A data packet sending module, used for encapsulating the video frame and the key frame information into a video data packet and sending it to an external device;
[0020] an information receiving module, configured to receive adjustment indication information sent by an external device, the adjustment indication information being used to instruct the electronic device to adjust the video encoding strategy, the adjustment indication information being sent by the external device when frame loss information obtained according to video frames and key frame information present in a received video data packet meets a preset condition, the key frame information being used to indicate a key frame in the video frame, and the frame loss information being used to indicate a loss status of the key frame;
[0021] A strategy adjustment module is used to adjust the video encoding strategy according to the adjustment indication information.
[0022] In a fifth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the video processing method provided by the embodiment of the present application.
[0023] In a sixth aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the processor is used to execute the video processing method provided in the embodiment of the present application by calling a computer program stored in the memory.
[0024] In an embodiment of the present application, a video data packet sent by an external device is received, the video data packet includes video frames and key frame information obtained by the external device encoding the video to be encoded, the key frame information is used to indicate the key frame in the video frame; according to the video frame and the key frame information existing in the video data packet, the frame loss information of the video data packet is obtained, the frame loss information is used to indicate the loss status of the key frame; if the frame loss information meets the preset condition, the adjustment indication information is sent to the external device, the adjustment indication information is used to instruct the external device to adjust the video encoding strategy. Thus, when the frame loss information meets the preset condition, the electronic device can instruct the external device to adjust the video encoding strategy, so that the external device encodes the video to be encoded according to the adjusted video encoding strategy, and then the external device does not need to encode the video to be encoded according to a fixed video encoding strategy every time, and the flexibility of video encoding is relatively good. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.
[0026] Figure 1 It is a flowchart of the video processing method provided in an embodiment of the present application.
[0027] Figure 2 It is a flowchart of another video processing method provided in an embodiment of the present application.
[0028] Figure 3 It is a structural schematic diagram of a video processing device provided in an embodiment of the present application.
[0029] Figure 4 It is a structural schematic diagram of another video processing device provided in an embodiment of the present application.
[0030] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0031] It should be noted that the terms "first", "second" and "third" in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules that are not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products or devices.
[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0033] The embodiments of the present application provide a video processing method, a video processing device, a storage medium and an electronic device, wherein the execution subject of the video processing method may be the video processing device provided in the embodiments of the present application, or an electronic device integrating the video processing device, wherein the video processing device may be implemented in hardware or software. The electronic device may be a device such as a smart phone, a tablet computer, a PDA, a laptop computer, etc. that is equipped with a processor and has video processing capabilities.
[0034] See also Figure 1 , Figure 1 : is a flow chart of a video processing method provided in an embodiment of the present application, which is applied to an electronic device. The flow chart may include:
[0035] In 101, a video data packet sent by an external device is received, where the video data packet includes a video frame and key frame information obtained by encoding by the external device, where the key frame information is used to indicate a key frame in the video frame.
[0036] In this embodiment, the external device encodes the video to be encoded that needs to be sent to the electronic device according to the corresponding video encoding strategy to obtain a video frame. The external device then obtains key frame information. The key frame information is used to indicate the key frames in the video frame, that is, to indicate which video frames in the video frame are key frames, such as which position the key frame exists. After obtaining the video frame and the key frame information, the external device encapsulates the video frame and the key frame information into a video data packet and sends it to the electronic device, and the electronic device receives the video data packet sent by the external device.
[0037] The external device can be trained to obtain a key frame judgment model. After obtaining the video frame, the external device can judge the video frame through the key frame judgment model to determine which video frames in the video frame are key frames, thereby obtaining key frame information. The key frame judgment model can only judge key frames of the same importance level, or can judge key frames of different importance levels. Specifically, the corresponding model can be trained according to actual judgment requirements. The importance level can be set by the user according to, for example, the content of the key frame.
[0038] In 102, frame loss information of the video data packet is obtained according to the video frame and key frame information in the video data packet, where the frame loss information is used to indicate the loss status of the key frame.
[0039] It is understandable that due to unstable network conditions, packet loss usually occurs during the transmission of video data packets on the network. Therefore, in this embodiment, after receiving the video data packet sent by the external device, the electronic device can obtain the frame loss information of the video data packet based on the video frame and key frame information in the video data packet. In this embodiment, the frame loss information is used to indicate the loss status of the key frame. In other words, in this embodiment, only the loss status of the key frame is considered, such as whether the key frame is lost, the position of the lost key frame, the number of lost key frames, etc.
[0040] For example, an electronic device can be trained to obtain a key frame judgment model. After obtaining a video data packet, the external device can use the key frame judgment model to determine which video frames in the video data packet are key frames, and then combine the key frame information to determine whether the key frames are lost, and which key frames are specifically lost, such as the key frames at which position, and then obtain the frame loss information.
[0041] In 103, if the frame loss information meets the preset condition, adjustment instruction information is sent to the external device, where the adjustment instruction information is used to instruct the external device to adjust the video encoding strategy.
[0042] In this embodiment, after obtaining the frame loss information, the electronic device may determine whether the frame loss information meets the preset condition. If the frame loss information meets the preset condition, the electronic device sends adjustment instruction information to the external device to instruct the external device to adjust the video encoding strategy. If the frame loss information does not meet the preset condition, the electronic device may not process it. The preset condition may be pre-set in the electronic device.
[0043] In an optional embodiment, the video encoding strategy includes the encoding density of the key frame and / or the encoding quality of the video frame. For example, if the frame loss information meets the preset conditions, the electronic device can instruct the external device to adjust the encoding density of the key frame and / or the encoding quality of the video frame.
[0044] For example, assuming that the preset condition is the number of key frames lost in a video data packet, such as the number of key frames lost continuously in a video data packet is greater than a second preset number, the electronic device instructs the external device to adjust the encoding density of the key frames and / or the encoding quality of the video frames when the frame loss information indicates that the number of key frames lost in the video data packet is greater than the second preset number. The quality of the video frame can be measured by resolution, whether noise reduction processing is performed, whether it has an HDR effect, etc. The second preset number can be set by the user or by the electronic device based on certain rules.
[0045] It is understandable that the number of key frames lost in the video data packet being greater than the second preset number also reflects to a certain extent that the current network condition is poor. When the network condition is poor, under the premise of the same coding density, the probability of losing key frames is also greater than when the network condition is good. Therefore, the electronic device can instruct the external device to reduce the coding density of the key frames, thereby reducing the probability of losing key frames, and then reducing the number of lost key frames. When the network condition is poor, the better the coding quality of the video frame, the slower the transmission speed. In order to increase the transmission speed, the electronic device can also instruct the external device to reduce the coding quality of the video frame.
[0046] In this embodiment, by receiving a video data packet sent by an external device, the video data packet includes video frames and key frame information obtained by the external device encoding the video to be encoded, and the key frame information is used to indicate the key frame in the video frame; according to the video frame and the key frame information existing in the video data packet, the frame loss information of the video data packet is obtained, and the frame loss information is used to indicate the loss status of the key frame; if the frame loss information meets the preset condition, the adjustment indication information is sent to the external device, and the adjustment indication information is used to instruct the external device to adjust the video encoding strategy. Therefore, when the frame loss information meets the preset condition, the electronic device can instruct the external device to adjust the video encoding strategy, so that the external device encodes the video to be encoded according to the adjusted video encoding strategy, and then the external device does not need to encode the video to be encoded according to a fixed video encoding strategy every time, and the flexibility of video encoding is relatively good.
[0047] In an optional embodiment, if the frame loss information meets the preset condition, before sending the adjustment indication information to the external device, the method further includes:
[0048] If the frame loss information of a first preset number of consecutive video data packets all indicate that a key frame is lost, it is determined that the frame loss information meets a preset condition; or,
[0049] If the key frames lost as indicated by the frame loss information of a first preset number of consecutive video data packets account for a first proportion greater than a first preset proportion of the key frames indicated by the key frame information of the first preset number of consecutive video data packets, it is determined that the frame loss information meets the preset condition.
[0050] It is understandable that if the frame loss information of multiple consecutive video data packets, such as the first preset number, all indicate the loss of key frames, or the frame loss information of multiple consecutive video data packets, such as the first preset number, indicates that the first proportion of the key frames indicated by the key frame information of the first preset number of consecutive video data packets is greater than the first preset proportion, it means that the current network condition is poor and the phenomenon of key frame loss is prone to occur. Therefore, the above conditions can be used as preset conditions. When the above preset conditions are met, in order to reduce the probability of key frame loss, the electronic device can instruct the external device to adjust the video encoding strategy, such as reducing the encoding density of the key frame. When the above preset conditions are met, in order to increase the transmission speed of the video data packet, the electronic device can also instruct the external device to reduce the encoding quality of the video frame.
[0051] The first preset number and the first preset ratio may be set by a user or by the electronic device based on certain rules.
[0052] In an optional embodiment, if the frame loss information satisfies a preset condition, before sending the adjustment indication information to the external device, the method further includes:
[0053] If the second proportion of key frames lost indicated by the frame loss information to the key frames indicated by the key frame information is greater than a second preset proportion, it is determined that the frame loss information meets the preset condition; or,
[0054] If the frame loss information indicates that the number of lost key frames is greater than a second preset number, it is determined that the frame loss information meets a preset condition.
[0055] It is understandable that if the frame loss information of the video data packet indicates that the second proportion of the lost key frames in the key frames indicated by the key frame information is greater than the second preset proportion, or the frame loss information of the video data packet indicates that the number of lost key frames is greater than the second preset number, it means that the current network condition is poor and the phenomenon of key frame loss is prone to occur. Therefore, the above conditions can be used as preset conditions. When the above preset conditions are met, in order to reduce the probability of key frame loss, the electronic device can instruct the external device to adjust the video encoding strategy, such as reducing the encoding density of the key frame. When the above preset conditions are met, in order to increase the transmission speed of the video data packet, the electronic device can also instruct the external device to reduce the encoding quality of the video frame.
[0056] The second preset ratio may be set by a user or by the electronic device based on a certain rule.
[0057] In an optional embodiment, the key frames include key frames of different importance levels. If the frame loss information meets a preset condition, before sending the adjustment indication information to the external device, the following is further included:
[0058] If the frame loss information indicates that the lost key frames include key frames whose importance level is greater than a preset importance level, it is determined that the frame loss information satisfies a preset condition.
[0059] It is understandable that if the frame loss information of the video data packet indicates that a key frame with an importance level greater than the preset importance level is lost, it means that the current network condition is poor and it is easy to lose key frames with an importance level greater than the preset importance level. Therefore, the above conditions can be used as preset conditions. When the above preset conditions are met, in order to reduce the probability of loss of key frames with an importance level greater than the preset importance level, the electronic device can instruct the external device to adjust the video encoding strategy, such as reducing the encoding density of key frames with an importance level greater than the preset importance level. When the above preset conditions are met, in order to increase the transmission speed of the video data packet, the electronic device can also instruct the external device to reduce the encoding quality of the video frame.
[0060] For example, the user can set the importance levels of different video frames in advance according to the content of the video frames, and then train the corresponding neural network model based on the different video frames and their corresponding importance levels until the model converges. After the video frames are subsequently encoded, the importance levels of the encoded video frames can be determined by the trained neural network model. The key frame information can also include the importance levels of the key frames, so that when the lost key frames are subsequently determined based on the frame loss information, it can be determined based on the key frame information whether the lost key frames are key frames with an importance level greater than the preset importance level.
[0061] The preset importance level may be set by the user or by the electronic device based on certain rules.
[0062] In an optional embodiment, the video processing method further includes:
[0063] If the frame loss information meets the preset condition, retransmission indication information is sent to the external device, and the retransmission indication information is used to instruct the external device to resend the video data packet.
[0064] In this embodiment, considering that the frame loss information meets the preset conditions, it usually means that the corresponding video data packet has lost key frames to a serious extent, that is, the quality of the video data packet is poor, resulting in poor playback quality. In order to improve the playback quality, the electronic device can instruct the external device to resend the video data packet in order to obtain a video data packet with relatively better quality.
[0065] In an optional embodiment, the video processing method further includes:
[0066] Acquire a video data packet to be compensated and its corresponding frame loss information to be compensated, the video data packet to be compensated being a video data packet whose key frame is lost as indicated by its corresponding frame loss information to be compensated;
[0067] Decoding the video data packet to be compensated to obtain the video frame to be compensated;
[0068] A compensated video frame is obtained according to the information of the lost frame to be compensated and the video frame to be compensated.
[0069] When the frame loss information of the video data packet does not meet the preset conditions, the video data packet may also have the phenomenon of losing key frames. For example, if the number of key frames lost in a certain video data packet is less than or equal to the second preset number, the electronic device may not instruct the external device to resend the video data packet. The loss of key frames will inevitably affect the quality of video playback. Therefore, the electronic device can determine these video data packets as video data packets to be compensated, and determine the frame loss information corresponding to these video data packets as frame loss information to be compensated. Subsequently, the electronic device can decode the video data packets to be compensated to obtain the video data frames to be compensated. The electronic device can obtain the compensated video frames based on the frame loss information to be compensated and the video frames to be compensated.
[0070] In an optional embodiment, obtaining a compensated video frame according to the to-be-compensated lost frame information and the to-be-compensated video frame includes:
[0071] Determine the position information of the lost key frame according to the information of the lost frame to be compensated;
[0072] Determine, from the video frame to be compensated, an adjacent key frame adjacent to the lost key frame according to the position information;
[0073] Interpolation is performed based on adjacent key frames to obtain compensated video frames.
[0074] In this embodiment, the electronic device can compensate the video frame by interpolation. Specifically, the electronic device can determine the position information of the lost key frame according to the information of the lost frame to be compensated. The electronic device then determines the adjacent key frame adjacent to the lost key frame from the video frame to be compensated according to the position information. The electronic device then performs interpolation processing based on the adjacent key frames to obtain the compensated video frame.
[0075] It can be understood that if there is no key frame adjacent to the lost key frame in the video frame to be compensated, or there is only one key frame adjacent to the lost key frame, the electronic device can also obtain a video data packet adjacent to the video data packet to be compensated, decode the video data packet to obtain a corresponding video frame, and then determine the adjacent key frame adjacent to the lost key frame from the video frame.
[0076] In an optional embodiment, obtaining a compensated video frame according to the to-be-compensated lost frame information and the to-be-compensated video frame includes:
[0077] According to the information of the lost frames to be compensated and the video frames to be compensated, a compensated video frame is obtained through a video compensation model.
[0078] It can be understood that the electronic device can pre-train the corresponding model based on the sample frame loss information and the sample video frames obtained by decoding the sample video data packets (including video data packets with lost key frames and video data packets without lost key frames) to obtain a video compensation model for obtaining compensated video frames based on the frame loss information to be compensated and the video frames to be compensated. When the electronic device subsequently obtains the frame loss information to be compensated and the video frames to be compensated, the electronic device can obtain the compensated video frames according to the frame loss information to be compensated and the video frames to be compensated through the video compensation model.
[0079] In an optional embodiment, after decoding the video data packet to be compensated to obtain the video frame to be compensated, the method further includes:
[0080] If the image quality of the video frame to be compensated is lower than the preset image quality, the video frame to be compensated is subjected to image quality enhancement processing to obtain an enhanced video frame;
[0081] According to the information of the lost frame to be compensated and the video frame to be compensated, a compensated video frame is obtained, including:
[0082] A compensated video frame is obtained according to the information of the lost frame to be compensated and the enhanced video frame.
[0083] It is understandable that when the image quality of the video frame to be compensated is lower than the preset image quality, the electronic device can also perform image quality enhancement processing on the video frame to be compensated to obtain an enhanced video frame, and then obtain a compensated video frame based on the frame loss information to be compensated and the enhanced video frame.
[0084] In an optional embodiment, the electronic device may pre-train a corresponding image quality enhancement model, so that when a video frame to be compensated whose image quality is lower than a preset image quality is obtained, the image quality enhancement model may be used to perform image quality enhancement processing on the video frame to be compensated to obtain an enhanced video frame.
[0085] Among them, the image quality of the video frame can be measured by resolution, whether noise reduction is performed, whether it has HDR effect, etc.
[0086] In an optional embodiment, if the frame rate of the video frame to be compensated is less than the preset frame rate, the electronic device may also perform frame rate improvement processing on the video frame to be compensated to obtain an improved video frame, and then obtain a compensated video frame based on the frame loss information to be compensated and the enhanced video frame.
[0087] In an optional embodiment, the external device may be a device for collecting the corresponding video to be encoded, and the electronic device may be a device for performing subsequent processing according to the received video data packets, such as beautification, adding a virtual background, etc. The external device and the electronic device may also be the sending end and the receiving end of a video call or the like.
[0088] In an optional embodiment, if many video frames are lost, such as a certain video data packet or a plurality of consecutive video data packets (a first preset number of video data packets) has a number of lost video frames greater than a preset number, there will be further video compensation, that is, hierarchical compensation. The preset number can be set by the user or by the electronic device based on certain rules.
[0089] See also Figure 2 , Figure 2 : is a flow chart of another video processing method provided by an embodiment of the present application, which is applied to an electronic device. The flow chart may include:
[0090] In 201, the video to be encoded is encoded according to the video encoding strategy to obtain video frames and key frame information.
[0091] In 202, the video frame and key frame information are encapsulated into a video data packet and sent to an external device.
[0092] In 203, adjustment indication information sent by an external device is received, and the adjustment indication information is used to instruct the electronic device to adjust the video encoding strategy. The adjustment indication information is sent by the external device when the frame loss information obtained based on the video frame and key frame information existing in the received video data packet meets the preset conditions. The key frame information is used to indicate the key frame in the video frame, and the frame loss information is used to indicate the loss status of the key frame.
[0093] In 204, the video encoding strategy is adjusted according to the adjustment indication information.
[0094] It can be understood that steps 201 to 204 correspond to the steps of the sending end, and steps 101 to 103 correspond to the steps of the receiving end, and the processing flows of the two are corresponding. Therefore, the specific implementation of steps 201 to 204 can refer to the previous embodiments and will not be repeated here.
[0095] See also Figure 3 , Figure 3 The video processing device 300 is applied to an electronic device, and includes: a data packet receiving module 301 , an information obtaining module 302 , and an information sending module 303 .
[0096] The data packet receiving module 301 is used to receive a video data packet sent by an external device, wherein the video data packet includes video frames and key frame information obtained by encoding the video to be encoded by the external device, and the key frame information is used to indicate a key frame in the video frame.
[0097] The information acquisition module 302 is used to acquire the frame loss information of the video data packet according to the video frame and the key frame information in the video data packet, wherein the frame loss information is used to indicate the loss status of the key frame.
[0098] The information sending module 303 is used to send adjustment indication information to the external device if the frame loss information meets a preset condition, wherein the adjustment indication information is used to instruct the external device to adjust the video encoding strategy.
[0099] In an optional embodiment, the information sending module 303 can be used to: if the frame loss information of a first preset number of consecutive video data packets all indicates the loss of key frames, then determine that the frame loss information meets the preset condition; or if the frame loss information of a first preset number of consecutive video data packets indicates that the lost key frames account for a first proportion of the key frames indicated by the key frame information of a first preset number of consecutive video data packets that is greater than a first preset proportion, then determine that the frame loss information meets the preset condition.
[0100] In an optional embodiment, the information sending module 303 can be used to: determine that the frame loss information satisfies a preset condition if a second proportion of key frames lost as indicated by the frame loss information among the key frames indicated by the key frame information is greater than a second preset proportion; or determine that the frame loss information satisfies the preset condition if the number of key frames lost as indicated by the frame loss information is greater than a second preset number.
[0101] In an optional embodiment, the key frames include key frames of different importance levels, and the information sending module 303 can be used to: if the frame loss information indicates that the lost key frames include key frames with an importance level greater than a preset importance level, determine that the frame loss information meets a preset condition.
[0102] In an optional embodiment, the video encoding strategy includes encoding density of key frames and / or encoding quality of video frames.
[0103] In an optional embodiment, the information sending module 303 may be used to: if the frame loss information satisfies a preset condition, send retransmission indication information to the external device, wherein the retransmission indication information is used to instruct the external device to resend the video data packet.
[0104] In an optional embodiment, the video processing device 300 may further include a video compensation module, which may be used to: obtain a video data packet to be compensated and its corresponding frame loss information to be compensated, wherein the video data packet to be compensated is a video data packet whose corresponding frame loss information to be compensated indicates a lost key frame; decode the video data packet to be compensated to obtain a video frame to be compensated; and obtain a compensated video frame based on the frame loss information to be compensated and the video frame to be compensated.
[0105] In an optional embodiment, the video compensation module can be used to: determine the position information of the lost key frame based on the information of the lost frame to be compensated; determine the adjacent key frames adjacent to the lost key frame from the video frame to be compensated based on the position information; and perform interpolation processing based on the adjacent key frames to obtain a compensated video frame.
[0106] In an optional embodiment, the video compensation module may be used to obtain a compensated video frame through a video compensation model according to the frame loss information to be compensated and the video frame to be compensated.
[0107] In an optional embodiment, the video compensation module can be used to: if the image quality of the video frame to be compensated is lower than the preset image quality, perform image quality enhancement processing on the video frame to be compensated to obtain an enhanced video frame; and obtain a compensated video frame based on the frame loss information to be compensated and the enhanced video frame.
[0108] See also Figure 4 , Figure 4 The video processing device 400 is applied to an electronic device, and includes: a video encoding module 401 , a data packet sending module 402 , an information receiving module 403 and a policy adjustment module 404 .
[0109] The video encoding module 401 is used to encode the video to be encoded according to the video encoding strategy, obtain video frames, and acquire key frame information.
[0110] The data packet sending module 402 is used to encapsulate the video frame and the key frame information into a video data packet and send it to an external device.
[0111] The information receiving module 403 is used to receive adjustment indication information sent by an external device, wherein the adjustment indication information is used to instruct the electronic device to adjust the video encoding strategy. The adjustment indication information is sent by the external device when the frame loss information obtained based on the video frame and key frame information present in the received video data packet meets a preset condition, the key frame information is used to indicate the key frame in the video frame, and the frame loss information is used to indicate the loss status of the key frame.
[0112] The strategy adjustment module 404 is used to adjust the video encoding strategy according to the adjustment indication information.
[0113] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the video processing method provided in this embodiment.
[0114] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the video processing method provided in the present embodiment by calling a computer program stored in the memory.
[0115] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0116] The electronic device 500 may include components such as a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the electronic device 500 may also include a screen.
[0117] The processor 501 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the electronic device as a whole.
[0118] The memory 502 may be used to store applications and data. The applications stored in the memory 502 include executable codes. The applications may be composed of various functional modules. The processor 501 executes various functional applications and data processing by running the applications stored in the memory 502.
[0119] In this embodiment, the processor 501 in the electronic device will load the executable code corresponding to the process of one or more applications into the memory 502 according to the following instructions, and the processor 501 will run the application stored in the memory 502, thereby implementing the video processing method described in any embodiment of the present application.
[0120] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the video processing method above, and will not be repeated here.
[0121] The video processing device provided in the embodiment of the present application belongs to the same concept as the video processing method in the above embodiment. Any method provided in the video processing method embodiment can be run on the video processing device. The specific implementation process is detailed in the video processing method embodiment and will not be repeated here.
[0122] It should be noted that, for the video processing method of the embodiment of the present application, a person skilled in the art can understand that all or part of the process of implementing the video processing method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the video processing method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RUM), etc.
[0123] It is understandable that in the specific implementation of this application, user information is involved, such as application usage behavior data, logs and other related data. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0124] For the video processing device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.
[0125] The above is a detailed introduction to a video processing method, device, storage medium and electronic device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A video processing method, applied to electronic equipment, characterized in that: include: Receiving a video data packet sent by an external device, the video data packet including a video frame and key frame information obtained by encoding the video to be encoded by the external device, the key frame information being used to indicate a key frame in the video frame; According to the video frames and the key frame information in the video data packet, frame loss information of the video data packet is obtained, where the frame loss information is used to indicate the loss status of the key frame, and the loss status includes whether a key frame is lost or the number of lost key frames; If the frame loss information of a first preset number of consecutive video data packets all indicate a lost key frame, determining that the frame loss information satisfies a preset condition; or, If the key frames lost as indicated by the frame loss information of a first preset number of consecutive video data packets account for a first proportion greater than a first preset proportion of the key frames indicated by the key frame information of the first preset number of consecutive video data packets, it is determined that the frame loss information meets the preset condition; If the frame loss information meets a preset condition, adjustment indication information is sent to the external device, where the adjustment indication information is used to instruct the external device to adjust a video encoding strategy.
2. The video processing method according to claim 1, characterized in that: If the frame loss information satisfies a preset condition, before sending the adjustment indication information to the external device, the method further includes: If the second proportion of the key frames lost indicated by the frame loss information to the key frames indicated by the key frame information is greater than a second preset proportion, it is determined that the frame loss information meets the preset condition; or, If the frame loss information indicates that the number of lost key frames is greater than a second preset number, it is determined that the frame loss information meets a preset condition.
3. The video processing method according to claim 1, characterized in that: The key frames include key frames of different importance levels. If the frame loss information meets a preset condition, before sending the adjustment indication information to the external device, the method further includes: If the frame loss information indicates that the lost key frames include key frames whose importance level is greater than a preset importance level, it is determined that the frame loss information meets a preset condition.
4. The video processing method according to claim 1, characterized in that: The video encoding strategy includes the encoding density of key frames and / or the encoding quality of video frames.
5. The video processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: If the frame loss information meets a preset condition, retransmission indication information is sent to the external device, where the retransmission indication information is used to instruct the external device to resend the video data packet.
6. The video processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a video data packet to be compensated and its corresponding frame loss information to be compensated, wherein the video data packet to be compensated indicates a video data packet whose key frame is lost due to the corresponding frame loss information to be compensated; Decoding the video data packet to be compensated to obtain a video frame to be compensated; A compensated video frame is obtained according to the to-be-compensated lost frame information and the to-be-compensated video frame.
7. The video processing method according to claim 6, characterized in that: The step of obtaining a compensated video frame according to the to-be-compensated lost frame information and the to-be-compensated video frame comprises: Determining the position information of the lost key frame according to the information of the lost frame to be compensated; Determining, according to the position information, adjacent key frames adjacent to the lost key frame from the video frame to be compensated; Interpolation processing is performed according to the adjacent key frames to obtain compensated video frames.
8. The video processing method according to claim 6, characterized in that: The step of obtaining a compensated video frame according to the to-be-compensated lost frame information and the to-be-compensated video frame comprises: A compensated video frame is obtained through a video compensation model according to the to-be-compensated lost frame information and the to-be-compensated video frame.
9. The video processing method according to claim 6, characterized in that: After decoding the video data packet to be compensated to obtain the video frame to be compensated, the method further includes: If the image quality of the video frame to be compensated is lower than the preset image quality, performing image quality enhancement processing on the video frame to be compensated to obtain an enhanced video frame; The step of obtaining a compensated video frame according to the to-be-compensated lost frame information and the to-be-compensated video frame comprises: A compensated video frame is obtained according to the to-be-compensated lost frame information and the enhanced video frame.
10. A video processing method, applied to an electronic device, characterized in that: include: Encode the video to be encoded according to the video encoding strategy to obtain a video frame, and obtain key frame information, where the key frame information is used to indicate a key frame in the video frame; Encapsulating the video frame and the key frame information into a video data packet and sending it to an external device; Receiving adjustment indication information sent by an external device, the adjustment indication information is used to instruct the electronic device to adjust the video encoding strategy, the adjustment indication information is sent by the external device when frame loss information obtained according to video frames and key frame information present in the received video data packet meets a preset condition, the frame loss information is used to indicate a loss status of the key frame, the loss status includes whether a key frame is lost or the number of lost key frames, if frame loss information of a first preset number of consecutive video data packets all indicate that key frames are lost, then the frame loss information meets the preset condition, or if frame loss information of a first preset number of consecutive video data packets indicates that a first proportion of lost key frames in the key frames indicated by the key frame information of the first preset number of consecutive video data packets is greater than a first preset proportion, then the frame loss information meets the preset condition; The video encoding strategy is adjusted according to the adjustment indication information.
11. A video processing device, applied to an electronic device, characterized in that: include: A data packet receiving module, used to receive a video data packet sent by an external device, wherein the video data packet includes a video frame and key frame information obtained by encoding the video to be encoded by the external device, and the key frame information is used to indicate a key frame in the video frame; an information acquisition module, configured to acquire frame loss information of the video data packet according to the video frames and the key frame information in the video data packet, wherein the frame loss information is used to indicate a loss status of the key frame, and the loss status includes whether a key frame is lost or the number of lost key frames; An information sending module is used to determine that the frame loss information meets a preset condition if the frame loss information of a first preset number of consecutive video data packets all indicates that key frames are lost, or to determine that the frame loss information meets the preset condition if a first proportion of key frames indicated by the key frame information of a first preset number of consecutive video data packets is greater than a first preset proportion, and if the frame loss information meets the preset condition, send adjustment indication information to the external device, wherein the adjustment indication information is used to instruct the external device to adjust a video encoding strategy.
12. A video processing device, applied to an electronic device, characterized in that: include: The video encoding module is used to encode the video to be encoded according to the video encoding strategy, obtain the video frame, and obtain the key frame information; A data packet sending module, used for encapsulating the video frame and the key frame information into a video data packet and sending it to an external device; an information receiving module, configured to receive adjustment indication information sent by an external device, the adjustment indication information being used to instruct the electronic device to adjust the video encoding strategy, the adjustment indication information being sent by the external device when frame loss information obtained according to video frames and key frame information present in a received video data packet meets a preset condition, the key frame information being used to indicate a key frame in the video frame, the frame loss information being used to indicate a loss condition of the key frame, the loss condition including whether a key frame is lost or the number of lost key frames, if frame loss information of a first preset number of consecutive video data packets all indicate lost key frames, then the frame loss information meets the preset condition, or if frame loss information of a first preset number of consecutive video data packets indicates that a first proportion of lost key frames in the key frames indicated by the key frame information of the first preset number of consecutive video data packets is greater than a first preset proportion, then the frame loss information meets the preset condition; A strategy adjustment module is used to adjust the video encoding strategy according to the adjustment indication information.
13. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is run on a computer, the computer executes the video processing method according to any one of claims 1 to 9 or 10.
14. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the video processing method according to any one of claims 1 to 9 or 10 by calling the computer program stored in the memory.
