Video optimization method, device, equipment and storage medium

By combining cloud and edge processing resources, deep content perception analysis is performed based on the network attributes and image characteristics of video frames, and dynamically adjusting the encoding mode, the problem of ignoring edge computing resources in the existing technology is solved, and the problem of ignoring edge computing resources is realized in a changing network environment. Efficient video streaming and high-quality video experience.

CN118764684BActive Publication Date: 2024-11-26BEIJING HONGYUAN INTELLIGENT CONTROL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411237934.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-26
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

Existing video optimization technologies ignore the potential of edge computing resources and the low latency and high efficiency advantages brought by processing data close to the user side, making it difficult to ensure high definition and fast loading video streaming in a changing network environment.

Method used

By determining the target processing method based on the network attributes and local image characteristics of the video frame, conducting deep content-aware frame analysis, combining cloud and edge processing resources, dynamically adjusting the encoding mode to optimize video streaming.

Benefits of technology

Improve the performance and efficiency of video streaming, ensure that users enjoy a high-quality video experience in a changing network environment, and improve service availability and user satisfaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118764684B_ABST
    Figure CN118764684B_ABST
Patent Text Reader

Abstract

The present application discloses a video optimization method, device, equipment and storage medium, which relates to the field of video frame technology. The method includes: determining the target processing method of the current video frame according to the first network attribute and local image features of the current video frame; the target processing method includes cloud processing or edge processing; controlling the target device corresponding to the target processing method to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain the analysis result of the current video frame; processing the current video frame according to the analysis result of the current video frame to obtain the target video frame; determining the target encoding mode of the target video frame according to the second network attribute and semantic content features of the target video frame; playing the target video frame based on the target encoding mode. The above technical scheme can improve the performance and efficiency of video streaming transmission, while ensuring that users can still enjoy high-quality video experience in a changing network environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, in particular to the field of video frame technology, and specifically to a video optimization method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of Internet technology, video streaming and playback has become a daily network behavior. High definition, no buffering, and fast loading have become basic expectations of users. In order to meet these expectations, a variety of video optimization technologies have emerged.

[0003] Some solutions have begun to use the powerful computing power of the cloud to handle video encoding and optimization tasks. These solutions usually try to optimize bandwidth usage and quality assurance during video transmission by re-encoding and quality control in the cloud. However, these technologies often ignore the potential of edge computing resources and the low latency and high efficiency advantages brought by processing data close to the user. Summary of the invention

[0004] The present application provides a video optimization method, apparatus, device and storage medium to improve the performance and efficiency of video stream transmission.

[0005] According to one aspect of the present application, a video optimization method is provided, the method comprising:

[0006] Determining a target processing method for the current video frame according to a first network attribute and a local image feature of the current video frame; the target processing method includes cloud processing or edge processing;

[0007] Controlling a target device corresponding to the target processing mode to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain an analysis result of the current video frame; and processing the current video frame according to the analysis result of the current video frame to obtain a target video frame;

[0008] Determining a target encoding mode of the target video frame according to a second network attribute and a semantic content feature of the target video frame;

[0009] Based on the target encoding mode, the target video frame is played.

[0010] According to another aspect of the present application, a video optimization device is provided, the device comprising:

[0011] A processing mode determination module, used to determine a target processing mode of the current video frame according to a first network attribute and a local image feature of the current video frame; the target processing mode includes cloud processing or edge processing;

[0012] A video frame determination module, used to control a target device corresponding to a target processing mode to perform deep content-aware frame analysis on a confidence level, a scene change detection result, and a frame redundancy index of a current video frame to obtain an analysis result of the current video frame; and to process the current video frame according to the analysis result of the current video frame to obtain a target video frame;

[0013] A coding mode determination module, used for determining a target coding mode of a target video frame according to a second network attribute and a semantic content feature of the target video frame;

[0014] The video frame playing module is used to play the target video frame based on the target encoding mode.

[0015] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0016] one or more processors;

[0017] A memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the video optimization methods provided in the embodiments of the present application.

[0019] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any one of the video optimization methods provided in the embodiments of the present application is implemented.

[0020] This application determines the target processing method of the current video frame according to the first network attribute and local image features of the current video frame; the target processing method includes cloud processing or edge processing; the target device corresponding to the target processing method is controlled to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain the analysis result of the current video frame; the current video frame is processed according to the analysis result of the current video frame to obtain the target video frame; the target encoding mode of the target video frame is determined according to the second network attribute and semantic content features of the target video frame; and the target video frame is played based on the target encoding mode. The above technical scheme can improve the performance and efficiency of video streaming transmission, while ensuring that users can still enjoy high-quality video experience in a changing network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a video optimization method provided according to Embodiment 1 of the present application;

[0022] Figure 2 is a flow chart of a video optimization method provided according to Embodiment 2 of the present application;

[0023] Figure 3 is a structural schematic diagram of a video optimization device provided according to Embodiment 3 of the present application;

[0024] Figure 4 It is a structural schematic diagram of an electronic device for implementing the video optimization method of an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] In addition, it should be noted that in the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of relevant data such as the first network attributes and local image features involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Embodiment 1

[0029] Figure 1 This is a flow chart of a video optimization method provided according to the first embodiment of the present application. This embodiment is applicable to the case of adaptive dynamic optimization of the playback of the target video, and can be executed by a video optimization device. The video optimization device can be implemented in the form of hardware and / or software. The video optimization device can be configured in a computer device, such as a server. Figure 1 As shown, the method includes:

[0030] S110. Determine a target processing method for the current video frame according to a first network attribute and a local image feature of the current video frame; the target processing method includes cloud processing or edge processing.

[0031] Among them, the current video frame refers to a frame of image transmitted continuously in the video stream. The first network attribute refers to the network characteristics of the current video frame and the network environment in which it is located, which may include at least one of network delay, network throughput and data packet loss rate. Local image features refer to some local features of the video frame itself, which may include at least one of image content, motion properties and details. The target processing method refers to the decision made based on the characteristics of the current video frame and the network attributes to determine whether to process in the cloud or at the edge. Cloud processing refers to uploading the video frame to a remote server for processing. Edge processing refers to processing close to the data source or on a local device.

[0032] Optionally, a decision score of the current video frame is determined based on the first network attribute and local image features of the current video frame; if the decision score meets the cloud processing conditions, cloud processing is determined as the target processing method for the current video frame; if the decision score meets the edge processing conditions, edge processing is determined as the target processing method for the current video frame.

[0033] Among them, the evaluation score used to determine whether the processing task of the current video frame should be executed by the cloud or the edge during decision scoring. The cloud processing condition refers to the condition under which the processing task of the current video frame is executed by the cloud, which can be when the decision score is greater than the decision score threshold, cloud processing is determined as the target processing method. Edge processing refers to the condition under which the processing task of the current video frame is executed by the edge, which can be when the decision score is less than the decision score threshold, cloud processing is determined as the target processing method.

[0034] Furthermore, if the decision score satisfies both the edge processing condition and the cloud processing condition, one of the edge processing and the cloud processing is selected as the target processing method for the current video frame.

[0035] Exemplarily, when the decision score is equal to the decision score threshold, one of edge processing and cloud processing is selected as the target processing method for the current video frame.

[0036] Among them, the decision scoring threshold is artificially set according to actual conditions or experience values, and the embodiments of the present application do not make specific limitations on this.

[0037] S120, controlling the target device corresponding to the target processing method to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain the analysis result of the current video frame; and processing the current video frame according to the analysis result of the current video frame to obtain the target video frame.

[0038] The target device refers to the device that performs the processing task of the current video frame, which may include the cloud or edge. The cloud refers to a remote cloud server or data center. The edge refers to the device end close to the data source, usually near the data collection point, such as a sensor, camera or IoT device. Confidence is used to indicate the reliability or accuracy of the current frame, involving the evaluation of frame clarity, noise level, motion blur, etc. The scene change detection result refers to the detection of whether a scene change has occurred between the current frame and the previous frame (or multiple frames). The scene change includes the appearance, disappearance, movement of objects, or the overall change of the scene. The frame redundancy index measures the degree of repetition or redundancy between the current frame and the previous and next frames. A high frame redundancy index indicates that the current frame may be similar or repeated to the previous frame, while a low frame redundancy index indicates that the current frame may contain new information or changes. Deep content-aware frame analysis refers to the process of using deep learning and computer vision technology to perform advanced analysis and understanding of video frames. The analysis result refers to the evaluation result of the importance of the current video frame, which determines whether the current video frame should be retained, discarded, or transmitted first. The target video frame refers to the video frame currently available for playback.

[0039] Optionally, the target video frame is obtained by processing the current video frame according to the analysis result of the current video frame. If the analysis result of the current video frame meets the interpolation condition, a candidate video frame is generated according to the motion features between the previous video frame of the current video frame and the next video frame of the current video frame, and the current video frame is replaced by the candidate video frame; the visual features of the candidate video frame are adjusted according to the first visual features between the candidate video frame and the previous video frame, and the second visual features between the candidate video frame and the next video frame, to obtain the target video frame.

[0040] Among them, the interpolation condition refers to the condition that the current video frame needs to be discarded and a new video frame needs to be inserted. The previous video frame refers to the video frame immediately before the current video frame in the video time sequence. The next video frame refers to the video frame immediately after the current video frame in the video time sequence. The motion feature refers to the motion between two consecutive video frames, and the moving direction and speed of the object or scene can be detected by methods such as optical flow estimation. The candidate video frame refers to a new video frame inserted in the time sequence corresponding to the current video frame. The first visual feature refers to the visual difference or change between the candidate video frame and the previous video frame, and can include at least one of a temporal continuity index, a spatial information continuity index, and a visual discontinuity index. The second visual feature refers to the visual difference or change between the candidate video frame and the next video frame, and can include at least one of a temporal continuity index, a spatial information continuity index, and a visual discontinuity index.

[0041] Furthermore, the target video frame can be determined in the following manner: if the first visual feature and the second visual feature satisfy the adjustment condition of the visual feature, the visual features of the candidate video frame are adjusted according to the first visual feature and the second visual feature until the adjustment condition of the visual feature is no longer satisfied, thereby obtaining the target video frame.

[0042] Among them, the adjustment conditions of the visual features are artificially set according to actual conditions or experience values, and the embodiments of the present application do not make specific limitations on this.

[0043] In an optional implementation, a quality score of the candidate video frame may be determined based on the first visual feature and the second visual feature; and the visual features of the candidate video frame may be adjusted based on the quality score.

[0044] Exemplarily, the quality score can be determined by the following formula:

[0045]

[0046] Among them, I refers to the quality score of the candidate video frame, which determines the visual quality and perception continuity of the newly generated frame. T refers to the temporal gradient, which is used to evaluate the temporal continuity of the frame with the previous and next frames. This is calculated by analyzing the speed and direction of frame changes in the time series. S refers to the spatial gradient, which evaluates the continuity of spatial information within the frame, which involves the layout and organization of pixels within the frame to ensure a smooth transition of spatial information. V refers to the visual discontinuity index, which measures the visual difference between the newly inserted frame and the original frame sequence. High discontinuity may cause visual jumps and affect the viewing experience. , and e are weight factors used to adjust the impact of temporal and spatial continuity as well as visual discontinuity in the total score.

[0047] It should be noted that this formula uses adaptive interpolation enhancement technology. The core purpose of adaptive interpolation enhancement technology is to generate continuous frames by calculation at the frame discontinuity caused by frame loss or other reasons, thereby reducing the jitter or jump of the picture perceived by the audience. By analyzing the temporal gradient and spatial gradient, this technology can ensure that the newly generated frames are highly consistent with the original video sequence visually. Temporal gradient analysis helps to ensure the natural flow of action, while spatial gradient ensures the coherence of the spatial relationship between scenes and objects. The negative impact of visual discontinuity is offset by the e weight, giving priority to reducing visual conflicts.

[0048] Understandably, this technology intelligently enhances video content by taking into account multiple visual parameters, ensuring that users can enjoy a high-quality video experience even under less-than-ideal network conditions. This is a valuable advantage for video content providers because it directly improves service availability and user satisfaction.

[0049] S130. Determine a target encoding mode of the target video frame according to the second network attribute and semantic content features of the target video frame.

[0050] The second network attribute refers to the network characteristics of the target video frame and the network environment in which it is located, which may be the bandwidth level. The semantic content feature refers to the high-level semantic information of the image content reflected in the target video frame, which may include one of the content priority score and the possible highest content priority score. The target coding mode refers to the mode used to convert the video signal into digital form and effectively compress and decompress the data during storage and transmission.

[0051] Optionally, a weighted calculation is performed on the second network attribute and semantic content feature of the target video frame to obtain the coding efficiency of the target video frame under the current coding mode; if the coding efficiency does not meet the mode adjustment condition, the current coding mode is determined as the target coding mode; if the coding efficiency meets the mode adjustment condition, the coding quality of the current coding mode is adjusted to obtain the target coding mode.

[0052] The current coding mode refers to the initial coding mode of the target video frame. The coding efficiency refers to the adaptability and efficiency of the target video frame under the current coding mode. The mode adjustment condition is manually set according to the actual situation or experience value.

[0053] Exemplarily, the calculation of coding efficiency can be implemented by the following formula:

[0054]

[0055] Among them, C refers to the coding efficiency, which reflects the adaptability and efficiency of the coding strategy under specific network and content conditions. EffectiveBandwidth refers to the current actual available bandwidth, which quantifies the current data transmission capacity of the network. TargetBandwidth refers to the target bandwidth, which is the ideal bandwidth level determined according to the quality of service requirements and user settings. ContentPriority refers to the priority score of the content, which is set according to the importance of the content or the preference of the audience. MaxContentPriority refers to the highest possible content priority score, which calibrates the maximum possible value of the content priority. t and i refer to weight factors, which are used to adjust the influence of bandwidth availability and content priority in the coding strategy decision.

[0056] It should be noted that the formula is designed to dynamically adjust the video encoding strategy to adapt to the different needs of network bandwidth changes and content importance. By evaluating the ratio of actual bandwidth to target bandwidth (adjusted by t), and the ratio of content priority to the highest priority (adjusted by i), a comprehensive coding efficiency score can be determined. If the actual available bandwidth is lower than the target bandwidth, this formula encourages the use of more efficient encoding modes, such as using a higher compression ratio, to reduce bandwidth requirements. Conversely, if the actual bandwidth is higher than the target bandwidth, the encoding mode can choose a lower compression ratio to improve video quality. At the same time, if the content has a high priority, more bandwidth resources can be allocated to ensure high-quality transmission of important content.

[0057] It can be understood that dynamic coding mode switching technology can maximize the overall network efficiency and video quality while ensuring priority transmission of key information.

[0058] S140: Play the target video frame based on the target encoding mode.

[0059] In an optional implementation, viewing feedback information of viewers is collected in real time, and the viewing feedback information is analyzed to obtain user satisfaction of the current video frame; according to the user satisfaction, the current video frame is adjusted to obtain a target video frame.

[0060] The viewer refers to the user who is currently watching the target video. Feedback refers to the feedback on the viewing experience provided by the user or automatically collected through technical means when watching the target video, which may include information on the flow, cleanliness, loading speed, etc. of the video playback. User satisfaction is used to measure the overall satisfaction of users with the video service.

[0061] Exemplarily, the determination of user satisfaction can be achieved by the following formula:

[0062]

[0063] Among them, P refers to user satisfaction. Kappa is a weight factor used to adjust the impact of average experience quality. AverageQualityOfExperience refers to the average experience quality, which is a comprehensive assessment based on factors such as video quality and smoothness during user viewing. MaxQualityOfExperience refers to the highest possible experience quality, which is the best experience quality that can be achieved in theory. BufferingRatio refers to the buffering ratio during video playback. A high buffering ratio usually means that buffering often occurs during playback, affecting the viewing experience. Lambda is a weight factor used to adjust the negative impact of the buffering ratio.

[0064] It should be noted that a high average quality of experience score reflects the clarity, color accuracy, encoding efficiency, and smoothness of playback of the video content. This is usually directly related to the response speed of the server and network conditions. A high buffering ratio directly affects the viewing experience. Frequent buffering will lead to audience dissatisfaction and frustration, and may even cause user churn. This formula calculates user satisfaction by considering the average quality of video playback and the frequency of buffering events. Ideally, user satisfaction will be high when the video playback quality is close to or equal to the maximum possible quality of experience. However, if buffering is often required during video playback, user satisfaction may be seriously affected even if the video quality is high.

[0065] Understandably, through this real-time feedback mechanism, service providers can not only continuously optimize their services, but also respond quickly to problems at the early stage, greatly improving user satisfaction and loyalty. The implementation of this strategy helps build a more stable, reliable and user-friendly video streaming service environment.

[0066] The embodiment of the present application determines the target processing mode of the current video frame according to the first network attribute and local image features of the current video frame; the target processing mode includes cloud processing or edge processing; the target device corresponding to the target processing mode is controlled to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain the analysis result of the current video frame; the current video frame is processed according to the analysis result of the current video frame to obtain the target video frame; the target encoding mode of the target video frame is determined according to the second network attribute and semantic content features of the target video frame; and the target video frame is played based on the target encoding mode. The above technical scheme can improve the performance and efficiency of video streaming transmission, while ensuring that users can still enjoy high-quality video experience in a changing network environment.

[0067] Embodiment 2

[0068] Figure 2 It is a flow chart of a video optimization method provided according to the second embodiment of the present application. Based on the technical solutions of the above embodiments, this embodiment refines "determining the target processing method of the current video frame according to the first network attributes and local image features of the current video frame" into "performing weighted calculations on the first network attributes of the current video frame to obtain the network quality of the current video frame; performing weighted calculations on the local image features of the current video frame to obtain the feature quality of the current video frame; determining the target processing method of the current video frame according to the network quality and feature quality of the current video frame". It should be noted that for the parts not described in detail in the embodiments of the present application, please refer to the relevant statements of other embodiments. Figure 2 As shown, the method includes:

[0069] S210: Perform weighted calculation on each first network attribute of the current video frame to obtain the network quality of the current video frame.

[0070] The network quality refers to an indicator used to evaluate the network status of the current video frame.

[0071] S220 , performing weighted calculation on each local image feature of the current video frame to obtain feature quality of the current video frame.

[0072] The feature quality refers to an indicator used to evaluate the complexity of local image features of the current video frame.

[0073] S230: Determine a target processing method for the current video frame according to the network quality and feature quality of the current video frame.

[0074] Optionally, if the network quality and feature quality of the current video frame meet the edge processing conditions, edge processing is determined as the target processing method for the current video frame; if the network quality and feature quality meet the cloud processing conditions, cloud processing is determined as the target processing method for the current video frame; if the network quality and feature quality meet both the edge processing conditions and the cloud processing conditions, edge processing or cloud processing is determined as the target processing method for the current video frame.

[0075] In an optional implementation, based on the cloud-edge collaborative decision engine, the decision score of the current video frame can be determined according to the network quality and feature quality of the current video frame; and the target processing method of the current video frame can be determined according to the decision score.

[0076] Among them, the cloud-edge collaborative decision engine refers to the technology used to achieve collaborative decision-making and data processing between cloud and edge devices.

[0077] Exemplarily, the decision score can be determined by the following formula:

[0078]

[0079] Here, D refers to the decision score, which is used to determine whether the processing task should be performed by the cloud or the edge.

[0080] It refers to network latency, measured in milliseconds, and represents the delay in data transmission from source to destination. It refers to network throughput, which indicates the data transmission rate of the network. Used to indicate the degree of network congestion. The lower the throughput, the higher the reciprocal value, indicating that the network congestion is more serious. It refers to the packet loss rate, expressed as a percentage. A high packet loss rate usually indicates an unstable network. It refers to the content complexity score, which is given according to the complexity of the video content (such as frequent scene changes, high dynamics, etc.). It refers to the vector size of the picture motion, which measures the speed and direction of the moving object in the video. Refers to the degree of change of pixels within a frame, evaluating the fluctuation of pixel values ​​within a single frame. , and is the weight coefficient, which is adjusted according to the specific requirements of video transmission and network conditions to balance the impact of network parameters. , and is a weight coefficient, which is adjusted according to the specific requirements of video transmission and network conditions to balance the impact of content characteristics. Refers to the network quality. Refers to characteristic quality.

[0081] It can be understood that the optimal location of the video processing task is determined by calculating D in the above formula. When the network conditions are good (i.e., low latency, high throughput, and low packet loss rate), the network-related score is low, and the impact of content complexity and video dynamics is more significant. If the video content is very complex or dynamic, cloud processing is more appropriate because cloud servers usually have more powerful processing capabilities to cope with these needs. Conversely, if the network conditions are poor, the relevant score increases, and edge processing may be more advantageous at this time. Edge processing can reduce the impact of network latency and process video data quickly and responsively, especially when the network throughput is insufficient to support high-quality video transmission. Through this dynamic allocation of processing tasks, the cloud-edge collaborative decision engine can maximize the user's viewing experience while ensuring video transmission efficiency. The implementation of this technology, especially in the field of real-time video streaming, is extremely challenging and requires accurate real-time data analysis and efficient resource management strategies.

[0082] S240, controlling the target device corresponding to the target processing method to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain the analysis result of the current video frame; and processing the current video frame according to the analysis result of the current video frame to obtain the target video frame.

[0083] For example, the analysis result may be presented in the form of a score, and the specific determination method may be implemented by the following formula:

[0084]

[0085] Here, F refers to the analysis result, that is, the importance score of the frame, which determines whether the frame should be retained, discarded or transmitted with priority. It refers to the confidence of object detection, indicating the reliability of the detected object. A high confidence indicates a high probability of correct detection. It refers to the confidence of object classification, which reflects the accuracy of the classification system in determining the object category. It refers to the result of scene change detection, indicating the significance of scene changes in the video, and is the basis for judging the screen transition or the appearance of a new scene. f refers to the frame redundancy index, which is evaluated by calculating the similarity between the current frame and the previous and next frames. The higher the value, the more repeated content there is with other frames. , and

[0086] Refers to the weight coefficient, which is used to adjust the influence of each detection signal in the overall importance score.

[0087] It should be noted that the formula combines various content analysis indicators to determine the retention value of each frame. The confidence of object detection and classification is directly related to the identification of significant and key content in the frame; if a frame contains an important object with high confidence (such as a scoring moment in a sports event or a key speaker in a live news broadcast), the retention of this frame has a high priority. The results of scene change detection help identify changes in the picture or the development of the storyline, which is particularly important for maintaining the audience's continuous viewing experience. The introduction of the frame redundancy index allows the system to identify and reduce the transmission of redundant data, especially in static or repetitive scenes. By reducing the transmission of these contents, bandwidth and storage resources can be significantly saved. For example, in a long shot, if the background and main objects remain unchanged for a long time, many frames may be regarded as redundant and can be restored through interpolation technology instead of transmitting them all.

[0088] It is understandable that this technology not only optimizes the use of bandwidth, but also ensures the integrity and transmission priority of key information in teaching videos, thereby improving the experience and effect of remote learning. Through such intelligent frame analysis, video streaming services can effectively reduce data traffic and improve network utilization without sacrificing video quality.

[0089] S250: Determine a target encoding mode of the target video frame according to the second network attribute and semantic content features of the target video frame.

[0090] S260: Play the target video frame based on the target encoding mode.

[0091] The embodiment of the present application performs weighted calculation on each first network attribute of the current video frame to obtain the network quality of the current video frame; performs weighted calculation on each local image feature of the current video frame to obtain the feature quality of the current video frame; determines the target processing method of the current video frame according to the network quality and feature quality of the current video frame; the target processing method includes cloud processing or edge processing; controls the target device corresponding to the target processing method to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain the analysis result of the current video frame; processes the current video frame according to the analysis result of the current video frame to obtain the target video frame; determines the target encoding mode of the target video frame according to the second network attribute and semantic content feature of the target video frame; and plays the target video frame based on the target encoding mode. The above technical scheme can improve the performance and efficiency of video streaming transmission, while ensuring that users can still enjoy high-quality video experience in a changing network environment.

[0092] Embodiment 3

[0093] Figure 3 This is a schematic diagram of the structure of a video optimization device provided in accordance with the third embodiment of the present application, which can be applied to the case of adaptive dynamic optimization of the playback of the target video. The video optimization device can be implemented in the form of hardware and / or software, and the video optimization device can be configured in a computer device, such as a server. Figure 3 As shown, the device comprises:

[0094] The processing mode determination module 310 is used to determine a target processing mode of the current video frame according to the first network attribute and the local image feature of the current video frame; the target processing mode includes cloud processing or edge processing;

[0095] The video frame determination module 320 is used to control the target device corresponding to the target processing mode to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain the analysis result of the current video frame; and process the current video frame according to the analysis result of the current video frame to obtain the target video frame;

[0096] The coding mode determination module 330 is used to determine a target coding mode of the target video frame according to the second network attribute and the semantic content feature of the target video frame;

[0097] The video frame playing module 340 is used to play the target video frame based on the target encoding mode.

[0098] The embodiment of the present application determines the target processing mode of the current video frame according to the first network attribute and local image features of the current video frame; the target processing mode includes cloud processing or edge processing; the target device corresponding to the target processing mode is controlled to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain the analysis result of the current video frame; the current video frame is processed according to the analysis result of the current video frame to obtain the target video frame; the target encoding mode of the target video frame is determined according to the second network attribute and semantic content features of the target video frame; and the target video frame is played based on the target encoding mode. The above technical scheme can improve the performance and efficiency of video streaming transmission, while ensuring that users can still enjoy high-quality video experience in a changing network environment.

[0099] Optionally, the video frame determination module 320 includes:

[0100] A video frame replacement unit, configured to generate a candidate video frame according to motion features between a previous video frame of the current video frame and a next video frame of the current video frame, and replace the current video frame with the candidate video frame if the analysis result of the current video frame meets the frame insertion condition;

[0101] The video frame determination unit is used to adjust the visual features of the candidate video frame according to the first visual features between the candidate video frame and the previous video frame, and the second visual features between the candidate video frame and the next video frame, so as to obtain the target video frame.

[0102] Optionally, the video frame determination unit is specifically configured to:

[0103] If the first visual feature and the second visual feature satisfy the visual feature adjustment condition, the visual feature of the candidate video frame is adjusted according to the first visual feature and the second visual feature until the visual feature adjustment condition is no longer satisfied, thereby obtaining a target video frame.

[0104] Optionally, the processing mode determination module 310 includes:

[0105] A network quality determination unit, configured to perform weighted calculation on each first network attribute of the current video frame to obtain the network quality of the current video frame;

[0106] A feature quality determination unit, used for performing weighted calculation on each local image feature of the current video frame to obtain the feature quality of the current video frame;

[0107] The processing mode determination unit is used to determine a target processing mode for the current video frame according to the network quality and feature quality of the current video frame.

[0108] Optionally, a processing mode determination unit is used to:

[0109] If the network quality and feature quality of the current video frame meet the edge processing condition, edge processing is determined as the target processing method of the current video frame;

[0110] If the network quality and feature quality meet the cloud processing conditions, cloud processing is determined as the target processing method for the current video frame;

[0111] If the network quality and feature quality meet both edge processing conditions and cloud processing conditions, edge processing or cloud processing is determined as the target processing method for the current video frame.

[0112] Optionally, the coding mode determination module 330 is specifically configured to:

[0113] A weighted calculation is performed on the second network attribute and the semantic content feature of the target video frame to obtain the coding efficiency of the target video frame in the current coding mode;

[0114] If the coding efficiency does not meet the mode adjustment condition, the current coding mode is determined as the target coding mode;

[0115] If the coding efficiency meets the mode adjustment condition, the coding quality of the current coding mode is adjusted to obtain the target coding mode.

[0116] The video optimization device provided in the embodiments of the present application can execute the video optimization method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects for executing each video optimization method.

[0117] Embodiment 4

[0118] Figure 4 4 is a schematic diagram of the structure of an electronic device 410 for implementing the video optimization method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0119] like Figure 4As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In RAM413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, ROM412 and RAM413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0120] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0121] The processor 411 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 performs the various methods and processes described above, such as a video optimization method.

[0122] In some embodiments, the video optimization method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the video optimization method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured as a video optimization method in any other appropriate manner (e.g., by means of firmware).

[0123] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable video optimization device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0125] In the context of the present application, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0128] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0129] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0130] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A video optimization method, characterized in that: include: Determining a target processing method for the current video frame according to a first network attribute and a local image feature of the current video frame; the target processing method includes cloud processing or edge processing; Controlling a target device corresponding to the target processing mode to perform deep content-aware frame analysis on the confidence, scene change detection result and frame redundancy index of the current video frame to obtain an analysis result of the current video frame; and processing the current video frame according to the analysis result of the current video frame to obtain a target video frame; Determining a target encoding mode of the target video frame according to a second network attribute and a semantic content feature of the target video frame; Based on the target encoding mode, playing the target video frame; The step of processing the current video frame according to the analysis result of the current video frame to obtain a target video frame includes: If the analysis result of the current video frame meets the frame insertion condition, a candidate video frame is generated according to the motion features between a previous video frame of the current video frame and a next video frame of the current video frame, and the current video frame is replaced by the candidate video frame; According to a first visual feature between the candidate video frame and the previous video frame, and a second visual feature between the candidate video frame and the next video frame, the visual feature of the candidate video frame is adjusted to obtain a target video frame; The step of adjusting the visual features of the candidate video frame according to the first visual features between the candidate video frame and the previous video frame and the second visual features between the candidate video frame and the next video frame to obtain the target video frame includes: If the first visual feature and the second visual feature satisfy a visual feature adjustment condition, adjusting the visual feature of the candidate video frame according to the first visual feature and the second visual feature until the visual feature adjustment condition is no longer satisfied, thereby obtaining a target video frame; Wherein, determining the target processing mode of the current video frame according to the first network attribute and the local image feature of the current video frame includes: Performing weighted calculation on each first network attribute of the current video frame to obtain the network quality of the current video frame; Performing weighted calculation on each local image feature of the current video frame to obtain a feature quality of the current video frame; A target processing method for the current video frame is determined according to the network quality and feature quality of the current video frame.

2. The method according to claim 1, characterized in that Determining a target processing method for the current video frame according to the network quality and feature quality of the current video frame includes: If the network quality and feature quality of the current video frame meet the edge processing condition, edge processing is determined as the target processing mode of the current video frame; If the network quality and the feature quality meet the cloud processing condition, determining cloud processing as the target processing mode of the current video frame; If the network quality and the feature quality satisfy both the edge processing condition and the cloud processing condition, the edge processing or the cloud processing is determined as the target processing mode of the current video frame.

3. The method according to claim 1, characterized in that Determining the target encoding mode of the target video frame according to the second network attribute and the semantic content feature of the target video frame includes: Performing weighted calculation on the second network attribute and the semantic content feature of the target video frame to obtain the coding efficiency of the target video frame in the current coding mode; If the coding efficiency does not meet the mode adjustment condition, determining the current coding mode as the target coding mode; If the coding efficiency meets the mode adjustment condition, the coding quality of the current coding mode is adjusted to obtain the target coding mode.

4. A video optimization device, characterized in that: include: A processing mode determination module, used to determine a target processing mode of the current video frame according to a first network attribute and a local image feature of the current video frame; The target processing method includes cloud processing or edge processing; A video frame determination module, used to control a target device corresponding to a target processing mode to perform deep content-aware frame analysis on a confidence level, a scene change detection result, and a frame redundancy index of a current video frame to obtain an analysis result of the current video frame; and to process the current video frame according to the analysis result of the current video frame to obtain a target video frame; A coding mode determination module, used for determining a target coding mode of a target video frame according to a second network attribute and a semantic content feature of the target video frame; A video frame playing module, used for playing the target video frame based on the target encoding mode; Wherein, the video frame determination module includes: A video frame replacement unit, configured to generate a candidate video frame according to motion features between a previous video frame of the current video frame and a next video frame of the current video frame, and replace the current video frame with the candidate video frame if the analysis result of the current video frame meets the frame insertion condition; A video frame determination unit, configured to adjust the visual features of the candidate video frame according to a first visual feature between the candidate video frame and a previous video frame, and a second visual feature between the candidate video frame and a next video frame, to obtain a target video frame; Wherein, the video frame determination unit is specifically used for: If the first visual feature and the second visual feature meet the visual feature adjustment condition, the visual feature of the candidate video frame is adjusted according to the first visual feature and the second visual feature until the visual feature adjustment condition is no longer met, thereby obtaining a target video frame; Wherein, the processing mode determination module includes: A network quality determination unit, configured to perform weighted calculation on each first network attribute of the current video frame to obtain the network quality of the current video frame; A feature quality determination unit, used for performing weighted calculation on each local image feature of the current video frame to obtain the feature quality of the current video frame; The processing mode determination unit is used to determine a target processing mode for the current video frame according to the network quality and feature quality of the current video frame.

5. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video optimization method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video optimization method as described in any one of claims 1 to 3 is implemented.

7. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the video optimization method according to any one of claims 1 to 3.