Stream processing method and device under weak network condition

Abstract

The invention discloses a streaming processing method and device under a weak network condition, and the method comprises the steps: obtaining an original video image frame of a video stream in a current data processing period, and obtaining a texture feature value and a motion change amplitude value of the original video image frame based on a GPU; based on a convolutional neural network, identifying the scene type of the original video image frame through the texture feature value and the motion change amplitude value; and when the current network condition of the video stream is in a weak network condition, adjusting an initial code rate corresponding to a scene type based on the scene type of the original video image frame, sending the adjusted code rate to a video editor, and encoding the video stream in the next data processing period. The texture feature value and the motion change amplitude value of the original video image frame are acquired based on the GPU, the scene type is identified by using the convolutional neural network, the code rate adjustment value is determined according to the scene type, and a precise basis is provided for subsequent code rate adjustment.

Description

Technical Field

[0001] The present invention relates to the technical field of stream processing, and in particular to a stream processing method and device under weak network conditions. Background Art

[0002] In the entire process of video streaming processing, first of all, in the video source acquisition stage, the device's own parameter settings are usually used to collect video. For example, ordinary cameras will shoot according to their fixed resolution and frame rate, and screen recording software will simply record the content displayed on the screen, and basically will not pre-process the video content. In the encoding stage, common video encoding standards are often directly adopted, such as H.264 and H.265. After the encoding parameters are set, the subsequent network conditions or scene differences of the video content will no longer be considered. In terms of network transmission, the monitoring of network status mainly focuses on basic network indicators, such as simply checking bandwidth and packet loss rate. When it is found that the network is not good, such as insufficient bandwidth, the video bit rate is generally directly reduced. This reduction method is relatively mechanical, which may be based on a predetermined ratio or a preset simple rule, and it operates indiscriminately on the entire video stream. In terms of transmission protocols, commonly used protocols such as RTMP and HLS are mainly to ensure that the video can be transmitted from the server to the client, but in a weak network environment, they cannot optimize the transmission according to the details of the video content. Summary of the invention

[0003] The purpose of the embodiment of the present invention is to provide a streaming processing method and device under weak network conditions, by obtaining the texture feature values ​​and motion change amplitude values ​​of the original video image frame based on the GPU, and using a convolutional neural network to identify the scene type, and determining the bit rate adjustment value according to the scene type, thereby providing an accurate basis for subsequent bit rate adjustment.

[0004] To solve the above technical problem, a first aspect of an embodiment of the present invention provides a streaming processing method under a weak network condition, comprising the following steps: Acquire an original video image frame of a video stream in a current data processing cycle, and obtain a texture feature value and a motion change amplitude value of the original video image frame based on a GPU; Based on a convolutional neural network, the scene type of the original video image frame is identified by the texture feature value and the motion change amplitude value, and the scene type includes: video conferencing scene, game live broadcast scene, remote monitoring and video on demand scene; When the current network condition of the video stream is in a weak network condition, the initial bit rate corresponding to the scene type is adjusted based on the scene type of the original video image frame, and the adjusted bit rate is sent to the video editor to encode the video stream in the next data processing cycle.

[0005] Furthermore, obtaining the texture feature value of the original video image frame based on the GPU includes: Each original video image frame is divided into a plurality of pixel blocks, and a texture feature index value of each pixel block is obtained based on a gray level co-occurrence matrix, wherein the texture feature indexes include: contrast, correlation, energy and entropy; Based on the texture feature index values ​​of the plurality of pixel blocks, a texture feature value of each original video image frame is calculated.

[0006] Furthermore, the calculating of the texture feature value of each original video image frame based on the texture feature index values ​​of the plurality of pixel blocks includes: Acquire the contrast, correlation, energy and entropy of the pixel block based on the gray level co-occurrence matrix, and calculate the texture feature sub-value of the pixel block; The texture feature value of the original video image frame is calculated based on the texture feature sub-values ​​of all the pixel blocks.

[0007] Furthermore, the calculation formula of the texture feature sub-value of the pixel block is: ; in, The original video image The texture feature sub-value of the pixel block, For the The contrast of a pixel block, For the The correlation of pixel blocks, For the The energy of a pixel block, For the The entropy of a pixel block, is the contrast weight value, is the relevance weight value, is the energy weight value, is the entropy weight value.

[0008] Furthermore, the calculation formula of the weight value of the texture feature index value is: ; in, is the weight value of the kth texture feature index, and when k=1, it is the contrast weight value , k=2 is the correlation weight value , k=3 is the energy weight value , when k=1, it is the entropy weight value , is the weight priority coefficient of the kth texture feature index, is the weight influence coefficient of the kth texture feature index.

[0009] Furthermore, the calculation formula of the texture feature value of the original video image frame is: ; ; in, is the texture feature value of the original video image frame, For the The weight coefficients of pixel blocks, the original video image frame is divided into Pixel blocks, The weight distribution adjustment parameters are for the pixel blocks in the original video image frame.

[0010] Further, obtaining the motion change amplitude value of the original video image frame includes: Obtaining a motion vector of each pixel in the original video image frame; Based on the motion vector, obtaining a displacement of the pixel point in two original video image frames in a current data processing cycle and a previous data processing cycle; Constructing an optical flow field of the original video image frame according to the displacements of all the pixel points; Based on the optical flow field, calculating the average amplitude and amplitude standard deviation of motion vectors of all pixels in the original video image frame; The motion change amplitude value of the original video image frame is calculated based on the average amplitude and the amplitude standard deviation.

[0011] Furthermore, the calculation formula of the motion change amplitude value of the original video image frame is: ; ; ; in, is the amplitude of motion change, is the average amplitude, is the median of the average amplitude, is the amplitude standard deviation, is the median of the amplitude standard deviation, and are the quartiles of the mean amplitude, and is the quartile of the standard deviation of the amplitude.

[0012] Furthermore, before adjusting the initial bit rate corresponding to the scene type, the method further includes: Obtaining a real-time bandwidth value, a real-time delay value, and a real-time packet loss rate of the video stream; Calculate a real-time network comprehensive status value based on the real-time bandwidth value, the real-time delay value and the real-time packet loss rate; If the real-time network comprehensive status value is less than a preset network status threshold, it is determined that the network status is in a weak network condition.

[0013] Accordingly, a second aspect of an embodiment of the present invention provides a stream processing device under a weak network condition, comprising: An image feature acquisition module is used to acquire the original video image frame of the video stream in the current data processing cycle, and obtain the texture feature value and motion change amplitude value of the original video image frame based on the GPU; A scene type recognition module, which is used to identify the scene type of the original video image frame through the texture feature value and the motion change amplitude value based on a convolutional neural network, and the scene types include: video conferencing scene, game live broadcast scene, remote monitoring and video on demand scene; The video encoding processing module is used to adjust the initial bit rate corresponding to the scene type based on the scene type of the original video image frame when the current network condition of the video stream is in a weak network condition, and send the adjusted bit rate to the video editor to encode the video stream in the next data processing cycle.

[0014] Accordingly, a third aspect of an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned streaming processing method under weak network conditions.

[0015] Accordingly, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the streaming processing method under the above-mentioned weak network condition is implemented.

[0016] The above technical solution of the embodiment of the present invention has the following beneficial technical effects: 1. By obtaining the texture feature values ​​and motion change amplitude values ​​of the original video image frames based on the GPU and using the convolutional neural network to identify the scene type, it can accurately distinguish different types of scenes such as video conferencing scenes, game live broadcast scenes, remote monitoring and video on demand scenes, providing an accurate basis for subsequent bit rate adjustment; 2. According to different scene types and the current weak network conditions, the initial bit rate of the corresponding scene is adjusted to avoid problems such as video freeze and buffering caused by too high bit rate under weak network conditions, effectively improving the transmission efficiency and playback quality of video streams under weak network environments, and enhancing the user viewing experience; 3. It can automatically adjust the bit rate and encoding processing according to changes in network conditions, so that the video streaming system can maintain good performance in different weak network environments, enhance the system's adaptability to complex network environments, and reduce the probability of video playback failure or serious quality degradation due to network instability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a method for processing a stream under a weak network condition provided by an embodiment of the present invention; Figure 2 It is a block diagram of a stream processing device module under weak network conditions provided by an embodiment of the present invention.

[0018] Reference numerals: 1. Image feature acquisition module, 2. Scene type recognition module, 3. Video encoding processing module. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0020] Please refer to Figure 1 A first aspect of an embodiment of the present invention provides a streaming processing method under a weak network condition, comprising the following steps: Step S100, obtaining the original video image frame of the video stream in the current data processing cycle, and obtaining the texture feature value and motion change amplitude value of the original video image frame based on the GPU.

[0021] Regularly obtain the original video image frames of the video stream in the current data processing cycle. The data processing cycle can be set according to actual needs and the frequency of changes in network conditions, such as obtaining a frame every 5 seconds or every 10 seconds. After obtaining the image frame, the powerful computing power of the GPU is used to extract the feature value.

[0022] Step S200, based on a convolutional neural network, the scene type of the original video image frame is identified through texture feature values ​​and motion change amplitude values, and the scene types include: video conferencing scene, game live broadcast scene, remote monitoring and video on demand scene.

[0023] When faced with an original video image frame to be identified, the calculated texture feature values ​​and motion change amplitude values ​​are input into the trained convolutional neural network model. The model will infer based on the previously learned feature patterns and output the probability distribution of the scene type to which the image frame belongs.

[0024] Step S300, when the current network condition of the video stream is in a weak network condition, the initial bit rate corresponding to the scene type is adjusted based on the scene type of the original video image frame, and the adjusted bit rate is sent to the video editor to encode the video stream in the next data processing cycle.

[0025] The adjusted bitrate is sent to the video editor, which will encode the video stream in the next data processing cycle based on this bitrate. The encoding process will compress and optimize the video stream based on the new bitrate requirements, using appropriate encoding algorithms and parameters to meet the transmission requirements in a weak network environment, ensuring that the video stream can be transmitted as stably as possible under weak network conditions and maintain a relatively good playback effect on the client, reducing the occurrence of undesirable phenomena such as freezes and buffering.

[0026] During the encoding process, the first step is to determine the appropriate encoding algorithm based on the received adjusted bit rate. If the bit rate requirement is low, a coding algorithm with high compression efficiency but relatively low computational complexity may be preferred, such as the Baseline Profile in the H.264 encoding standard. Although it is simplified in some advanced features, it can achieve better compression effects with relatively less computing resources in a weak network environment.

[0027] After determining the encoding algorithm, key encoding parameters should be set according to the bit rate. The frame rate of the video may be appropriately reduced. For example, it can be reduced from the original 60fps to 30fps or even lower, reducing the number of video frames that need to be encoded per unit time, thereby reducing the amount of data. The resolution will also be adjusted. For example, it can be reduced from high-definition 1080p (1920×1080) to standard-definition 720p (1280×720) or lower resolution, reducing the amount of encoded data by reducing the number of pixels.

[0028] During the encoding process, the quantization parameters of the video are also fine-tuned. The quantization parameter determines the degree of trade-off in image details during encoding. A higher quantization parameter means that more image details will be discarded in exchange for a lower amount of data. For example, when the network is weak and the bit rate is low, the quantization parameter will be increased, so that the high-frequency information of the image (such as some subtle textures, edges, etc.) is compressed more, but this will also cause a certain degree of degradation in image quality.

[0029] In addition, the encoding process also takes into account the scene characteristics of the video. For video conferencing scenes with small motion changes, more optimization may be made in spatial encoding, using the similarity within the image frame for compression; while for game live broadcast scenes with large motion changes, more emphasis will be placed on temporal encoding to use the correlation between adjacent frames for compression, while optimizing the encoding of motion vectors to more accurately describe the movement of objects, while ensuring the basic smoothness of the video, and reducing the bit rate to the greatest extent possible, so that the video stream can be stably transmitted in a weak network environment and played with acceptable quality at the receiving end.

[0030] Furthermore, obtaining the texture feature value of the original video image frame based on the GPU in step S100 includes: Step S111, each original video image frame is divided into a number of pixel blocks, and a texture feature index value of each pixel block is obtained based on a gray level co-occurrence matrix, where the texture feature indexes include: contrast, correlation, energy and entropy.

[0031] To obtain texture feature values, each original video image frame may be divided into several pixel blocks, and the texture feature index value of each pixel block, such as contrast, correlation, energy and entropy, is obtained based on the gray level co-occurrence matrix, and then the texture feature value of each original video image frame is calculated based on these texture feature index values ​​of several pixel blocks. For example, a weight is assigned to each texture feature index through a specific weighted summation formula, and each texture feature index value of the pixel block is multiplied by the corresponding weight and then added to obtain a texture feature sub-value, and then the texture feature sub-values ​​of all pixel blocks are combined to calculate the texture feature value of the image frame.

[0032] The original video image frame is divided into multiple pixel blocks according to specific rules and sizes. For example, an image frame with a resolution of 1920×1080 can be divided into pixel blocks of 32×32 pixels. The purpose of this is to facilitate the analysis and calculation of the local texture features of the image and decompose the complex image into relatively simple and representative units.

[0033] For each pixel block, a grayscale co-occurrence matrix is ​​constructed based on the grayscale values ​​of its pixels. The grayscale co-occurrence matrix is ​​a two-dimensional matrix used to count the probability of occurrence of pixel pairs with different grayscale values ​​in a certain direction (such as horizontal, vertical, diagonal, etc.) and distance. For example, for an 8-bit grayscale image (grayscale value range 0 - 255), you can determine a direction (such as horizontal direction, distance 1 pixel), and then traverse the pixel pairs in the pixel block, and count the number of occurrences of pixel pairs with grayscale values ​​g and f in this direction and distance, and use it as the element value of the g-th row and f-th column in the grayscale co-occurrence matrix.

[0034] Among them, contrast reflects the degree of change of grayscale values ​​in pixel blocks. High contrast indicates that the image texture has obvious grayscale differences, such as the black and white checkerboard pattern has a high contrast; low contrast indicates that the grayscale changes are relatively gentle, such as the pure color area or the grayscale gradient area has a low contrast. Correlation measures the linear relationship between pixel pairs with different grayscale values ​​in the pixel block. When the grayscale value change of the pixel pair shows a strong linear law, the correlation is high; otherwise, the correlation is low. Energy represents the uniformity of the grayscale value distribution in the pixel block and the regularity of the texture. High energy values ​​mean that there are more repeated patterns or regular textures. For example, regular texture patterns (such as brick wall textures) will have higher energy values; while chaotic textures have lower energy values. Entropy is used to measure the randomness of the texture of the pixel block. The higher the entropy, the richer the texture changes in the pixel block and the greater the randomness. For example, the entropy value of the grass texture in the natural scene is higher; low entropy means that the texture is relatively regular, which may be a pure color area or a simple repeated pattern.

[0035] Step S112, calculating the texture feature value of each original video image frame based on the texture feature index values ​​of a plurality of pixel blocks.

[0036] For each divided pixel block, texture feature index values ​​are extracted to describe its texture characteristics. Common texture feature indexes include gray-level co-occurrence matrix (GLCM) related features, such as energy, contrast, correlation, entropy, etc. Taking the gray-level co-occurrence matrix as an example, the matrix is ​​constructed by counting the probability of occurrence of pixel grayscale pairs at a certain distance and direction in the image, and then the energy (reflecting the uniformity of the image grayscale distribution and the coarseness of the texture) and contrast (reflecting the clarity of the texture, the higher the contrast, the clearer the texture) and other index values ​​are calculated from this matrix.

[0037] By integrating the texture feature information originally scattered on each pixel block, one or more texture feature values ​​that can represent the texture characteristics of the entire original video image frame are obtained. This means that when processing and analyzing the video image in the future, there is no need to pay attention to the detailed texture of each specific pixel block, but to directly perform macro-level comparison, classification, and recognition operations based on the texture feature values ​​of the entire frame, which greatly simplifies the processing flow and is also more conducive to grasping the overall texture characteristics of the video image.

[0038] Furthermore, the step S112 of calculating the texture feature value of each original video image frame based on the texture feature index values ​​of the plurality of pixel blocks includes: Step S112a, obtaining the contrast, correlation, energy and entropy of the pixel block based on the gray level co-occurrence matrix, and calculating the texture feature sub-value of the pixel block.

[0039] Step S112b, calculating the texture feature value of the original video image frame based on the texture feature sub-values ​​of all pixel blocks.

[0040] In a specific implementation of the embodiment of the present invention, the calculation formula of the texture feature sub-value of the pixel block is: ; in, The original video image The texture feature sub-value of the pixel block, For the The contrast of a pixel block, For the The correlation of pixel blocks, For the The energy of a pixel block, For the The entropy of a pixel block, is the contrast weight value, is the relevance weight value, is the energy weight value, is the entropy weight value.

[0041] The above formula can calculate the texture feature sub-values ​​of pixel blocks more carefully by considering factors such as the importance, priority and influence of different texture feature indicators to adapt to scenes with higher requirements for texture feature analysis.

[0042] The first term in the above formula , contrast reflects the degree of change and difference of grayscale values ​​within a pixel block. The weight reflects the relative importance of contrast in the overall texture feature evaluation. In an image, high contrast areas often correspond to the edges, outlines, or significant texture changes of objects. Calculate the quantitative results of the difference in grayscale values ​​within a pixel block based on methods such as grayscale co-occurrence matrix. For example, for a pixel block with obvious black and white stripes, its contrast value will be high, indicating that the texture of the pixel block has strong grayscale changes, which may imply specific texture structures or object features.

[0043] Item 2 In the above equation, the correlation measures the linear relationship between pairs of pixels with different gray values ​​in a pixel block. Determined by the AHP method, it indicates the importance of correlation in the overall texture feature consideration. In images with regular textures, there is usually a strong correlation between pixels, which reflects the directionality and repetitiveness of the texture. If the application scenario is the classification or detection of such regular textures, May be relatively large. The value that reflects the degree of linear correlation between pixels is calculated by analyzing the gray level co-occurrence matrix of the pixel block. For example, for a pixel block with horizontal line texture, the gray value of the pixel in the horizontal direction changes more regularly, and the correlation value will be higher, indicating that this texture has a strong linear dependence in the horizontal direction. Correction factor Set the contrast and entropy Combined with, entropy measures the randomness of texture. When entropy is high, it means that the texture changes are more complex and random, which may weaken the contribution of correlation to texture features. , when the entropy is large (texture randomness is strong), the proportion of the correlation term in the texture feature sub-value will be reduced, because the regular texture information reflected by the correlation may be disturbed by randomness. When the contrast is high, it means that the texture may have a more obvious structure, and the correlation information may be more valuable. The factor will be relatively large, making the correlation term more effective in the calculation.

[0044] Item 3 middle, We have determined the importance of energy in the evaluation of texture features. In some image compression or image quality assessment applications, energy information is critical. For example, for images with a lot of repeated patterns or uniform textures, such as a blue sky background or a solid-colored wall, the energy value is high. If the task is to compress such images, may be large because its regularity can be exploited for more efficient compression. Energy value Calculated based on the gray-level co-occurrence matrix, it reflects the concentration and regularity of the gray value distribution within the pixel block. A high energy value means that there are more similar gray value combinations within the pixel block, and the texture is more regular. For example, in an image of a wall composed of bricks of the same color, the energy value of the pixel block will be higher, indicating that its texture has strong uniformity and repeatability. Correction factor The relationship between contrast and energy is considered. When the contrast is high, it means that there may be obvious changes or structures in the texture. At this time, the uniformity represented by energy may be relatively less prominent, so it is divided by The influence of the energy term on the texture feature subvalues ​​is adjusted to some extent based on contrast. If the contrast is very low, the energy term may be more representative of the texture, and this factor will be relatively large, making the energy term more influential in the calculation.

[0045] Item 4 In the example, entropy measures the randomness and uncertainty of the texture. This shows the importance of entropy in the overall texture feature judgment. In natural scene images, such as forests and mountains, textures often have high randomness and large entropy values. If the application is the classification or feature extraction of natural scene images, It may be assigned a larger value because entropy can well reflect the complex and changing texture characteristics in such scenes. It is calculated by the gray-level co-occurrence matrix of the pixel block and is used to quantify the randomness of the texture. For example, in a grass image, due to the influence of factors such as the shape, growth direction and lighting of the grass leaves, the entropy value of the pixel block will be higher, indicating that its texture has rich randomness and uncertainty. The relationship between contrast, correlation, entropy and energy is comprehensively considered. When the contrast and correlation are high, it means that the texture may have a certain structure and directionality. At this time, the randomness represented by entropy is relatively weak, and this factor will be relatively small, reducing the role of the entropy term in the texture feature sub-value. On the contrary, if the contrast and correlation are low, the texture may be more random, the influence of the entropy term will be relatively increased, and the factor will be relatively large, so that the entropy term can better reflect the random characteristics of the texture in the calculation. In general, this factor dynamically adjusts the entropy term according to the situation of other texture feature indicators, so as to more reasonably integrate the entropy information into the calculation of the texture feature sub-value.

[0046] Furthermore, the calculation formula of the weight value of the texture feature index value is: ; in, is the weight value of the kth texture feature index, and when k=1, it is the contrast weight value , k=2 is the correlation weight value , k=3 is the energy weight value , when k=1, it is the entropy weight value , is the weight priority coefficient of the kth texture feature index, is the weight influence coefficient of the kth texture feature index.

[0047] Optionally, the calculation formula of the texture feature value of the original video image frame is: ; ; in, is the texture feature value of the original video image frame, For the The weight coefficients of pixel blocks, the original video image frame is divided into Pixel blocks, The weight distribution adjustment parameters are for the pixel blocks in the original video image frame.

[0048] In the above formula, is a fixed scaling factor associated with the standard form of the two-dimensional Gaussian distribution, ensuring that the entire distribution mathematically satisfies the normalization requirements of the probability density function, that is, ensuring that the integral of the function over the entire two-dimensional plane (calculating the sum of the weights of all positions) is 1. is the variance, It plays a role in controlling the scaling of weight distribution. The smaller, The larger the value, the more dramatic the change in the spatial weight will be. The weight in the central area will be more concentrated, and the weight will drop sharply when it is slightly farther away from the center. The bigger, The smaller it is, the smoother the transition of weight from the center to the edge will be, the advantage of the weight in the central area will not be so obvious, and the edge area can also obtain a relatively considerable weight value.

[0049] In addition, the formula It is an exponential function that is used to determine the weight change of the pixel block relative to the center of the image based on its position in the image, so that the weight shows a distribution characteristic that is centered on the image and gradually decreases to the surrounding areas. and are the horizontal and vertical coordinates of the center of the pixel block, and They correspond to half of the image width and height, which is the coordinate position of the image center. What is calculated is the square of the distance between the center of the pixel block and the center of the image on the two-dimensional plane. This design makes the value of this term larger for pixel blocks farther away from the center of the image, and in the exponential function, as the value of this term increases, the value of the entire exponential function will decrease, and thus the corresponding weight will also decrease. For example, for a pixel block located exactly in the center of the image, , then the exponent part is , the weight can reach a relative maximum value; while for pixel blocks far from the center, the value of this item will be significantly greater than 0, resulting in the value of the exponential part being less than 1, and the weight is reduced accordingly. Related to the above, it further controls the rate at which the weight changes with the distance from the center of the image. After confirmation, is a fixed value, and the square of the distance work together to determine the decay rate of the exponential function. For example, a small (That is, the larger ) will cause the weight to decay rapidly with increasing distance, strengthening the weight advantage of the central area; while larger (Smaller ) makes the weight decay more slowly, making the weight distribution at different positions of the image more uniform and smooth.

[0050] In another specific implementation of the embodiment of the present invention, obtaining the motion change amplitude value of the original video image frame in step S100 includes: Step S121, obtaining the motion vector of each pixel in the original video image frame.

[0051] A motion vector is a vector that describes the direction and size of a pixel's motion in a video sequence. In video processing, motion vectors are usually obtained through a pixel matching algorithm between adjacent frames.

[0052] Step S122, based on the motion vector, obtaining the displacement of a pixel point in two original video image frames in the current data processing cycle and the previous data processing cycle.

[0053] Step S123: constructing an optical flow field of the original video image frame according to the displacement of all pixels.

[0054] Step S124, based on the optical flow field, calculate the average amplitude and amplitude standard deviation of the motion vectors of all pixels in the original video image frame.

[0055] Step S125, calculating the motion change amplitude value of the original video image frame based on the average amplitude and the amplitude standard deviation.

[0056] The acquisition of the motion change amplitude value is based on calculating the pixel motion vector of each original video image frame. By comparing two adjacent original video image frames, the displacement of each pixel between the two adjacent frames is determined, and then the optical flow field of the original video image frame is constructed. The vector size and direction in the optical flow field can reflect the speed and direction of the object in the image, that is, it reflects the motion change amplitude of the image. On this basis, the average amplitude of all pixel motion vectors and the standard deviation of the amplitude are further calculated as the motion change amplitude value to measure the intensity of the movement.

[0057] When it is determined that the network is weak, the corresponding initial bit rate is adjusted according to the scene type of the original video image frame previously identified. Different scene types have different requirements and tolerances for bit rates. For example, video conferencing scenes may pay more attention to the clarity and smoothness of facial expressions and movements of characters. When the network is weak, the bit rate of some background areas can be appropriately reduced, but the image quality of the main body of the characters must be guaranteed; game live broadcast scenes need to be adjusted according to the type of game and the dynamic degree of the picture. If it is a competitive game and the picture moves violently, the bit rate of the key action area may be prioritized when the network is weak, and the image quality of non-key areas may be appropriately reduced; remote monitoring scenes may pay more attention to the key information of the monitoring target when the network is weak, such as the outline of the person or object, the position change, etc., and adjust the bit rate accordingly; video on demand scenes can be based on the user's playback settings and video content. For example, the picture quality of plot videos can be reduced to ensure playback smoothness when the network is weak, while action videos balance the picture quality and smoothness to a certain extent.

[0058] Furthermore, the calculation formula of the motion change amplitude value of the original video image frame is: ; ; ; in, is the amplitude of motion change, is the average amplitude, is the median of the average amplitude, is the amplitude standard deviation, is the median of the amplitude standard deviation, and are the quartiles of the mean amplitude, and is the quartile of the standard deviation of the amplitude.

[0059] It is the lower quartile of the average amplitude set, which means that the average amplitude of about 25% of the data points is smaller than this value. The data is divided into the lower part, which reflects the position of the lower end of the average amplitude distribution and can be used to measure the central tendency of the data at the low end. is the upper quartile of the average amplitude set. About 75% of the data points have an average amplitude less than this value, which represents the higher end of the average amplitude distribution and helps to understand the concentration trend of the data at the high end. For the average amplitude, these two quartiles can show the degree of dispersion of the data and the distribution characteristics of different positions. For example, when analyzing the motion of image frames, the average amplitude in different scenes and It can reflect the approximate range of differences in exercise intensity in the overall sample. and They are the lower quartile and upper quartile of the amplitude standard deviation set. From the perspective of the amplitude standard deviation, they describe the distribution of the amplitude standard deviation at different positions, that is, they reflect the distribution characteristics of the uniformity of the motion change amplitude in the sample. For example, in different video contents, the approximate range of motion uniformity can be reflected by these two quartiles.

[0060] It is the interquartile range of the average amplitude, which represents the distribution range of the middle 50% of the average amplitude data, that is, the distribution span after excluding the 25% extreme values ​​at both ends. This indicator can be used to measure the degree of dispersion of the average amplitude data, and compared with directly using the full range (maximum value minus minimum value), it is less susceptible to the influence of extreme values ​​and can more robustly reflect the range of data changes. For example, when comparing the stability of motion intensity of different types of video image frames, It can reflect the fluctuation of the average amplitude in the mainstream part (the middle 50%). It is the interquartile range of the amplitude standard deviation, which also reflects the distribution range of the middle 50% of the amplitude standard deviation data. It is used to measure the dispersion of the amplitude standard deviation, that is, the fluctuation range of the uniformity of the motion change amplitude. When analyzing the stability of motion uniformity in different video scenes, It is an important reference indicator that helps us understand the uniformity range of motion change amplitude in most cases. In the subsequent formula for calculating the motion change amplitude value, and As the denominator, it plays a role of standardization, so that the deviation of the mean amplitude and the amplitude standard deviation from their respective medians can be measured on a relatively uniform scale, avoiding the situation where an item has an unreasonable proportion in the comprehensive evaluation due to different magnitudes or distribution ranges of the data themselves, thereby more fairly and accurately considering the contribution of the two to the motion change amplitude index.

[0061] It is the median of the average amplitude set. It divides the average amplitude data into two. Half of the data points have an average amplitude less than this value, and the other half have an average amplitude greater than this value, representing the middle position level of the average amplitude data. When analyzing the motion intensity of image frames, the median can reflect the motion intensity at an intermediate level. It is not as easily affected by extremely large or small average amplitudes as the mean, and can better reflect the central trend of the data. For example, in some videos that contain occasional high-intensity motion images and a large number of low-intensity motion images, the median can more robustly show the motion intensity under normal circumstances. It is the median of the amplitude standard deviation set, which reflects the middle level of the amplitude standard deviation data, that is, it represents the middle state of the uniformity of the motion change amplitude. It also has the characteristics of robustness, and is not excessively disturbed by extreme uniformity or extreme non-uniformity. It can accurately reflect the uniformity of the motion change amplitude in most cases. For example, in a video with a mixture of motion scenes of different complexity, it can grasp the general level of overall motion uniformity. In the formula, and As a reference point, the average amplitude to be evaluated and amplitude standard deviation By comparing, the relative position of the current data in the entire distribution is measured by calculating the relative deviation, and then participating in the construction of the motion change amplitude index value, so that the index value can comprehensively consider the difference with the middle level of the overall data and more comprehensively reflect the comprehensive characteristics of the motion change.

[0062] In summary, The current average amplitude is calculated Relative to the median of the mean amplitude The degree of deviation is divided by the interquartile range of the mean amplitude The standardized deviation can intuitively reflect the relative position of the current average amplitude in the entire average amplitude distribution. A positive value indicates that it is higher than the median level, and the larger the value, the higher the relative deviation, that is, the stronger the exercise intensity; a negative value indicates that it is lower than the median level, and the larger the absolute value, the weaker the exercise intensity. It depicts the difference between the current average amplitude and the overall average level in a relatively standardized way.

[0063] Similarly, is the current amplitude standard deviation Relative to the median of the amplitude standard deviation The standardized result of the deviation degree reflects the relative position of the uniformity of the current movement change amplitude in the entire amplitude standard deviation distribution. A positive value indicates that it is more uneven (or more dispersed) than the middle level, and a negative value indicates that it is more uniform. The absolute value reflects the degree of difference in relative uniformity, which helps to accurately measure the difference in uniformity between the current movement change amplitude and the overall sample.

[0064] In addition, before adjusting the initial bit rate corresponding to the scene type in step S300, the method further includes: Step S301, obtaining the real-time bandwidth value, real-time delay value and real-time packet loss rate of the video stream.

[0065] Comprehensively acquiring the above three parameters is equivalent to comprehensively acquiring the real-time status of the network from different angles, laying the foundation for the subsequent accurate assessment of the comprehensive status of the network and avoiding the one-sided problem caused by relying on only a single parameter to judge the network situation.

[0066] Step S302, calculating a real-time network comprehensive status value based on the real-time bandwidth value, the real-time delay value and the real-time packet loss rate.

[0067] Different network parameters have different degrees of influence on network conditions. By combining these three parameters into a comprehensive status value through reasonable calculation methods (for example, different weights can be assigned to each parameter based on its importance in actually affecting the network experience, and then the weighted sum can be calculated, or a specific comprehensive evaluation model can be used, etc.), the current state of the network can be presented more intuitively and concisely. This comprehensive status value eliminates the ambiguity of judgment that may occur when considering each parameter separately, so that the network status has a unified and comparable measurement standard. For example, looking at the bandwidth value alone may make you feel that the network is okay, but combined with the higher delay value and packet loss rate, the comprehensive status value can accurately reflect the actual situation that the network is not ideal, thereby providing a more accurate basis for subsequent decisions (such as whether to determine it as a weak network and the corresponding bit rate adjustment operations), which helps to achieve a more refined network adaptive video transmission strategy.

[0068] Step S303: If the real-time comprehensive network status value is less than the preset network status threshold, it is determined that the network status is in a weak network condition.

[0069] Setting a preset network status threshold is a measurement boundary determined based on a large amount of network usage experience and video transmission quality requirements. When the real-time network comprehensive status value is less than this threshold, the network status is judged to be in a weak network condition, which means that the current network environment is already difficult to support the normal transmission of the video according to the original initial bit rate and other parameters. This judgment can trigger the subsequent corresponding response mechanism in a timely manner, such as reducing the video bit rate to reduce the demand for network bandwidth, so as to ensure the continuity of the video and the basic viewing experience as much as possible in a weak network environment, and avoid long-term freezes and inability to play due to poor network. The situation seriously affects the user experience. At the same time, this clear judgment method also facilitates the automated video transmission system to respond quickly, automatically adjust the video transmission strategy according to the real-time changes of the network, improve the adaptability and robustness of the entire video stream transmission under different network conditions, and ensure that the video service can run stably in a variety of network environments.

[0070] Accordingly, please refer to Figure 2 A second aspect of an embodiment of the present invention provides a streaming processing device under a weak network condition, comprising: Image feature acquisition module 1, which is used to obtain the original video image frame of the video stream in the current data processing cycle, and obtain the texture feature value and motion change amplitude value of the original video image frame based on the GPU; The scene type recognition module 2 is used to recognize the scene type of the original video image frame through texture feature values ​​and motion change amplitude values ​​based on a convolutional neural network. The scene types include: video conferencing scene, game live broadcast scene, remote monitoring and video on demand scene; The video encoding processing module 3 is used to adjust the initial bit rate corresponding to the scene type based on the scene type of the original video image frame when the current network condition of the video stream is in a weak network condition, and send the adjusted bit rate to the video editor to encode the video stream in the next data processing cycle.

[0071] Accordingly, a third aspect of an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned streaming processing method under weak network conditions.

[0072] Accordingly, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the streaming processing method under the above-mentioned weak network condition is implemented.

[0073] The embodiment of the present invention aims to protect a streaming processing method and device under weak network conditions, wherein the method includes: obtaining the original video image frame of the video stream in the current data processing cycle, and obtaining the texture feature value and motion change amplitude value of the original video image frame based on the GPU; based on the convolutional neural network, identifying the scene type of the original video image frame through the texture feature value and the motion change amplitude value, and the scene types include: video conferencing scene, game live broadcast scene, remote monitoring and video on demand scene; when the current network condition of the video stream is in a weak network condition, based on the scene type of the original video image frame, adjusting the initial bit rate corresponding to the scene type, and sending the adjusted bit rate to the video editor to encode the video stream in the next data processing cycle. The above technical solution has the following effects: 1. By obtaining the texture feature values ​​and motion change amplitude values ​​of the original video image frames based on the GPU and using the convolutional neural network to identify the scene type, it can accurately distinguish different types of scenes such as video conferencing scenes, game live broadcast scenes, remote monitoring and video on demand scenes, providing an accurate basis for subsequent bit rate adjustment; 2. According to different scene types and the current weak network conditions, the initial bit rate of the corresponding scene is adjusted to avoid problems such as video freeze and buffering caused by too high bit rate under weak network conditions, effectively improving the transmission efficiency and playback quality of video streams under weak network environments, and enhancing the user viewing experience; 3. It can automatically adjust the bit rate and encoding processing according to changes in network conditions, so that the video streaming system can maintain good performance in different weak network environments, enhance the system's adaptability to complex network environments, and reduce the probability of video playback failure or serious quality degradation due to network instability.

[0074] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0075] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A streaming processing method under weak network conditions, characterized in that: The steps include: Acquire an original video image frame of a video stream in a current data processing cycle, and obtain a texture feature value and a motion change amplitude value of the original video image frame based on a GPU; Based on a convolutional neural network, the scene type of the original video image frame is identified by the texture feature value and the motion change amplitude value, and the scene type includes: video conferencing scene, game live broadcast scene, remote monitoring and video on demand scene; When the current network condition of the video stream is in a weak network condition, the initial bit rate corresponding to the scene type is adjusted based on the scene type of the original video image frame, and the adjusted bit rate is sent to the video editor to encode the video stream in the next data processing cycle.

2. The method for processing streaming under weak network conditions according to claim 1, characterized in that: The obtaining the texture feature value of the original video image frame based on the GPU includes: Each original video image frame is divided into a plurality of pixel blocks, and a texture feature index value of each pixel block is obtained based on a gray level co-occurrence matrix, wherein the texture feature indexes include: contrast, correlation, energy and entropy; Based on the texture feature index values ​​of the plurality of pixel blocks, a texture feature value of each original video image frame is calculated.

3. The method for processing streaming under weak network conditions according to claim 2, characterized in that: The step of calculating the texture feature value of each original video image frame based on the texture feature index values ​​of the plurality of pixel blocks comprises: Acquire the contrast, correlation, energy and entropy of the pixel block based on the gray level co-occurrence matrix, and calculate the texture feature sub-value of the pixel block; The texture feature value of the original video image frame is calculated based on the texture feature sub-values ​​of all the pixel blocks.

4. The method for processing streaming under weak network conditions according to claim 3, characterized in that: The calculation formula of the texture feature sub-value of the pixel block is: ； in, The original video image The texture feature sub-value of the pixel block, For the The contrast of a pixel block, For the The correlation of pixel blocks, For the The energy of a pixel block, For the The entropy of a pixel block, is the contrast weight value, is the relevance weight value, is the energy weight value, is the entropy weight value.

5. The method for processing streaming under weak network conditions according to claim 4, characterized in that: The calculation formula of the weight value of the texture feature index value is: ； in, is the weight value of the kth texture feature index, and when k=1, it is the contrast weight value , k=2 is the correlation weight value , k=3 is the energy weight value , when k=1, it is the entropy weight value , is the weight priority coefficient of the kth texture feature index, is the weight influence coefficient of the kth texture feature index.

6. The method for processing streaming under weak network conditions according to claim 2, characterized in that: The calculation formula of the texture feature value of the original video image frame is: ； ； in, is the texture feature value of the original video image frame, For the The weight coefficients of pixel blocks, the original video image frame is divided into Pixel blocks, The weight distribution adjustment parameters are for the pixel blocks in the original video image frame.

7. The method for processing streaming under weak network conditions according to claim 1, characterized in that: The obtaining of the motion change amplitude value of the original video image frame includes: Obtaining a motion vector of each pixel in the original video image frame; Based on the motion vector, obtaining a displacement of the pixel point in two original video image frames in a current data processing cycle and a previous data processing cycle; Constructing an optical flow field of the original video image frame according to the displacements of all the pixel points; Based on the optical flow field, calculating the average amplitude and amplitude standard deviation of motion vectors of all pixels in the original video image frame; The motion change amplitude value of the original video image frame is calculated based on the average amplitude and the amplitude standard deviation.

8. The method for processing streaming under weak network conditions according to claim 7, characterized in that: The calculation formula of the motion change amplitude value of the original video image frame is: ； ； ； in, is the amplitude of motion change, is the average amplitude, is the median of the average amplitude, is the amplitude standard deviation, is the median of the amplitude standard deviation, and are the quartiles of the mean amplitude, and is the quartile of the standard deviation of the amplitude.

9. The method for processing streams under weak network conditions according to any one of claims 1 to 8, characterized in that: Before adjusting the initial bit rate corresponding to the scene type, the method further includes: Obtaining a real-time bandwidth value, a real-time delay value, and a real-time packet loss rate of the video stream; Calculate a real-time network comprehensive status value based on the real-time bandwidth value, the real-time delay value and the real-time packet loss rate; If the real-time network comprehensive status value is less than a preset network status threshold, it is determined that the network status is in a weak network condition.

10. A streaming processing device under weak network conditions, characterized in that: include: An image feature acquisition module is used to acquire the original video image frame of the video stream in the current data processing cycle, and obtain the texture feature value and motion change amplitude value of the original video image frame based on the GPU; A scene type recognition module, which is used to identify the scene type of the original video image frame through the texture feature value and the motion change amplitude value based on a convolutional neural network, and the scene types include: video conferencing scene, game live broadcast scene, remote monitoring and video on demand scene; The video encoding processing module is used to adjust the initial bit rate corresponding to the scene type based on the scene type of the original video image frame when the current network condition of the video stream is in a weak network condition, and send the adjusted bit rate to the video editor to encode the video stream in the next data processing cycle.

Images