Video stream transmission rate adjusting method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video stream transmission rate adjusting method and device, electronic equipment and a storage medium, and belongs to the technical field of live broadcast and the technical field of digital medical treatment. The method comprises the following steps: acquiring a live video stream and a preset initial transmission rate; performing framing processing on the live video stream to obtain first target frame block data; performing content similarity calculation on the first target frame block data to obtain content similar data; determining a first playing rate of the first target frame block data according to the content similar data; obtaining a first cache occupancy rate of the client; and adjusting the initial transmission rate according to the first cache occupancy rate and the first playing rate to obtain a first target transmission rate. According to the embodiment of the invention, the influence on the video content can be reduced on the basis of adjusting the transmission rate of the video stream.

Description

Technical Field

[0001] The present application relates to the field of live broadcast and digital medical technology, and in particular to a method and device for adjusting a video stream transmission rate, an electronic device, and a storage medium. Background Art

[0002] At present, in digital medical scenarios, live broadcast technology can be used to achieve functions such as health management and remote consultation. In financial scenarios, live broadcast technology can be used to achieve online consultation, product introduction and other functions. In related technologies, in order to enable the client to support more live video streams, the transmission rate of the video stream will be adjusted according to the network conditions. However, the above adjustment method will affect the video content, such as the video content being not smooth and stuck. Therefore, how to provide a method for adjusting the video stream transmission rate to reduce the impact on the video content has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method and device for adjusting the transmission rate of a video stream, an electronic device and a storage medium, aiming to reduce the impact on the video content on the basis of adjusting the transmission rate of the video stream.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for adjusting a video stream transmission rate, which is applied to a server. The method includes:

[0005] Acquire a live video stream and a preset initial transmission rate; wherein the initial transmission rate represents the rate at which the server transmits the live video stream to a preset client;

[0006] Performing frame processing on the live video stream to obtain first target frame block data;

[0007] Calculating content similarity on the first target frame block data to obtain content similarity data;

[0008] Determine a first playback rate of the first target frame block data according to the content similarity data; wherein the first playback rate represents a rate at which the client plays the first target frame block data;

[0009] Obtaining a first cache occupancy rate of the client;

[0010] The initial transmission rate is adjusted according to the first cache occupancy rate and the first playback rate to obtain a first target transmission rate; wherein the first target transmission rate represents the rate at which the server transmits the first target frame block data to the client.

[0011] In some embodiments, adjusting the initial transmission rate according to the first buffer occupancy rate and the first playback rate to obtain a first target transmission rate includes:

[0012] If the first buffer occupancy rate is less than or equal to a preset first occupancy rate, adjusting the initial transmission rate to be greater than the first playback rate to obtain the first target transmission rate;

[0013] If the first buffer occupancy rate is greater than the first occupancy rate and less than a preset second occupancy rate, adjusting the initial transmission rate to be equal to the first playback rate; wherein the second occupancy rate is greater than the first occupancy rate, obtaining the first target transmission rate;

[0014] If the first cache occupancy rate is greater than or equal to the second occupancy rate, the initial transmission rate is adjusted to be less than the first playback rate to obtain the first target transmission rate.

[0015] In some embodiments, the live video stream includes an original image, and the step of performing frame processing on the live video stream to obtain first target frame block data includes:

[0016] Acquire two original images of adjacent frames to obtain an original image pair;

[0017] Performing frame difference calculation on the original image pair to obtain frame change data;

[0018] If the frame change data is less than or equal to a preset threshold, the live video stream is frame-processed according to the original image to obtain the first target frame block data.

[0019] In some embodiments, performing frame difference calculation on the original image pair to obtain frame change data includes:

[0020] Performing grayscale conversion on the original image pair to obtain a grayscale image pair;

[0021] Grayscale difference calculation is performed on the grayscale image pair to obtain the frame change data.

[0022] In some embodiments, the first target frame block data includes a target image, and the performing content similarity calculation on the first target frame block data to obtain content similarity data includes:

[0023] Acquire two target images of adjacent frames to obtain a target image pair;

[0024] Performing brightness similarity calculation on the target image pair to obtain brightness similarity data;

[0025] Performing contrast similarity calculation on the target image pair to obtain contrast similarity data;

[0026] Performing structural similarity calculation on the target image pair to obtain structural similarity data;

[0027] The content similarity data is obtained according to the brightness similarity data, the contrast similarity data, and the structure similarity data.

[0028] In some embodiments, the method further comprises:

[0029] transmitting the first target frame block data to the client according to the first target transmission rate;

[0030] Obtaining a second cache occupancy rate after the client receives the first target frame block data;

[0031] Performing frame processing on the live video stream to obtain second target frame block data;

[0032] The first target transmission rate is adjusted according to the second cache occupancy and the second target frame block data to obtain a second target transmission rate; wherein the second target transmission rate represents the rate at which the server transmits the second target frame block data to the client.

[0033] In some embodiments, determining the first playback rate of the first target frame block data according to the content similarity data includes:

[0034] A mapping relationship is established according to a preset content similarity range and a preset playback rate range;

[0035] The content similarity data is subjected to play rate mapping processing according to the mapping relationship to obtain the first play rate.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a device for adjusting a video stream transmission rate, the device being applied to a server, and the device comprising:

[0037] A data acquisition module, used to acquire a live video stream and a preset initial transmission rate; wherein the initial transmission rate represents a rate at which the server transmits the live video stream to a preset client;

[0038] A frame processing module, used for performing frame processing on the live video stream to obtain first target frame block data;

[0039] A content similarity calculation module, used for performing content similarity calculation on the first target frame block data to obtain content similarity data;

[0040] A playback rate determination module, configured to determine a first playback rate of the first target frame block data according to the content similarity data; wherein the first playback rate represents a rate at which the client plays the first target frame block data;

[0041] A cache occupancy acquisition module, used to acquire a first cache occupancy of the client;

[0042] A transmission rate adjustment module is used to adjust the initial transmission rate according to the first cache occupancy rate and the first playback rate to obtain a first target transmission rate; wherein the first target transmission rate represents the rate at which the server transmits the first target frame block data to the client.

[0043] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0045] The video stream transmission rate adjustment method and device, electronic device and storage medium proposed in the present application determine the first playback rate through the content similarity data of the first target frame block data, and adjust the initial transmission rate according to the first playback rate and the first cache occupancy rate to obtain the first target transmission rate. It can be seen that the adjustment process of the first target transmission rate takes into account the content similarity data of the first target frame block. Therefore, the embodiment of the present application can reduce the impact on the content of the first target frame block on the basis of realizing the transmission rate adjustment. When the embodiment of the present application is applied to the remote consultation scenario in digital medicine, the consultation efficiency and quality can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a method for adjusting a video stream transmission rate provided in an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of a video stream transmission communication link according to an embodiment of the present application;

[0048] Figure 3 This is a schematic diagram of a live video stream frame processing performed in an embodiment of the present application;

[0049] Figure 4A yes Figure 1 Flow chart of step S102 in FIG.

[0050] Figure 4B is another schematic diagram of performing frame processing on a live video stream according to an embodiment of the present application;

[0051] Figure 5 yes Figure 4A Flow chart of step S402 in FIG.

[0052] Figure 6 yes Figure 1 Flow chart of step S103 in FIG.

[0053] Figure 7 yes Figure 1 Flow chart of step S104 in FIG.

[0054] Figure 8 yes Figure 1 Flow chart of step S106 in FIG.

[0055] Fig.9A is another flow chart of the method for adjusting the video stream transmission rate provided in an embodiment of the present application;

[0056] Fig. 9B is a schematic diagram of the first target frame block data of an embodiment of the present application;

[0057] Fig.10 It is a structural schematic diagram of a device for adjusting a video stream transmission rate provided in an embodiment of the present application;

[0058] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] At present, in digital medical scenarios, live broadcast technology can be used to achieve functions such as health management and remote consultation. In financial scenarios, live broadcast technology can be used to achieve online consultation, product introduction and other functions. In related technologies, in order to enable the client to support more live video streams, the transmission rate of the video stream will be adjusted according to the network conditions. However, the above adjustment method will affect the video content, such as the video content being not smooth and stuck. Therefore, how to provide a method for adjusting the video stream transmission rate to reduce the impact on the video content has become a technical problem that needs to be solved urgently.

[0065] Based on this, the embodiments of the present application provide a method and device for adjusting the transmission rate of a video stream, an electronic device, and a storage medium, aiming to reduce the impact of transmission rate adjustment on video content.

[0066] The recommended method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the method for adjusting the video stream transmission rate in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] The method for adjusting the video stream transmission rate provided in the embodiment of the present application relates to the field of artificial intelligence technology. The method for adjusting the video stream transmission rate provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for adjusting the video stream transmission rate, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0072] Reference Figure 1 , an embodiment of the present application provides a method for adjusting the video stream transmission rate, which is applied to a server. Figure 1 The method may include but is not limited to steps S101 to S106.

[0073] Step S101, obtaining a live video stream and a preset initial transmission rate; wherein the initial transmission rate indicates the rate at which the server transmits the live video stream to a preset client;

[0074] Step S102, performing frame processing on the live video stream to obtain first target frame block data;

[0075] Step S103, calculating the content similarity of the first target frame block data to obtain content similarity data;

[0076] Step S104, determining a first playback rate of the first target frame block data according to the content similarity data; wherein the first playback rate represents a rate at which the client plays the first target frame block data;

[0077] Step S105, obtaining the first cache occupancy rate of the client;

[0078] Step S106, adjusting the initial transmission rate according to the first buffer occupancy rate and the first playback rate to obtain a first target transmission rate; wherein the first target transmission rate represents the rate at which the server transmits the first target frame block data to the client.

[0079] It should be noted that, refer to Figure 2In some embodiments, the video stream transmission communication link includes a camera device, a remote video server, an edge server and a client. Among them, the camera device is used to generate a live video stream. For example, when the embodiment of the present application is applied to large-scale scenes such as conference live broadcast and product introduction, the camera device can be an independent device with a video stream transmission function; when the present application is applied to small-scale scenes such as remote consultation and online consultation, the camera device can be a camera module loaded on the terminal, that is, the camera device only has a recording function, and does not have a video stream transmission function. The transmission of the video stream depends on the loaded terminal. The remote video server is used to store the live video stream generated by multiple cameras. When the client needs to watch the live video stream, in response to the operation triggered by the object on the client, the edge server corresponding to the client obtains the live video stream from the remote video server. The triggered operation can be the operation of the object selecting a live explanation of a related product in the insurance APP, or the operation of accepting a remote consultation, etc., which is not specifically limited in the embodiment of the present application. The edge server caches the live video stream and sends the live video stream to the client according to the preset transmission rate, thereby realizing the transmission of the video stream. The transmission rate refers to the rate at which the edge server transmits the video stream to the client, usually expressed as a bit rate. The higher the transmission rate, the faster the video stream is transmitted, but this also requires support from higher bandwidth and network speed. Therefore, the relevant technology will adjust the transmission rate according to the network conditions. However, the method of adjusting the transmission rate in the relevant technology does not take into account the content of the video stream, so it is easy to cause the video stream content to be jerky and stuck.

[0080] Based on this, the embodiment of the present application provides a method for adjusting the transmission rate of a video stream, which can adjust the transmission rate in combination with the content of the video stream and the playback rate of the video stream, thereby reducing the impact on the content of the video stream on the basis of adjusting the transmission rate. The following is an explanation of the method for adjusting the transmission rate of a video stream provided by the embodiment of the present application in combination with steps S101 to S106. It should be noted that the server described in steps S101 to S106 refers to Figure 2 The edge server in step S101 to step S106 is described in the client and Figure 2 Same as the client in .

[0081] In steps S101 to S106 shown in the embodiment of the present application, the first playback rate is determined by the content similarity data of the first target frame block data, and the initial transmission rate is adjusted according to the first playback rate and the first cache occupancy rate to obtain the first target transmission rate. It can be seen that the adjustment process of the first target transmission rate takes into account the content similarity data of the first target frame block. Therefore, the embodiment of the present application can reduce the impact on the content of the first target frame block on the basis of realizing the transmission rate adjustment. When the embodiment of the present application is applied to the remote consultation scenario in digital medicine, the consultation efficiency and quality can be improved.

[0082] In step S101 of some embodiments, the server obtains a live video stream from a remote video server and caches the live video stream. It is understandable that when the embodiments of the present application are applied to digital medical scenarios, the live video stream can be a video stream corresponding to health management, a video stream corresponding to remote consultation, etc. When the embodiments of the present application are applied to financial scenarios, the live video stream can be a video stream corresponding to an insurance product introduction, a video stream corresponding to insurance knowledge training, etc. The server also obtains a preset initial transmission rate, which refers to the rate at which the live video stream is transmitted to the client.

[0083] In step S102 of some embodiments, the server performs frame segmentation processing on the cached live video stream to divide the live video stream into a plurality of frame block data, and uses one of the frame block data as the first target frame block data. Figure 3 In some embodiments, the live video stream may be divided into k frame blocks, and the number of image frames included in each frame block may be the same or different.

[0084] Reference Figure 4A In some embodiments, the live video stream includes an original image, and step S102 includes but is not limited to steps S401 to S403.

[0085] Step S401, acquiring two original images of adjacent frames to obtain an original image pair;

[0086] Step S402, performing frame difference calculation on the original image pair to obtain frame change data;

[0087] Step S403: If the frame change data is less than or equal to a preset threshold, the live video stream is frame-processed according to the original image to obtain first target frame block data.

[0088] In step S401 of some embodiments, the live video stream includes a plurality of original images, and the plurality of original images are arranged in a frame sequence. Two adjacent original images in the frame sequence are obtained, and the two original images are taken as an original image pair.

[0089] In step S402 of some embodiments, frame difference calculation is performed on the original image pair to determine the pixel difference between the corresponding two original images to obtain frame change data.

[0090] Reference Figure 5 In some embodiments, step S402 includes but is not limited to steps S501 to S502.

[0091] Step S501, performing grayscale conversion on the original image pair to obtain a grayscale image pair;

[0092] Step S502, calculating the grayscale difference of the grayscale image pair to obtain frame change data.

[0093] In step S501 of some embodiments, any of the following operations is performed on the original image pair to achieve grayscale conversion and obtain a grayscale image pair: average method, weighted average method, maximum method, minimum method, weighted component method, mean method, etc.

[0094] In step S502 of some embodiments, grayscale difference calculation is performed on the grayscale image pair to determine the pixel difference of the grayscale image pair and obtain frame change data. It is understandable that the grayscale difference calculation can be performed by any of the following methods: mean square error method, peak signal-to-noise ratio method, etc., which is not specifically limited in the embodiments of the present application.

[0095] In step S403 of some embodiments, if the frame change data is less than or equal to a preset threshold, it indicates that the pixel difference of the original image pair is small. At this time, the original image of the adjacent frame to the original image pair is obtained, and the frame difference calculation is performed to determine whether there is an original image with a small difference from the original image pair in the adjacent frame. The original image pairs with adjacent frames and small pixel differences are taken as the original image set, and the live video stream is framed according to the original image set to obtain the first target frame block data. For example, referring to Figure 4B, the live video stream includes N frames of original images, where N is an arbitrary positive integer. The first frame of the original image and the second frame of the original image are obtained, and the first frame of the original image and the second frame of the original image are used as the first original image pair. If the frame change data of the first original image pair is less than or equal to the preset threshold, the original image of the frame adjacent to the second frame of the original image is obtained, that is, the third frame of the original image. The second frame of the original image and the third frame of the original image are used as the second original image pair. If the frame change data of the second original image pair is less than or equal to the preset threshold, the original image of the frame adjacent to the third frame of the original image is obtained, that is, the fourth frame of the original image. Repeat the above operation until the frame change data of a certain original image pair is greater than the preset threshold. For example, the 3rd frame original image and the 4th frame original image are taken as the third original image pair, if the frame change data of the third original image pair is greater than a preset threshold, the first original image pair and the second original image pair are taken as the original image set, and the live video stream is frame-processed according to the original image set to obtain the first target frame block data, that is, the first target frame block data includes the 1st frame original image, the 2nd frame original image, and the 3rd frame original image.

[0096] It can be understood that the above operation can be repeated on other frames of original images in the live video stream based on the 4th frame of original image to obtain the second target frame block data.

[0097] It is understandable that the preset threshold can be adaptively set according to the actual situation, and the embodiment of the present application does not specifically limit this. When the frame change data is greater than the preset threshold, it indicates that the corresponding two original images have a large pixel difference, which may be caused by content switching and other reasons. For example, in the remote consultation scenario of digital medicine, when the object describing the disease switches from object A to object B, the above situation may occur. It can be seen that the advantage of steps S401 to S403 of the embodiment of the present application is that the live video stream can be framed into multiple target frame block data, and each target frame block data includes multiple original images with small pixel differences. Therefore, in subsequent operations, the playback rate, transmission rate and other operations can be performed in units of one target frame block data, which not only reduces the amount of data for operation processing, but also enables the playback rate and transmission rate to be more adapted to the pixel situation of the target frame block data.

[0098] In step S103 of some embodiments, the content similarity calculation refers to calculating the similarity of the content between each frame image in the first target frame block data to obtain content similarity data. It can be seen that the content similarity data can reflect the degree of change of the content in the first target frame block data. When the content similarity data is larger, it indicates that the frames of images in the first target frame block data are more similar. For example, in the remote consultation scene in digital medicine, the first target frame block data may correspond to a scene of introducing the condition. Since only the mouth, head, hands, etc. of the object may have slight movements at this time, the frames of images in the first target frame block data are similar. When the content similarity data is smaller, it indicates that the frames of images in the first target frame block data are less similar. Still taking the remote consultation scene in digital medicine as an example, although the object is still orally describing the condition, in order to explain the condition in more detail, the object shows the medical image of the diseased part in the live broadcast screen. Therefore, compared with the situation where only the object orally describes the condition, the situation where the medical image of the diseased part is displayed will cause a greater change in the content of the corresponding frame image, thereby reducing the content similarity data of the first target frame block data. Among them, the medical images may be images generated by medical instruments such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), X-ray images, electroencephalograms, and optical photographic lights.

[0099] Reference Figure 6 In some embodiments, the first target frame block data includes a target image, and step S103 includes but is not limited to steps S601 to S605.

[0100] Step S601, acquiring two target images of adjacent frames to obtain a target image pair;

[0101] Step S602, performing brightness similarity calculation on the target image pair to obtain brightness similarity data;

[0102] Step S603, performing contrast similarity calculation on the target image pair to obtain contrast similarity data;

[0103] Step S604, performing structural similarity calculation on the target image pair to obtain structural similarity data;

[0104] Step S605, obtaining content similarity data according to the brightness similarity data, the contrast similarity data, and the structure similarity data.

[0105] In step S601 of some embodiments, the first target frame block data includes a plurality of target images, and the plurality of target images are arranged in a frame sequence. Two adjacent target images in the frame sequence are acquired, and the two target images are taken as a target image pair.

[0106] In step S602 of some embodiments, brightness measurement is performed on the target image pair, and brightness similarity calculation is performed based on the two brightness data obtained by measurement to obtain brightness similarity data.

[0107] In step S603 of some embodiments, contrast measurement is performed on the target image pair, and contrast similarity calculation is performed based on two contrast data obtained by measurement to obtain contrast similarity data.

[0108] In step S604 of some embodiments, structural data is obtained according to the brightness data and the contrast data, and structural similarity calculation is performed according to the two structural data to obtain structural similarity data.

[0109] In step S605 of some embodiments, the content similarity data of a target image pair in the kth target frame block is calculated based on the brightness similarity data, the contrast similarity data, the structure similarity data and the following formula (1): It can be understood that the method for determining the target image pair is similar to the method for determining the original image pair, and this embodiment of the present application will not be described in detail.

[0110]

[0111] in, Represents brightness similarity data, represents contrast similarity data, Represents structural similarity data. Assuming that the kth frame block data includes n target images, then i≤n-1. It can be understood that when k is 1, the content similarity data of a target image pair in the first target frame block data can be calculated according to formula (1).

[0112] Based on the calculation result of formula (1), the content similarity data of the target frame block data can be calculated according to the following formula (2): When k is 1, the content similarity data of the first target frame block data can be calculated according to formula (2):

[0113]

[0114] In step S104 of some embodiments, the first playback rate refers to the number of frames per second presented when the first target frame block data is played on the client. Since the object is not sensitive to the perception of the playback rate, that is, when the value of the content similarity data is larger, even if the playback rate of the corresponding frame block data is increased, it will not affect the object's senses. On the contrary, when the value of the content similarity data is smaller, if a larger playback rate is used at this time, the picture will be discontinuous. Therefore, the first playback rate of the first target frame block data can be determined according to the content similarity data.

[0115] Reference Figure 7 In some embodiments, step S104 includes but is not limited to steps S701 to S702.

[0116] Step S701, constructing a mapping relationship according to a preset content similarity range and a preset playback rate range;

[0117] Step S702: Perform playback rate mapping processing on content-similar data according to the mapping relationship to obtain a first playback rate.

[0118] In step S701 of some embodiments, the present application embodiment determines content similarity data in a mapping manner. Specifically, a numerical range of content similarity data, that is, a content similarity range, is preset, such as The numerical range of the playback rate is set, that is, the playback rate range, which can be [0.8, 1]. According to the linear mapping function g(·), the mapping relationship shown in the following formula (3) is constructed.

[0119]

[0120] In step S702 of some embodiments, when k=1, according to equation (3) and the content similarity data of the first target frame block data The first playback rate θ(x1) can be mapped.

[0121] The advantage of determining the first playback rate by constructing a mapping relationship from step S701 to step S702 is that the numerical range of the first playback rate can be controlled, which is conducive to use in different application scenarios. For example, for the remote consultation scenario in digital medicine, since it is necessary to display medical images of the diseased part, the smoothness and clarity of the video picture are required to be high, so the corresponding playback rate range can be set to a large numerical range. For the insurance introduction scenario of financial technology, the smoothness and clarity of the video picture are required to be low, so the corresponding playback rate range can be set to a small numerical range. In addition, the linear mapping function is simple and easy to implement, so the method provided in the embodiment of the present application is easy to implement.

[0122] In step S105 of some embodiments, the first cache occupancy rate refers to the occupancy rate of a specific cache area in the client memory. The specific cache area may be a video cache area, an area where the video cache and image cache are combined, an area where the video cache and other data caches are combined, etc., which is not specifically limited in the embodiments of the present application.

[0123] In step S106 of some embodiments, when the first cache occupancy rate is smaller, the client is more likely to experience video playback jamming. When the first cache occupancy rate is larger, the client is more likely to experience frame data loss. In addition, when the first playback rate is faster, the first target frame block data is smoother, but a faster transmission rate is required to support it. It can be seen that in order to ensure the rationality of the setting of the first target frame block transmission rate, the initial transmission rate needs to be adjusted according to the first cache occupancy rate and the first playback rate to obtain the first target transmission rate. At this time, the obtained first target transmission rate is more in line with the playback situation of the client.

[0124] Reference Figure 8 In some embodiments, step S106 includes but is not limited to steps S801 to S803.

[0125] Step S801, if the first buffer occupancy rate is less than or equal to the preset first occupancy rate, the initial transmission rate is adjusted to be greater than the first playback rate to obtain a first target transmission rate;

[0126] Step S802, if the first buffer occupancy rate is greater than the first occupancy rate and less than the preset second occupancy rate, adjusting the initial transmission rate to be equal to the first playback rate; wherein the second occupancy rate is greater than the first occupancy rate, obtaining the first target transmission rate;

[0127] Step S803: If the first buffer occupancy rate is greater than or equal to the second occupancy rate, the initial transmission rate is adjusted to be less than the first playback rate to obtain a first target transmission rate.

[0128] In some embodiments, in step S801 to step S803, the upper limit and the lower limit of the occupancy rate are preset, that is, the second occupancy rate T upper and the first occupancy rate T lower If the first cache occupancy rate is O t At the first occupancy rate T lower and the second occupancy rate T upper If the first buffer occupancy rate is O, the initial transmission rate is adjusted to be consistent with the first playback rate. t Less than the first occupancy rate T lower , then adjust the initial transmission rate to be greater than the first playback rate. t Greater than the second occupancy rate T upper , then the initial transmission rate is adjusted to be less than the first playback rate. Specifically, the initial transmission rate can be adjusted according to the following formula (4) to obtain the first target transmission rate T(x1).

[0129]

[0130] It can be understood that the first occupancy rate Tlower It can be set to 50% or other values. The second occupancy rate T upper It can be set to 80% or other values, which are not specifically limited in the present embodiment. In addition, the method of adjusting the initial transmission rate to be greater than the first playback rate and the method of adjusting the initial transmission rate to be less than the first playback rate shown in formula (4) are only exemplary, that is, according to actual needs, the initial transmission rate can also be adjusted. and The value of is adjusted, and this embodiment of the present application does not make any specific limitation on this.

[0131] Reference Fig.9A and Fig. 9B In some embodiments, the method for adjusting the video stream transmission rate provided in the embodiments of the present application also includes but is not limited to steps S901 to S904.

[0132] Step S901, transmitting first target frame block data to a client according to a first target transmission rate;

[0133] Step S902, obtaining a second buffer occupancy rate after the client receives the first target frame block data;

[0134] Step S903, performing frame processing on the live video stream to obtain second target frame block data;

[0135] Step S904, adjusting the first target transmission rate according to the second cache occupancy and the second target frame block data to obtain a second target transmission rate; wherein the second target transmission rate represents the rate at which the server transmits the second target frame block data to the client.

[0136] In steps S901 to S903 of some embodiments, the first target frame block data is transmitted to the client according to the first target transmission rate. After the client receives the complete first target frame block data, the first target frame block data can be played according to the first playback rate. Since different target frame block data have different numbers of target images, that is, different target frame block data n have different values, in order to ensure that the second target frame block data can also be played smoothly, it is necessary to determine the change in the cache occupancy rate after the client receives the first target frame block data, and thus obtain the second cache occupancy rate. Among them, the second target frame block data is adjacent to the first target frame block data, and the method for obtaining the second target frame block data is similar to the method for obtaining the first target frame block data, that is, both are obtained by framing the live video stream, which will not be repeated in the embodiments of the present application.

[0137] In step S904 of some embodiments, content similarity calculation is performed on the second target frame block data to obtain content similarity data of the second target frame block data. Similar data based on this content Determine the second playback rate θ(x2) of the second target frame block data. Adjust the first target transmission rate according to the second cache occupancy rate and the second playback rate θ(x2) to obtain the second target transmission rate. The method of the second playback rate θ(x2) and the second target transmission rate is similar to the method of the first target frame block, and will not be described in detail in the embodiments of the present application.

[0138] In some embodiments, the above operation is performed on all target frame block data in the live video stream until the kth target frame block data is transmitted.

[0139] The advantage of steps S901 to S904 is that the second target transmission rate of the second target frame block data can be adjusted according to the second cache occupancy rate fed back by the client, thereby reducing the stuttering and other playback problems caused by the mismatch of the client cache occupancy rate while meeting the network requirements.

[0140] The embodiment of the present application determines the first playback rate through the content similarity data of the first target frame block data, and adjusts the initial transmission rate according to the first playback rate and the first cache occupancy rate to obtain the first target transmission rate. It can be seen that the adjustment process of the first target transmission rate takes into account the content similarity data of the first target frame block. Therefore, the embodiment of the present application can reduce the impact on the content of the first target frame block on the basis of realizing the transmission rate adjustment. When the embodiment of the present application is applied to the remote consultation scenario in digital medicine, the consultation efficiency and quality can be improved.

[0141] Reference Fig.10 The embodiment of the present application further provides a device for adjusting the transmission rate of a video stream, which can implement the above-mentioned method for adjusting the transmission rate of a video stream, and the device includes:

[0142] The data acquisition module 1001 is used to acquire a live video stream and a preset initial transmission rate; wherein the initial transmission rate represents the rate at which the server transmits the live video stream to a preset client;

[0143] A frame processing module 1002 is used to perform frame processing on the live video stream to obtain first target frame block data;

[0144] A content similarity calculation module 1003 is used to perform content similarity calculation on the first target frame block data to obtain content similarity data;

[0145] A playback rate determination module 1004 is configured to determine a first playback rate of the first target frame block data according to the content similarity data; wherein the first playback rate indicates a rate at which the client plays the first target frame block data;

[0146] The cache occupancy acquisition module 1005 is used to acquire the first cache occupancy of the client;

[0147] The transmission rate adjustment module 1006 is used to adjust the initial transmission rate according to the first cache occupancy rate and the first playback rate to obtain a first target transmission rate; wherein the first target transmission rate represents the rate at which the server transmits the first target frame block data to the client.

[0148] The specific implementation of the video stream transmission rate adjustment device is basically the same as the specific implementation of the above-mentioned video stream transmission rate adjustment method, and will not be repeated here.

[0149] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method for adjusting the video stream transmission rate when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0150] Reference Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0151] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0152] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the method for adjusting the video stream transmission rate of the embodiment of the present application;

[0153] Input / output interface 1103, used to implement information input and output;

[0154] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0155] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0156] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0157] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for adjusting the video stream transmission rate is implemented.

[0158] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] The video stream transmission rate adjustment method and device, electronic device and storage medium provided in the embodiments of the present application determine the first playback rate through the content similarity data of the first target frame block data, and adjust the initial transmission rate according to the first playback rate and the first cache occupancy rate to obtain the first target transmission rate. It can be seen that the adjustment process of the first target transmission rate takes into account the content similarity data of the first target frame block. Therefore, the embodiments of the present application can reduce the impact on the content of the first target frame block on the basis of realizing the transmission rate adjustment. When the embodiments of the present application are applied to the remote consultation scenario in digital medicine, the consultation efficiency and quality can be improved.

[0160] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0161] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0162] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0164] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0165] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0166] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0167] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0170] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for adjusting a video stream transmission rate, characterized in that: The method is applied to the server, and the method includes: Acquire a live video stream and a preset initial transmission rate; wherein the initial transmission rate represents the rate at which the server transmits the live video stream to a preset client; Performing frame processing on the live video stream to obtain first target frame block data; Calculating content similarity on the first target frame block data to obtain content similarity data; Determine a first playback rate of the first target frame block data according to the content similarity data; wherein the first playback rate represents a rate at which the client plays the first target frame block data; Obtaining a first cache occupancy rate of the client; The initial transmission rate is adjusted according to the first cache occupancy rate and the first playback rate to obtain a first target transmission rate; wherein the first target transmission rate represents the rate at which the server transmits the first target frame block data to the client.

2. The method according to claim 1, characterized in that The step of adjusting the initial transmission rate according to the first buffer occupancy rate and the first playback rate to obtain a first target transmission rate includes: If the first buffer occupancy rate is less than or equal to a preset first occupancy rate, adjusting the initial transmission rate to be greater than the first playback rate to obtain the first target transmission rate; If the first buffer occupancy rate is greater than the first occupancy rate and less than a preset second occupancy rate, adjusting the initial transmission rate to be equal to the first playback rate; wherein the second occupancy rate is greater than the first occupancy rate, obtaining the first target transmission rate; If the first cache occupancy rate is greater than or equal to the second occupancy rate, the initial transmission rate is adjusted to be less than the first playback rate to obtain the first target transmission rate.

3. The method according to claim 1, characterized in that The live video stream includes an original image, and the frame processing of the live video stream to obtain first target frame block data includes: Acquire two original images of adjacent frames to obtain an original image pair; Performing frame difference calculation on the original image pair to obtain frame change data; If the frame change data is less than or equal to a preset threshold, the live video stream is frame-processed according to the original image to obtain the first target frame block data.

4. The method according to claim 3, characterized in that The performing frame difference calculation on the original image pair to obtain frame change data includes: Performing grayscale conversion on the original image pair to obtain a grayscale image pair; Grayscale difference calculation is performed on the grayscale image pair to obtain the frame change data.

5. The method according to claim 1, characterized in that The first target frame block data includes a target image, and the content similarity calculation is performed on the first target frame block data to obtain content similarity data, including: Acquire two target images of adjacent frames to obtain a target image pair; Performing brightness similarity calculation on the target image pair to obtain brightness similarity data; Performing contrast similarity calculation on the target image pair to obtain contrast similarity data; Performing structural similarity calculation on the target image pair to obtain structural similarity data; The content similarity data is obtained according to the brightness similarity data, the contrast similarity data, and the structure similarity data.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: transmitting the first target frame block data to the client according to the first target transmission rate; Obtaining a second cache occupancy rate after the client receives the first target frame block data; Performing frame processing on the live video stream to obtain second target frame block data; The first target transmission rate is adjusted according to the second cache occupancy and the second target frame block data to obtain a second target transmission rate; wherein the second target transmission rate represents the rate at which the server transmits the second target frame block data to the client.

7. The method according to claim 6, characterized in that The determining a first playback rate of the first target frame block data according to the content similarity data comprises: A mapping relationship is established according to a preset content similarity range and a preset playback rate range; The content similarity data is subjected to play rate mapping processing according to the mapping relationship to obtain the first play rate.

8. A device for adjusting the transmission rate of a video stream, characterized in that: The device is applied to a server, and includes: A data acquisition module, used to acquire a live video stream and a preset initial transmission rate; wherein the initial transmission rate represents the rate at which the server transmits the live video stream to a preset client; A frame processing module, used for performing frame processing on the live video stream to obtain first target frame block data; A content similarity calculation module, used for performing content similarity calculation on the first target frame block data to obtain content similarity data; A playback rate determination module, configured to determine a first playback rate of the first target frame block data according to the content similarity data; wherein the first playback rate represents a rate at which the client plays the first target frame block data; A cache occupancy acquisition module, used to acquire a first cache occupancy of the client; A transmission rate adjustment module is used to adjust the initial transmission rate according to the first cache occupancy rate and the first playback rate to obtain a first target transmission rate; wherein the first target transmission rate represents the rate at which the server transmits the first target frame block data to the client.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images