Video frame transmission method, device, electronic device and storage medium

Abstract

The present disclosure provides a video frame transmission method, device, electronic device and storage medium, which relate to the field of computer technology, and in particular to the field of cloud desktop technology. The specific implementation scheme is: obtain the video frame to be transmitted; select the target area coordinates from the area selector, the area selector stores multiple area coordinates, each area coordinate is determined based on the dynamic area of ​​multiple continuous video frames that have completed transmission; perform video encoding on the area corresponding to the target area coordinates in the video to be transmitted to obtain video encoding data, and perform image encoding on the image blocks that have changed compared with the previous adjacent video frame in the remaining area of ​​the video to be transmitted to obtain image encoding data; transmit the video encoding data and image encoding data to the client. Transmission traffic can be saved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of cloud desktop technology. Background Art

[0002] Cloud desktop, also known as desktop virtualization or cloud computer, is a new model that replaces traditional computers. Cloud servers can transmit video streams to terminals. After receiving the video streams, the client can render and display the cloud desktop based on the video streams. In order to ensure clarity, cloud servers currently usually use a higher bit rate to transmit video streams. Summary of the invention

[0003] The present disclosure provides a video frame transmission method, device, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, a video frame transmission method is provided, the method comprising:

[0005] Get the video frame to be transmitted;

[0006] Selecting target region coordinates from a region selector, wherein the region selector stores a plurality of region coordinates, each region coordinate being determined based on a dynamic region of a plurality of consecutive video frames that have completed transmission;

[0007] Performing video encoding on the area corresponding to the coordinates of the target area in the video to be transmitted to obtain video encoding data, and performing image encoding on the image blocks in the remaining areas of the video to be transmitted that have changed compared with the previous adjacent video frame to obtain image encoding data;

[0008] The video encoding data and the image encoding data are transmitted to a client.

[0009] According to a second aspect of the present disclosure, a video frame transmission device is provided, the device comprising:

[0010] An acquisition module is used to acquire video frames to be transmitted;

[0011] A selection module, used for selecting target region coordinates from a region selector, wherein the region selector stores a plurality of region coordinates, each of which is determined based on a dynamic region of a plurality of continuous video frames that have completed transmission;

[0012] An encoding module, configured to perform video encoding on the area corresponding to the coordinates of the target area in the video to be transmitted to obtain video encoding data, and to perform image encoding on the image blocks in the remaining areas of the video to be transmitted that have changed compared with the previous adjacent video frame to obtain image encoding data;

[0013] A transmission module is used to transmit the video encoding data and the image encoding data to a client.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flow chart of a video frame transmission method provided by an embodiment of the present disclosure;

[0023] Figure 2 An exemplary schematic diagram of a region selector provided by an embodiment of the present disclosure;

[0024] Figure 3 is an exemplary schematic diagram of a result of marking an image block in a video frame provided by an embodiment of the present disclosure;

[0025] Figure 4 is an exemplary schematic diagram of a histogram for determining a candidate dynamic region provided by an embodiment of the present disclosure;

[0026] Figure 5 is an exemplary flow chart of a cloud desktop display method provided by an embodiment of the present disclosure;

[0027] Figure 6 is an exemplary schematic diagram of a cloud desktop interface provided by an embodiment of the present disclosure;

[0028] Figure 7is a structural schematic diagram of a video frame transmission device provided by an embodiment of the present disclosure;

[0029] Figure 8 It is a block diagram of an electronic device used to implement the video frame transmission method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] The embodiment of the present application provides a video frame transmission method, which is applied in the cloud desktop scenario, and is specifically applied to the process of a cloud server transmitting a video stream to a client. The method is executed by the cloud server, such as Figure 1 As shown, the method includes:

[0032] S101: Obtain a video frame to be transmitted.

[0033] The video frame to be transmitted is a video frame in the video stream that the cloud server needs to transmit to the client, and each video frame included in the video stream may include a video screen. For example, when a user uses a cloud desktop and uses a small video playback window in a web page to watch a video, each video frame of the video stream includes the web page displayed in the cloud desktop and the video screen played in the video playback window in the web page.

[0034] S102: Select the target area coordinates from the area selector.

[0035] The region selector stores a plurality of region coordinates, each of which is determined based on the dynamic region of a plurality of consecutive video frames that have been transmitted. For example, a region coordinate may be determined based on the dynamic region of a plurality of consecutive video frames transmitted within a preset time period at intervals of a preset time period; or, a region coordinate may be determined based on the dynamic region of a specified number of video frames each time a specified number of video frames are transmitted. The method for determining region coordinates will be described in detail in subsequent embodiments.

[0036] It is understandable that if the same area in multiple consecutive video frames continues to change, it means that the area may be a video playback window, that is, a dynamic area.

[0037] S103 , performing video encoding on the area corresponding to the coordinates of the target area in the video to be transmitted to obtain video encoding data, and performing image encoding on the image blocks in the remaining areas of the video to be transmitted that have changed compared with the previous adjacent video frame to obtain image encoding data.

[0038] Taking the scenario of playing a video in a small video playback window in a web page as an example, the changed image block may be the content in the search box in the web page.

[0039] S104: Transmit the video encoding data and the image encoding data to the client.

[0040] Among them, after the client receives the video encoding data, it can render the video in the target area coordinates in the cloud desktop based on the video encoding data, and render the area corresponding to each image block in the cloud desktop based on the image encoding of each image block received. The remaining image blocks that have not changed do not need to be re-rendered.

[0041] By adopting this method, after obtaining the video frame to be transmitted, the target area coordinates can be selected from the area selector, the area corresponding to the target area coordinates in the video to be transmitted is video encoded, and the image blocks that have changed in the video frame to be transmitted are image encoded, and there is no need to perform video encoding on other areas. Compared with video encoding of a complete video frame, the embodiment of the present disclosure reduces the amount of data transmitted to the client, so that the transmission bit rate can be reduced, and a higher-definition video stream can be transmitted with a smaller bit rate, which can save traffic. In the case of a poor network, the user can still use the cloud desktop smoothly. In addition, since each area coordinate in the area selector is determined based on the dynamic area of ​​multiple continuous video frames that have completed transmission, the target area coordinates selected from the area selector can represent the dynamic area in the video stream within a period of time, which can avoid affecting the display effect due to frequent changes in the area for video encoding. It can be seen that the embodiment of the present disclosure can save transmission traffic without affecting the display effect of the cloud desktop.

[0042] In some embodiments of the present disclosure, as an optional implementation, the above S102 may be specifically implemented as: selecting the latest stored region coordinates from the region selector as the target region coordinates.

[0043] It is understandable that the latest stored region coordinates are determined based on the dynamic regions of multiple consecutive video frames that have recently completed transmission. The dynamic region in the most recently completed video frame is likely to be the same as the dynamic region in the current video frame to be transmitted, so the latest stored region coordinates can be used as the region that needs to be video encoded in the video frame to be transmitted, which can reduce the amount of video encoding data and save transmission traffic.

[0044] As another optional implementation, the region selector also stores the category identifier of each region coordinate, and the difference between the region coordinates with the same category identifier is less than a preset threshold. In addition, the region selector is a first-in-first-out queue with a fixed length, that is, after the region selector is full, if new region coordinates need to be stored later, the region coordinates stored in the region selector earliest will be removed, so that the new region coordinates can be stored.

[0045] On this basis, the above S102 can also be implemented as follows:

[0046] The target category identifier with the highest proportion in the region selector is determined, and the target region coordinates that are last stored in the region selector are selected from the region coordinates with the target category identifiers.

[0047] As an example, Figure 2 As shown, Figure 2 The area selector shown in stores 8 area coordinates, namely, area coordinates 1-8, and the corresponding category identifiers are 1, 1, 1, 1, 1, 1, 1, 0, respectively. It can be determined that category identifier 1 accounts for the largest proportion, and then area coordinate 7 is selected as the target area coordinate.

[0048] It should be noted that Figure 2 As an example only, the number of region coordinates stored in the region selector in actual implementation is not limited thereto. Each region coordinate in the embodiment of the present disclosure may include the coordinates of the four corners of the rectangular region, or include the upper left corner coordinates and the lower right corner coordinates of the rectangular region, or include the lower left corner coordinates and the upper right corner coordinates of the rectangular region.

[0049] When calculating the difference between the area coordinates, the difference of the coordinates of each corner included in the two area coordinates is calculated respectively. For example, when the area coordinates include the coordinates of the upper left corner and the lower right corner of the rectangular area, the difference of the coordinates of the upper left corner and the difference of the coordinates of the lower right corner of the two area coordinates are calculated respectively. If each calculated difference is less than a preset threshold, it is determined that the difference between the two area coordinates is less than the preset threshold.

[0050] With this method, since the region selector is a first-in, first-out queue of fixed length, the region coordinates currently stored in the region selector are determined based on the video frames transmitted in the most recent period of time, and can represent the dynamic region in the video stream in the most recent period of time. In addition, since the target region coordinates in the disclosed embodiment come from the region coordinates corresponding to the target category identifier with the highest proportion, it is possible to avoid performing video encoding according to a temporarily determined inaccurate region coordinate, improve the accuracy of the determined target dynamic region, and avoid frequent and substantial adjustments to the video encoding region. And under the premise of ensuring accuracy, selecting the region coordinates most recently stored in the region selector can improve the stability and smoothness of the cloud desktop display.

[0051] In some embodiments of the present disclosure, after acquiring the video to be transmitted, the cloud server can divide the video frame to be transmitted into multiple image blocks of the same size according to a preset division size, and compare the video frame to be transmitted with each image block in the previous adjacent video frame, and mark the changed image blocks in the video frame to be transmitted as a first marking value, and mark the unchanged image blocks as a second marking value.

[0052] It should be noted that, in the embodiments of the present disclosure, the manner of dividing each to-be-transmitted video frame into image blocks is the same, and the same preset division size is used.

[0053] For example, for a 1920×1080 video frame, the preset division size may be 20×20, that is, the size of each image block is 20×20, so each video frame may be divided into 96×54 image blocks.

[0054] Optionally, when comparing the video frame to be transmitted with each image block in the previous adjacent video frame, memcmp may be used for comparison, and memcmp is a byte-by-byte comparison function. As an example, the first mark value may be 1, and the second mark value may be 0. That is, the changed image block is marked as 1, and the unchanged image block is marked as 0. Figure 3 As shown, Figure 3 FIG. 2 shows two video frames after being marked. The image blocks marked as 1 are changed image blocks, and the image blocks marked as 0 are unchanged image blocks. Figure 3 The shaded areas in the two video frames are areas with more image blocks marked as 1 in the video frames.

[0055] Accordingly, image encoding is performed on image blocks in the remaining area of ​​the video to be transmitted that have changed compared with the previous adjacent video frame, which can be specifically implemented as follows:

[0056] Image encoding is performed on image blocks whose marking values ​​in the remaining areas of the video to be transmitted are the first marking values.

[0057] By adopting this method, the changed image blocks in adjacent video frames can be compared, and then the changed image blocks except the video encoding area can be encoded. The amount of image encoding data is smaller than the amount of video encoding data, so the amount of data transmitted from the cloud server to the client can be reduced, and the encoded data of the changed image blocks can be accurately transmitted to the client, saving transmission traffic without affecting the display effect of the cloud desktop.

[0058] In some embodiments of the present disclosure, after marking each image block in the video frame to be transmitted, the method further includes:

[0059] The first mark values ​​of each row of image blocks in the video frame to be transmitted are accumulated, and a row histogram is generated based on the accumulated values ​​of each row; the first mark values ​​of each column of image blocks in the video frame to be transmitted are accumulated, and a column histogram is generated based on the accumulated values ​​of each column.

[0060] Based on the maximum rectangle in the row histogram and the maximum rectangle in the column histogram, a candidate dynamic region in the to-be-transmitted video frame is determined.

[0061] Optionally, the direction of the row (e.g., the left direction) can be used as the vertical axis of the row histogram, the height of each image block can be used as the width of each rectangle in the row histogram, and the accumulated value of each row can be used as the height of the rectangle corresponding to the row in the row histogram. That is, the width of each rectangle in the row histogram is the same as the height of a single image block.

[0062] Similarly, the direction of the column (for example, the right direction) can be used as the vertical axis of the column histogram, the width of each image block can be used as the width of each rectangle in the row histogram, and the accumulated value of each column can be used as the height of the rectangle corresponding to the column in the column histogram. That is, the width of each rectangle in the column histogram is the same as the width of a single image block.

[0063] For example, Figure 4 As shown, Figure 4 , a marked video frame is shown in FIG. , the histogram on the left side of the video frame is a row histogram, and the histogram below is a column histogram.

[0064] It can be seen that the height of the top rectangle in the row histogram is 3, and the change in the y-axis direction is 3, which is the cumulative value of the 3 1s appearing in the first row of the video frame; the height of the second rectangle is 1, and the change in the y-axis direction is 1, which is the cumulative value of the 1s appearing in the second row of the video frame; similarly, the heights of the subsequent rectangles are 4, 4, 3, and 2, respectively.

[0065] Similarly, the leftmost rectangle in the column histogram has a height of 1 and a change of 1 in the y-axis direction, which is the cumulative value of 1 appearing in the first column of the video frame. The second rectangle has a height of 3 and a change of 3 in the y-axis direction, which is the cumulative value of 1 appearing in the second column of the video frame. Similarly, the heights of the subsequent rectangles are 6, 3, 3, and 1, respectively.

[0066] Since the image patches are square, the width of each rectangle in both the row and column histograms is the same.

[0067] Then, the largest rectangle in the row histogram and the largest rectangle in the column histogram can be determined.

[0068] In one implementation, the cloud server may determine the number of rows of the candidate dynamic region in the video frame to be transmitted based on the maximum rectangle in the row histogram. The number of columns of the candidate dynamic region in the video frame to be transmitted may be determined based on the maximum rectangle in the column histogram. The region corresponding to the number of rows and columns in the video frame to be transmitted is then used as the candidate dynamic region.

[0069] Specifically, the number of rows in the video to be transmitted corresponding to an edge in the largest rectangle in the row histogram that is parallel to the rows of the video frame to be transmitted can be used as the number of rows in the candidate dynamic area in the video frame to be transmitted, and the number of columns in the video frame to be transmitted corresponding to an edge in the largest rectangle in the column histogram that is parallel to the columns of the video frame to be transmitted can be used as the number of columns in the candidate dynamic area in the video frame to be transmitted.

[0070] Still Figure 4 For example, Figure 4 The bold rectangle in the row histogram on the left is the largest rectangle in the row histogram, and the edge of the rectangle on the x-axis corresponds to the 3rd to 5th rows of the video frame.

[0071] and Figure 4 The bold rectangle in the column histogram below is the largest rectangle in the column histogram, and the edge of the rectangle on the x-axis corresponds to the 2nd to 5th columns of the video frame.

[0072] Then, the overlapping areas from the 3rd to the 5th row and from the 2nd to the 5th column can be taken as candidate dynamic areas, that is, Figure 4 The shaded area in is the candidate dynamic area.

[0073] It is understandable that the changes of each frame in the video stream are rapid and disorderly. Even in the video window area, some image blocks in two adjacent video frames may not change, and image blocks in areas outside the video window area may also change. Figure 3 The shaded area in the middle is the actual video window area. It can be seen that Figure 3 In the two video frames shown, there are also some image blocks marked as 0 in the video window area, and there are also some image blocks marked as 1 outside the video window area. The image blocks that change outside the video window area will interfere with the determination of the video window area in each video frame, making it difficult to accurately determine the dynamic area corresponding to the video window in each video frame. The method of the embodiment of the present disclosure is equivalent to converting the complex problem of determining the candidate dynamic area into the problem of finding the largest rectangle in the histogram. By finding the problem of the largest rectangle, the candidate dynamic area in the video frame can be accurately located, and the complexity of calculating the largest rectangle is low, and the calculation time is short. According to experiments, the calculation time is less than 1 millisecond. It can be seen that the method can be used to accurately locate the candidate dynamic area using simple calculations.

[0074] After determining the candidate dynamic regions in the video frame to be transmitted, a dynamic region can be determined based on multiple candidate dynamic regions determined from multiple consecutive video frames, the region coordinates of the determined dynamic region are added to the region selector, and a category identifier is set.

[0075] Optionally, the cloud server may use a candidate dynamic region with the largest proportion among the multiple candidate dynamic regions as the determined dynamic region.

[0076] For example, among multiple candidate dynamic areas, 95% of the candidate dynamic areas are 90×90 areas at the same position, and 5% of the candidate dynamic areas are 86×86 areas, then the 90×90 area can be used as the dynamic area to be determined. This can avoid the situation where the selected dynamic area is inaccurate due to small changes in individual pictures in the video window, and make the determined dynamic area as consistent as possible with the actual video window area.

[0077] After determining the candidate dynamic region in the video frame to be transmitted, the cloud server can determine whether the currently determined dynamic region that does not participate in the decision meets the decision conditions. If the decision conditions are met, a dynamic region is decided from the multiple determined candidate dynamic regions. If the decision conditions are not met, the dynamic candidate region is saved first, and after completing the transmission of the video frame to be transmitted, the candidate dynamic region of the next video frame to be transmitted can be determined until the decision conditions are met and a decision is made.

[0078] The decision condition may be made once for each preset duration, or once for each preset number of candidate dynamic regions obtained. Accordingly, a dynamic region is determined based on multiple candidate dynamic regions determined from multiple consecutive video frames, which may be specifically implemented as follows:

[0079] At every preset time period, a dynamic area is determined from the dynamic areas determined based on each video frame within the preset time period; the preset time period may be pre-set based on experience, for example, may be 5 seconds.

[0080] Alternatively, each time a specified number of candidate dynamic regions are obtained, a dynamic region is determined from the specified number of candidate regions. The specified number may be preset based on experience, for example, may be 100.

[0081] The dynamic area determined by the above method can reflect the actual dynamic area within a preset duration or the most recent preset number of video frames. Adding the dynamic area to the area selector for subsequent video encoding can make the area actually used for video encoding more stable and avoid frequent changes, making the cloud desktop playback smoother and more stable.

[0082] After a dynamic region is determined, the region coordinates of the dynamic region can be added to the region selector. If the region selector is full, the region coordinates that first enter the region selector are removed, so that the region coordinates of the latest determined dynamic region are added to the end of the queue. Figure 2 On this basis, coordinate 1 can be removed, coordinates 2 to 8 can be shifted left by one position, and then the newly determined area coordinates can be added to the original coordinate 8 position.

[0083] In addition, the region coordinates of the latest dynamic region can be compared with each region coordinate stored in the region selector. If the difference between the region coordinates of the latest dynamic region and the region coordinates corresponding to one of the category identifiers is less than a preset threshold, the same category identifier can be set for the region coordinates of the latest dynamic region.

[0084] If the differences between the region coordinates of the latest determined dynamic region and all the stored region coordinates are greater than a preset threshold, a new category identifier may be set for the dynamic region.

[0085] Still Figure 2 For example, after removing coordinate 1, if the difference between the region coordinates of the latest determined dynamic region and each region coordinate from coordinates 2 to 7 is less than a preset threshold, the region coordinates of the latest determined dynamic region can be set to 1.

[0086] By adopting this method, the region coordinates determined each time can be stored in the region selector, and a category identifier can be set for the region coordinates, so that the cloud server can select a region coordinate with the largest category identifier as the target dynamic region. This can avoid frequent switching of the region for video encoding and make the region for video encoding more stable.

[0087] For example, when using the cloud desktop, if you use a small video window on a web page to play a video, such as a TV series or movie, in the small video window, when playing a character dialogue or a character monologue, there may be a situation where only the actor's facial expression changes in the continuously played picture, and the background remains unchanged. In this case, the dynamic area determined may only be the actor's facial area in the small video window, and the rest of the area in the small video window will be determined as an unchanged image block. If the dynamic area is subsequently encoded, when a scene switch occurs in the small video window, the problem of not being able to perform video encoding for the entire small video window area in time may occur, causing the cloud desktop playback to freeze. In the disclosed embodiment, it is possible to avoid using a smaller dynamic area that appears briefly as a video encoding area. Even if the latest determined dynamic area is very small, video encoding will still be performed according to the coordinates of the area with the most category identifiers, avoiding the above-mentioned playback freeze problem.

[0088] like Figure 5 As shown, Figure 5 The following is an exemplary flow chart of a cloud desktop display method implemented based on the method of an embodiment of the present disclosure, the method comprising:

[0089] S501, capturing desktop screen pixels.

[0090] Among them, the cloud server can obtain each frame of the picture that will be displayed on the client desktop, that is, the video frame to be transmitted in the above embodiment.

[0091] S502: Divide the pixels of the desktop image into blocks to obtain a plurality of image blocks.

[0092] The pixel division of the desktop image is to divide the video frame to be transmitted into a plurality of image blocks of the same size as described in the above embodiment.

[0093] S503: Compare each image block with an image block at the same position in the previous frame to see whether it has changed.

[0094] The specific comparison method and the method for marking the image blocks may refer to the relevant description of the above embodiment, which will not be repeated here.

[0095] S504: Determine a video area and a picture area.

[0096] The video area is the target dynamic area in the video frame to be transmitted, and the picture area is the area of ​​the image block marked as 1 except the target dynamic area.

[0097] S505: Encode the video region and the picture region.

[0098] Optionally, the cloud server may perform video encoding on the video area; perform image encoding on the changed image blocks outside the video area; and do not perform encoding on the unchanged image blocks outside the video area.

[0099] S506: The client renders respectively.

[0100] The client renders the desktop screen based on the video encoding and the image encoding, thereby updating the changed parts of each frame displayed by the cloud desktop.

[0101] Using the above method, after testing, in the process of playing videos using a small window of a web page in a cloud desktop, for a 162.3MB video stream, in the relevant technology, it is necessary to completely transmit 162.3MB of data, and the network speed requirement is 0.54M / s, that is, 0.54M of data needs to be transmitted per second. Using the method of the embodiment of the present disclosure to transmit the same video stream, the amount of data required to be transmitted is 82.56MB, which is reduced by half. And under the same bit rate, the required network speed is 0.28M / s, that is, 0.28M of data needs to be transmitted per second, and the requirement for network speed can also be reduced by half. It can be seen that using this method can save half of the amount of transmission data and reduce the requirements for the network. In addition, the original 1080P resolution video requires a 4K bit rate to maintain the required clarity. In the scenario of a small window video, that is, for a resolution within 720P, only a 2K bit rate is required to achieve the required clarity. In this scenario, the bit rate can be reduced by half and half of the traffic can be saved.

[0102] As shown in Table 1, Table 1 is the test data obtained through the test. During the test, the network packet analysis software (wireshark) was used to capture the data transmitted from the cloud server to the client for 5 minutes, and the frame rate was 25.

[0103] "Static" in Table 1 means switching to the computer desktop, and the user using the cloud desktop does not perform any operation. "Small window video" means opening a web page and playing a video in a small window of the web page. "Full screen video" means opening the video in the web page in full screen.

[0104] The second to fourth rows in Table 1 are test data for three existing cloud desktop related products. The cloud desktop in row 5 is a cloud desktop product before optimization related to the embodiment of the present disclosure. The cloud desktop 2.0 in row 6 is a product implemented using the method of the embodiment of the present disclosure. Each test data in Table 1 includes the amount of data transmitted and the transmission rate.

[0105] Table 1

[0106]

[0107]

[0108] Figure 6An exemplary schematic diagram of a cloud desktop interface displayed by a client provided in an embodiment of the present disclosure, the outermost rectangular frame represents the display interface frame of the client, and when playing a video in the browser webpage of the client, the cloud server can only encode the video area when transmitting each video frame. For the remaining changed areas, such as the search terms recommended in the search box may change, and the TV series recommended under the related recommendations may also change. The cloud server can encode the image blocks corresponding to these changed areas in each video frame, and there is no need to encode the remaining unchanged parts, and the client does not need to re-render the remaining unchanged areas.

[0109] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0110] Corresponding to the above method embodiment, the present disclosure also provides a video frame transmission device, such as Figure 7 As shown, the device comprises:

[0111] An acquisition module 701 is used to acquire a video frame to be transmitted;

[0112] A selection module 702, configured to select target region coordinates from a region selector, wherein the region selector stores a plurality of region coordinates, each region coordinate being determined based on a dynamic region of a plurality of consecutive video frames that have completed transmission;

[0113] The encoding module 703 is used to perform video encoding on the area corresponding to the coordinates of the target area in the video to be transmitted to obtain video encoding data, and to perform image encoding on the image blocks in the remaining area of ​​the video to be transmitted that have changed compared with the previous adjacent video frame to obtain image encoding data;

[0114] The transmission module 704 is used to transmit the video encoding data and the image encoding data to the client.

[0115] In another embodiment, the region selector further stores a category identifier of each region coordinate, the difference between region coordinates with the same category identifier is less than a preset threshold, and the region selector is a first-in-first-out queue with a fixed length;

[0116] The selection module 702 is specifically used for:

[0117] Determine the target category identifier with the highest proportion in the region selector;

[0118] From the region coordinates with the target category identifier, select the target coordinate region that was last stored in the region selector.

[0119] In another embodiment, the apparatus further comprises:

[0120] A division module, used to divide the video frame to be transmitted into a plurality of image blocks of the same size according to a preset division size;

[0121] A marking module, used for comparing the video frame to be transmitted with each image block in the previous adjacent video frame, marking the changed image blocks in the video frame to be transmitted as a first marking value, and marking the unchanged image blocks as a second marking value;

[0122] The encoding module 703 is specifically configured to perform image encoding on the image blocks whose marking values ​​in the remaining areas of the video to be transmitted are the first marking values.

[0123] In another embodiment of the present disclosure, the device further includes:

[0124] A generating module, configured to accumulate the first marking values ​​of each row of image blocks in the video frame to be transmitted, generate a row histogram based on the accumulated values ​​of each row, and accumulate the first marking values ​​of each column of image blocks in the video frame to be transmitted, and generate a column histogram based on the accumulated values ​​of each column;

[0125] A determination module, used to determine a candidate dynamic area in a video frame to be transmitted based on a maximum rectangle in a row histogram and a maximum rectangle in a column histogram;

[0126] The decision module is used to decide a dynamic area according to multiple candidate dynamic areas determined from multiple consecutive video frames, add the area coordinates of the decided dynamic area to the area selector, and set a category identifier.

[0127] In another embodiment of the present disclosure, the decision module is specifically configured to:

[0128] At every preset time length, a dynamic area is determined from the dynamic areas determined based on each video frame within the preset time length; or,

[0129] Each time a specified number of candidate dynamic regions are obtained, a dynamic region is determined from the specified number of candidate regions.

[0130] In another embodiment of the present disclosure, the decision module is specifically configured to select a candidate dynamic region with the largest proportion among the multiple candidate dynamic regions as the decided dynamic region.

[0131] In another embodiment of the present disclosure, the width of each rectangle in the row histogram is the same as the height of a single image block, and the width of each rectangle in the column histogram is the same as the width of a single image block; the determination module is specifically used to:

[0132] Based on the maximum rectangle in the row histogram, determining the number of rows of the candidate dynamic area in the video frame to be transmitted;

[0133] Based on the maximum rectangle in the column histogram, determining the number of columns of the candidate dynamic area in the video to be transmitted;

[0134] The area corresponding to the row number and the column number in the video frame to be transmitted is taken as a candidate dynamic area.

[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0136] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0137] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0138] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0139] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the video frame transmission method. For example, in some embodiments, the video frame transmission method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the video frame transmission method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the video frame transmission method in any other appropriate manner (e.g., by means of firmware).

[0140] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0145] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0147] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video frame transmission method, the method comprising: Get the video frame to be transmitted; Selecting target region coordinates from a region selector, wherein the region selector stores a plurality of region coordinates, each region coordinate being determined based on a dynamic region of a plurality of consecutive video frames that have completed transmission; Performing video encoding on the area corresponding to the coordinates of the target area in the video to be transmitted to obtain video encoding data, and performing image encoding on the image blocks in the remaining areas of the video to be transmitted that have changed compared with the previous adjacent video frame to obtain image encoding data; Transmitting the video encoding data and the image encoding data to a client; After obtaining the video frame to be transmitted, the method further includes: Accumulating the first mark value of each row of image blocks in the video frame to be transmitted, generating a row histogram based on the accumulated value of each row, and accumulating the first mark value of each column of image blocks in the video frame to be transmitted, generating a column histogram based on the accumulated value of each column, wherein the first mark value indicates that the image block has changed compared with the previous adjacent video frame; Determine a candidate dynamic region in the to-be-transmitted video frame based on a maximum rectangle in the row histogram and a maximum rectangle in the column histogram; According to a plurality of candidate dynamic regions determined from a plurality of continuous video frames, a dynamic region is determined, the region coordinates of the determined dynamic region are added to the region selector, and a category identifier is set.

2. The method according to claim 1, wherein: The region selector also stores a category identifier of each region coordinate, the difference between region coordinates with the same category identifier is less than a preset threshold, and the region selector is a first-in-first-out queue with a fixed length; The step of selecting the target area coordinates from the area selector includes: Determine the target category identifier with the highest proportion in the region selector; From the region coordinates with the target category identifier, the target coordinate region last stored in the region selector is selected.

3. The method according to claim 1 or 2, after obtaining the video frame to be transmitted, the method further comprises: Dividing the video frame to be transmitted into a plurality of image blocks of the same size according to a preset division size; Compare the video frame to be transmitted with each image block in the previous adjacent video frame, mark the changed image blocks in the video frame to be transmitted as a first mark value, and mark the unchanged image blocks as a second mark value; The performing image encoding on the image blocks in the remaining area of ​​the video to be transmitted that have changed compared with the previous adjacent video frame comprises: Image encoding is performed on image blocks whose marking values ​​are the first marking values ​​in the remaining areas of the video to be transmitted.

4. The method according to claim 1, wherein: The step of determining a dynamic region based on a plurality of candidate dynamic regions determined from a plurality of consecutive video frames comprises: At every preset time length, a dynamic area is determined from the dynamic areas determined based on each video frame within the preset time length; or, Whenever a specified number of candidate dynamic regions are obtained, a dynamic region is determined from the specified number of candidate regions.

5. The method according to claim 1, wherein: The step of determining a dynamic region based on a plurality of candidate dynamic regions determined from a plurality of consecutive video frames comprises: A candidate dynamic region with the largest proportion among the multiple candidate dynamic regions is used as the determined dynamic region.

6. The method according to claim 1, wherein: The width of each rectangle in the row histogram is the same as the height of a single image block, and the width of each rectangle in the column histogram is the same as the width of a single image block; and determining the candidate dynamic area in the video frame to be transmitted based on the maximum rectangle in the row histogram and the maximum rectangle in the column histogram includes: Based on the largest rectangle in the row histogram, determining the number of rows of the candidate dynamic area in the video frame to be transmitted; Based on the maximum rectangle in the column histogram, determining the number of columns of the candidate dynamic area in the video to be transmitted; The area corresponding to the number of rows and the number of columns in the video frame to be transmitted is used as the candidate dynamic area.

7. A video frame transmission device, the device comprising: An acquisition module is used to acquire video frames to be transmitted; A selection module, used for selecting target region coordinates from a region selector, wherein the region selector stores a plurality of region coordinates, each of which is determined based on a dynamic region of a plurality of continuous video frames that have completed transmission; An encoding module, configured to perform video encoding on the area corresponding to the coordinates of the target area in the video to be transmitted to obtain video encoding data, and to perform image encoding on the image blocks in the remaining areas of the video to be transmitted that have changed compared with the previous adjacent video frame to obtain image encoding data; A transmission module, used for transmitting the video encoding data and the image encoding data to a client; The device also includes: A generating module, configured to accumulate the first marking values ​​of each row of image blocks in the video frame to be transmitted, generate a row histogram based on the accumulated values ​​of each row, and accumulate the first marking values ​​of each column of image blocks in the video frame to be transmitted, generate a column histogram based on the accumulated values ​​of each column, wherein the first marking value indicates that the image block has changed compared with the previous adjacent video frame; A determination module, configured to determine a candidate dynamic region in the to-be-transmitted video frame based on a maximum rectangle in the row histogram and a maximum rectangle in the column histogram; The decision module is used to decide a dynamic area according to multiple candidate dynamic areas determined from multiple consecutive video frames, add the area coordinates of the decided dynamic area to the area selector, and set a category identifier.

8. The device according to claim 7, wherein: The region selector also stores a category identifier of each region coordinate, the difference between region coordinates with the same category identifier is less than a preset threshold, and the region selector is a first-in-first-out queue with a fixed length; The selection module is specifically used for: Determine the target category identifier with the highest proportion in the region selector; From the region coordinates with the target category identifier, the target coordinate region last stored in the region selector is selected.

9. The device according to claim 7 or 8, further comprising: A division module, used for dividing the video frame to be transmitted into a plurality of image blocks of the same size according to a preset division size; a marking module, used for comparing the video frame to be transmitted with each image block in the previous adjacent video frame, marking the changed image blocks in the video frame to be transmitted as a first marking value, and marking the unchanged image blocks as a second marking value; The encoding module is specifically configured to perform image encoding on image blocks whose marking values ​​in the remaining areas of the video to be transmitted are the first marking values.

10. The device according to claim 7, wherein: The decision module is specifically used for: At every preset time length, a dynamic area is determined from the dynamic areas determined based on each video frame within the preset time length; or, Whenever a specified number of candidate dynamic regions are obtained, a dynamic region is determined from the specified number of candidate regions.

11. The device according to claim 7, wherein: The decision module is specifically configured to select a candidate dynamic region with the largest proportion among the multiple candidate dynamic regions as the decided dynamic region.

12. The device according to claim 7, wherein: The width of each rectangle in the row histogram is the same as the height of a single image block, and the width of each rectangle in the column histogram is the same as the width of a single image block; the determining module is specifically used to: Based on the largest rectangle in the row histogram, determining the number of rows of the candidate dynamic area in the video frame to be transmitted; Based on the maximum rectangle in the column histogram, determining the number of columns of the candidate dynamic area in the video to be transmitted; The area corresponding to the number of rows and the number of columns in the video frame to be transmitted is used as the candidate dynamic area.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images