An image processing method, apparatus, storage medium, and electronic device

Abstract

This disclosure provides an image processing method, apparatus, storage medium, and electronic device, relating to the field of image transmission, which can solve the problem of noticeable lag experienced by users when using cloud desktops under insufficient bandwidth networks. The specific technical solution is as follows: encoding the current frame image to generate a first bitstream; obtaining a predicted transmission time window based on the first bitstream; obtaining a maximum transmission time window based on average encoding time, average decoding time, and average display time; if the predicted transmission time window is greater than the maximum transmission time window, then executing a first preset step, including: discarding the current frame image; acquiring the next frame image and encoding the next frame image to generate a second bitstream, the number of bytes in the second bitstream being the number of bytes in the first bitstream reduced according to a preset first algorithm; using the next frame image as the new current frame image and the second bitstream as the new first bitstream; obtaining a new predicted transmission time window based on the new first bitstream and determining whether the predicted transmission time window is greater than the maximum transmission time window. This invention aims to improve user experience.

Description

Technical Field

[0001] This disclosure relates to the field of image transmission, and more particularly to an image processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Cloud desktops have been widely adopted across various industries. A cloud desktop system runs the operating system's desktop (i.e., the cloud desktop) on a cloud server. Users only need a client and an internet connection to access the server anytime, anywhere and operate their own private cloud desktop. However, in practical applications, the user experience of using a cloud desktop is closely related to the network. With sufficient bandwidth and low latency, users can achieve the same experience as using a local computer. With insufficient bandwidth and high latency, users will experience noticeable lag or even be unable to use the cloud desktop properly. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, storage medium, and electronic device that can solve the problem of users experiencing significant lag or even being unable to use cloud desktops normally under network conditions with insufficient bandwidth and high latency. The technical solution is as follows:

[0004] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0005] The current frame image is acquired and encoded to generate a first bitstream, wherein the number of bytes in the first bitstream is a preset number of bytes;

[0006] The predicted transmission time window of the current frame image is obtained based on the first bitstream and the current predicted bandwidth;

[0007] The maximum transmission time window is obtained based on the average encoding time, average decoding time, average display time, and preset maximum tolerable delay for each frame of image.

[0008] If the predicted transmission time window is greater than the maximum transmission time window, then a first preset step is executed, the first preset step including:

[0009] Discard the current frame image;

[0010] The next frame image is acquired and encoded to generate a second bitstream. The number of bytes in the second bitstream is the number of bytes after the number of bytes in the first bitstream is reduced according to a preset first algorithm.

[0011] The next frame image is used as the new current frame image, and the second bitstream is used as the new first bitstream;

[0012] A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is greater than the maximum transmission time window.

[0013] The image processing method provided in this disclosure can adjust the number of bytes in the encoded bitstream of the image according to the current predicted bandwidth of the network. After adjusting the number of bytes in the encoded bitstream, the predicted transmission time window of the image is obtained based on the adjusted number of bytes in the bitstream. When the predicted transmission time window of the image is less than or equal to the maximum transmission time window, the encoded bitstream of the image is sent to the receiving device. By adjusting the number of bytes in the encoded bitstream of the image, the transmission latency of the image can be reduced, which can prevent the stuttering phenomenon caused by network congestion in advance. This avoids the problem that users will experience obvious stuttering or even be unable to use the cloud desktop normally under network conditions with insufficient bandwidth and high latency, thus improving the user experience.

[0014] In one embodiment, the method further includes:

[0015] If the predicted transmission time window is smaller than the maximum transmission time window, then a second preset step is executed, the second preset step including:

[0016] Send the first bitstream to the receiving device;

[0017] The next frame image is acquired and encoded to generate a third bitstream. The number of bytes in the third bitstream is the number of bytes after the number of bytes in the first bitstream is increased according to a preset second algorithm.

[0018] The next frame image is used as the new current frame image, and the third bitstream is used as the new first bitstream;

[0019] A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is less than the maximum transmission time window.

[0020] By executing the second preset step when the predicted transmission time window is less than the maximum transmission time window, the bitstream of the encoded image can be gradually increased to improve the image clarity while ensuring smooth image transmission. The bitstream of the encoded image can be dynamically adjusted to improve image quality and achieve a better user experience while meeting the requirements of image transmission delay.

[0021] In one embodiment, before acquiring the current frame image and encoding the current frame image, the method further includes:

[0022] Acquire at least one frame of image;

[0023] Encode each frame of the at least one image and obtain the encoding time of each frame;

[0024] After encoding each frame of the at least one image, the at least one image is sent to the receiving device, and the average encoding time of each frame is obtained based on the encoding time of each frame.

[0025] By encoding each frame of at least one acquired frame before encoding the current frame image and obtaining the encoding time of each frame image, the encoding time of each frame image can be accurately obtained, and then the maximum transmission time window can be obtained based on the encoding time of each frame image.

[0026] In one embodiment, before acquiring the current frame image and encoding the current frame image, the method further includes:

[0027] The receiving end device receives the average decoding time and average display time of each frame of the image sent by the receiving end device. The average decoding time and average display time are obtained by the receiving end device after receiving the at least one frame of the image, decoding and displaying each frame of the at least one frame of the image.

[0028] By receiving the average decoding time and average display time of each frame of image sent by the receiving device before encoding the current frame image, the maximum transmission time window can be obtained based on the decoding time and average display time of each frame image.

[0029] In one embodiment, obtaining the predicted transmission time window of the current frame image based on the first bitstream and the current predicted bandwidth includes:

[0030] Wp = (P / B) * 1000, where Wp is the predicted transmission time window, P is the number of bytes in the first bitstream, and B is the current predicted bandwidth.

[0031] The predicted transmission time window can be accurately calculated using the above formula.

[0032] In one embodiment, obtaining the maximum transmission time window based on the average encoding time, average decoding time, average display time, and preset maximum tolerable delay per frame includes:

[0033] Wr = Tn – E – D – S, where Wr is the maximum transmission time window, Tn is the preset maximum tolerable delay, E is the average encoding time, D is the average decoding time, and S is the average display time.

[0034] The maximum transmission time window can be accurately calculated using the above formula.

[0035] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0036] The current frame image acquisition module is used to acquire the current frame image and encode the current frame image to generate a first bitstream, wherein the number of bytes in the first bitstream is a preset number of bytes;

[0037] The prediction transmission time window generation module is used to obtain the prediction transmission time window of the current frame image based on the first bitstream and the current prediction bandwidth.

[0038] The maximum transmission time window generation module is used to obtain the maximum transmission time window based on the average encoding time, average decoding time, average display time, and preset maximum tolerable delay of each frame of image.

[0039] The first preset step execution module is configured to execute a first preset step if the predicted transmission time window is greater than the maximum transmission time window. The first preset step includes:

[0040] Discard the current frame image;

[0041] The next frame image is acquired and encoded to generate a second bitstream. The number of bytes in the second bitstream is the number of bytes after the number of bytes in the first bitstream is reduced according to a preset first algorithm.

[0042] The next frame image is used as the new current frame image, and the second bitstream is used as the new first bitstream;

[0043] A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is greater than the maximum transmission time window.

[0044] In one embodiment, the apparatus further includes:

[0045] The second preset step execution module is used to execute a second preset step if the predicted transmission time window is less than the maximum transmission time window. The second preset step includes:

[0046] Send the first bitstream to the receiving device;

[0047] The next frame image is acquired and encoded to generate a third bitstream. The number of bytes in the third bitstream is the number of bytes after the number of bytes in the first bitstream is increased according to a preset second algorithm.

[0048] The next frame image is used as the new current frame image, and the third bitstream is used as the new first bitstream;

[0049] A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is less than the maximum transmission time window.

[0050] In one embodiment, the apparatus further includes:

[0051] The average encoding time acquisition module is used for:

[0052] Acquire at least one frame of image;

[0053] Encode each frame of the at least one image and obtain the encoding time of each frame;

[0054] After encoding each frame of the at least one image, the at least one image is sent to the receiving device, and the average encoding time of each frame is obtained based on the encoding time of each frame.

[0055] In one embodiment, the apparatus further includes:

[0056] The average decoding time receiving module is used to receive the average decoding time and average display time of each frame of the image sent by the receiving device. The average decoding time and the average display time are obtained by the receiving device after receiving the at least one frame of the image, decoding and displaying each frame of the at least one frame of the image.

[0057] In one embodiment, the predicted transmission time window generation module is used for:

[0058] Wp = (P / B) * 1000, where Wp is the predicted transmission time window, P is the number of bytes in the first bitstream, and B is the current predicted bandwidth.

[0059] In one embodiment, the maximum transmission time window generation module is used for:

[0060] Wr = Tn – E – D – S, where Wr is the maximum transmission time window, Tn is the preset maximum tolerable delay, E is the average encoding time, D is the average decoding time, and S is the average display time.

[0061] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device including a processor and a memory, the memory storing at least one computer instruction, the instruction being loaded and executed by the processor to perform the steps performed in the image processing method described in any one of the first aspects.

[0062] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing at least one computer instruction, the instruction being loaded and executed by a processor to perform the steps performed in the image processing method described in any one of the first aspects.

[0063] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0065] Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this disclosure;

[0066] Figure 2 This is a flowchart of an image processing method provided in an embodiment of the present disclosure. Figure 1 ;

[0067] Figure 3 This is a schematic diagram of a cloud desktop system provided in an embodiment of this disclosure;

[0068] Figure 4 This is a flowchart of an image processing method provided in an embodiment of the present disclosure. Figure 2 ;

[0069] Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure. Figure 1 ;

[0070] Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure. Figure 2 ;

[0071] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0073] Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this disclosure. For example... Figure 1 As shown, the system includes a transmitting device 101 and a receiving device 102. The transmitting device 101 and the receiving device 102 can communicate with each other, and the transmitting device 101 can transmit acquired images to the receiving device 102 via a network. This image processing system can be applied to cloud desktop systems or other image transmission scenarios; this embodiment does not impose any limitations.

[0074] When this image processing system is applied to a cloud desktop system, the source device in the server can act as the image sending device, and the client device can act as the receiving device. The source device acquires the current frame image (i.e., the current display screen of the cloud desktop) and encodes the current frame image to generate a first bitstream, the number of bytes in the first bitstream being a preset number of bytes; then, based on the first bitstream and the current predicted bandwidth of the network, a predicted transmission time window for the current frame image is obtained; and based on the average encoding time, average decoding time, average display time, and preset maximum tolerable delay for each frame image (i.e., each frame of the cloud desktop display screen), a maximum transmission time window is obtained; if the predicted transmission time window is greater than the maximum transmission time window, then a preset step is executed, the first preset step including: 1. Discarding the current frame image; 2. Acquiring the next frame image (i.e., the next frame of the cloud desktop display screen) and encoding the next frame image to generate a second bitstream, the number of bytes in the second bitstream being the number of bytes in the first bitstream reduced according to a preset algorithm; 3. Using the next frame image as the new current frame image and using the second bitstream as the new first bitstream. The first bitstream is sent to the client device when the predicted transmission time window is less than or equal to the maximum transmission time window.

[0075] The image processing system provided in this disclosure can adjust the number of bytes in the encoded image bitstream according to the current predicted bandwidth of the network, and send the encoded image bitstream to the receiving device when the predicted transmission time window of the image is less than or equal to the maximum transmission time window. This avoids the problem that users will experience obvious lag or even be unable to use the cloud desktop normally under network conditions with insufficient bandwidth and high latency, thus improving the user experience.

[0076] The following is combined with Figure 2 The embodiments provide a further detailed explanation of how the image processing system provided in this disclosure performs image processing. Figure 2 This is a flowchart of an image processing method provided in an embodiment of this disclosure. Figure 2 As shown, the method includes:

[0077] S201. Obtain the current frame image and encode the current frame image to generate a first bitstream, the number of bytes in the first bitstream being a preset number of bytes.

[0078] In this embodiment, before acquiring the current frame image, at least one frame image is acquired first; then each frame image in the at least one frame image is encoded and the encoding time of each frame image is acquired; after encoding each frame image in the at least one frame image, the at least one frame image is sent to the receiving end device and the average encoding time of each frame image is obtained based on the encoding time of each frame image.

[0079] Furthermore, the receiving end device receives the average decoding time and average display time of each frame of the image. The average decoding time and average display time are obtained by the receiving end device after receiving the at least one frame of the image, decoding and displaying each frame of the at least one frame of the image.

[0080] For example, in a cloud desktop system, before acquiring the current display screen, the source device acquires several frames of the display screen transmitted within a preset time period (e.g., 1 second), encodes each frame of the display screen, and obtains the encoding time of each frame. After encoding each frame, the source device sends the display screen to the client device, and the average encoding time of each frame is obtained based on its encoding time. For instance, when the frame rate per second (FPS) is 30, i.e., the FPS period is 1 second, and the number of frames transmitted per second is 30, the encoding time of the first frame is E1, the encoding time of the second frame is E2, ..., the encoding time of the 30th frame is E30, and the average encoding time E = (E1 + E2 + ... + E30) / 30.

[0081] After receiving the several frames of display images, the client device decodes and displays each frame of the several frames of display images, then obtains the decoding time and display time of each frame of the several frames of display images, and then obtains the average decoding time and average display time of each frame of display images based on the decoding time and display time of each frame of display images.

[0082] For example, after receiving 30 frames of display images transmitted by the source device within an FPS cycle, the client device decodes each frame of the 30 frames, obtaining the decoding time of the first frame as D1, the decoding time of the second frame as D2, ..., the decoding time of the 30th frame as D30, and the average decoding time D = (D1 + D2 + ... + D30) / 30. The client device then displays each frame after decoding, obtaining the display time of the first frame as S1, the display time of the second frame as S2, ..., the display time of the 30th frame as S30, and the average decoding time S = (S1 + S2 + ... + S30) / 30. After obtaining this average decoding time and average display time, the client device sends these average decoding times and average display times to the source device.

[0083] For example, after obtaining the average encoding time and receiving the average decoding time and average display time sent by the receiving device, the obtained current frame image is encoded to obtain a first bitstream, the number of bytes in the first bitstream being a preset number of bytes. For example, in this embodiment, after encoding the current frame image, the number of bytes in the first bitstream is 1000 bytes.

[0084] S202. Obtain the predicted transmission time window of the current frame image based on the first bitstream and the current predicted bandwidth.

[0085] In this step, the predicted transmission time window of the current frame image can be calculated using formula (1).

[0086] Wp=(P / B)*1000 (1).

[0087] Where Wp is the predicted transmission time window, P is the number of bytes in the first bitstream, and B is the current predicted bandwidth (unit: Bps). In this embodiment, the current predicted bandwidth B of the network is obtained simultaneously with the acquisition of the current frame image. Any method of predicting bandwidth in the prior art can be used to obtain the current predicted bandwidth of the network, and no specific limitation is made here.

[0088] S203. Obtain the maximum transmission time window based on the average encoding time, average decoding time, average display time, and preset maximum tolerable delay for each frame of image.

[0089] In this step, after obtaining the average encoding time of each frame and receiving the average decoding time and average display time of each frame sent by the receiving device, the maximum transmission time window can be calculated by formula (2).

[0090] Wr=Tn–E–D–S (2).

[0091] Where Wr is the maximum transmission time window, Tn is the preset maximum tolerable delay, E is the average encoding time, D is the average decoding time, and S is the average display time. After the transmitting device obtains the average encoding time E of each frame and receives the average decoding time D and average display time S of each frame sent by the receiving device, the maximum transmission time window Wr can be calculated according to formula (2). Tn represents the maximum tolerable delay of each frame during transmission, that is, the time taken from the start of acquisition of each frame, through encoding, transmission, and decoding, to the final display cannot exceed Tn. Tn is closely related to the network bandwidth. If the network bandwidth is large, the value of Tn can be large; if the network bandwidth is small, the value of Tn can be small.

[0092] S204. If the predicted transmission time window is greater than the maximum transmission time window, then execute the first preset step.

[0093] In this embodiment, the first preset step includes:

[0094] 1. Discard the current frame image;

[0095] 2. Acquire the next frame image and encode the next frame image to generate a second bitstream. The number of bytes in the second bitstream is the number of bytes after reducing the number of bytes in the first bitstream according to a preset first algorithm.

[0096] 3. Take the next frame image as the new current frame image and take the second bitstream as the new first bitstream.

[0097] 4. Based on the new first bitstream and the current predicted bandwidth, obtain the predicted transmission time window and determine whether the predicted transmission time window is greater than the maximum transmission time window.

[0098] For example, when the number of bytes P in the first bitstream of the current frame image is 1000 bytes, the predicted transmission time window Wp obtained based on the number of bytes in the first bitstream and the current predicted bandwidth is 20ms, while the maximum transmission time window Wr is 10ms, then the first preset steps are executed: 1. Discard the current frame image; 2. Obtain the next frame image and encode the next frame image to generate a second bitstream, the number of bytes in the second bitstream is 900 bytes, which is the number of bytes after reducing the number of bytes in the first bitstream from 1000 by 10%; 3. Use the next frame image as the new current frame image and the second bitstream as the new first bitstream; 4. Based on the number of bytes 900 in the new first bitstream and the predicted transmission time window Wp obtained based on the current predicted bandwidth, determine whether the predicted transmission time window is greater than the maximum transmission time window. If so, then the first preset steps are executed repeatedly until the predicted transmission time window is less than or equal to the maximum transmission time window, and the first bitstream is sent to the receiving device so that the receiving device can decode the first bitstream to generate the current frame image.

[0099] In another implementation, if the predicted transmission time window is smaller than the maximum transmission time window, a second preset step is executed, which includes:

[0100] 1. Send the first bitstream to the receiving device;

[0101] 2. Acquire the next frame image and encode the next frame image to generate a third bitstream. The number of bytes in the third bitstream is the number of bytes in the first bitstream after increasing the number of bytes according to the preset second algorithm.

[0102] 3. Take the next frame image as the new current frame image and take the third bitstream as the new first bitstream;

[0103] 4. Based on the new first bitstream and the current predicted bandwidth, obtain the predicted transmission time window and determine whether the predicted transmission time window is less than the maximum transmission time window.

[0104] For example, when the number of bytes P in the first bitstream of the current frame image is 1000 bytes, the predicted transmission time window Wp obtained based on the number of bytes in the first bitstream and the current predicted bandwidth is 20ms, while the maximum transmission time window Wr is 30ms, then the second preset step is executed: 1. Send the first bitstream to the receiving device so that the receiving device can decode the first bitstream to generate the current frame image; 2. Obtain the next frame image and encode the next frame image to generate a third bitstream, the number of bytes in the third bitstream is 1100 bytes, which is the number of bytes after increasing the number of bytes in the first bitstream 1000 by 10%; 3. Use the next frame image as the new current frame image and the third bitstream as the new first bitstream; 4. Based on the number of bytes in the new first bitstream 1100 and the current predicted bandwidth, obtain the predicted transmission time window Wp and determine whether the predicted transmission time window is less than the maximum transmission time window. If so, then repeat the second preset step until the predicted transmission time is equal to the maximum transmission time window.

[0105] The image processing method provided in this disclosure can adjust the number of bytes in the encoded bitstream of the image according to the current predicted bandwidth of the network. After adjusting the number of bytes in the encoded bitstream, the predicted transmission time window of the image is obtained based on the adjusted number of bytes in the bitstream. When the predicted transmission time window of the image is less than or equal to the maximum transmission time window, the encoded bitstream of the image is sent to the receiving device. By adjusting the number of bytes in the encoded bitstream of the image, the transmission latency of the image can be reduced, which can prevent the stuttering phenomenon caused by network congestion in advance. This avoids the problem that users will experience obvious stuttering or even be unable to use the cloud desktop normally under network conditions with insufficient bandwidth and high latency, thus improving the user experience.

[0106] The image processing method provided in the embodiments of this disclosure will be further described in detail below.

[0107] First, a maximum tolerable delay Tn is preset. The maximum tolerable delay Tn is defined as the time that cannot be exceeded from the start of image capture at the source end, through encoding, transmission, and decoding, to the final display.

[0108] The network bandwidth is estimated using the bandwidth prediction module, resulting in a predicted bandwidth B (unit: Bps).

[0109] The statistics module calculates the average encoding time E (in milliseconds) for all frames per second.

[0110] Average encoding time refers to the total time spent encoding N frames of images transmitted within a period T, divided by N.

[0111] For example, in a cloud desktop, 30 frames are transmitted in each FPS cycle (usually 1 second). The encoding time of the first frame is E1, the encoding time of the second frame is E2, and the encoding time of the 30th frame is E30. Then the average encoding time E = (E1 + E2 + ... + E30) / 30.

[0112] Using the method described above, calculate the average decoding time D (in milliseconds) for all frames per second.

[0113] Average decoding time refers to the total time spent decoding N frames of images transmitted within a period T (e.g., each FPS period mentioned above), divided by N.

[0114] Using the method described above, the average display time S (in milliseconds) of all frames within the statistical period T is calculated.

[0115] Average display time refers to the total time taken to display N frames of images transmitted within a period T (e.g., each FPS period mentioned above), divided by N.

[0116] For each frame, the size of the bitstream output after encoding by the encoder is defined as the output bitstream P (unit: bytes).

[0117] Define (P / B)*1000 as the predicted transmission time window Wp, and Tn–E–DS as the maximum transmission time window Wr.

[0118] When (P / B)*1000>(Tn–E–D–S), the current frame is actively discarded at the encoding end, and the encoder output bitstream is dynamically reduced (i.e., the resolution is reduced), a new frame is regenerated, and (P / B)*1000<=(Tn–E–DS) is satisfied before transmission.

[0119] When (P / B)*1000<(Tn–E–DS), the encoder output bitstream P is dynamically increased (i.e., the clarity is improved) starting from the next frame, and (P / B)*1000<=(Tn–E–D–S) is satisfied before transmission.

[0120] Dynamically reducing or increasing refers to adjusting the encoder's output bitstream parameters proportionally, such as based on 10% of the currently used output bitstream size, and then resetting the encoder.

[0121] When Wp is less than Wr, the image can be transmitted smoothly. Furthermore, when Wp is less than Wr, the encoder will gradually increase the output bitrate to improve image clarity. When Wp is greater than Wr, adjusting the encoder's output bitrate can proactively prevent packet loss and stuttering caused by network congestion.

[0122] This solution dynamically calculates whether each encoded frame can reach the client within the maximum tolerable latency based on the predicted bandwidth. If not, it actively drops frames and adjusts the encoder bitstream to meet the latency requirements. Conversely, while meeting the latency requirements, it dynamically adjusts the encoded bitstream to improve image quality and achieve a better user experience.

[0123] The invention focuses on: determining whether (P / B)*1000>(Tn–E–D–S) is true, and dynamically adjusting P.

[0124] The application scenarios of this invention can be, for example, as follows: Figure 3 The cloud desktop system shown consists of two main parts: the source server and the client server. The source server generally refers to... Figure 3 The acquisition module and encoder are included in the middle. The client generally refers to a terminal device or a software system consisting of a decoder and a display module.

[0125] The main workflow of this solution is as follows: Figure 4 As shown:

[0126] Step 1: The encoder, decoder, and display module periodically calculate the average encoding time E, average decoding time D, and average display time S, respectively.

[0127] This periodicity is typically one second.

[0128] Step 2: Predict the bandwidth in real time at the source end and output the predicted bandwidth value B;

[0129] Step 3: At the source end, each frame after acquisition is encoded by the encoder to obtain the bitstream size P;

[0130] Step four: On the source end, determine the result using the formula (P / B)*1000>(Tn–E–D–S). If the formula calculates the result, proceed to either step five or step six.

[0131] Step 5: If the formula does not hold true, it means that the current bitstream size meets the transmission delay in the current scenario. The data is then directly handed over to the transmission module for transmission. At the same time, the encoder is notified to increase the output bitstream, which means improving the clarity.

[0132] This improvement can vary depending on the encoding algorithm, for example, an improvement of 10% each time;

[0133] Step six: If the formula is true, it means that the current bitstream size does not meet the transmission delay in the current scenario. In this case, the current bitstream is discarded, the encoder is notified to reduce the output bitstream, i.e., reduce the resolution, and then the process returns to step three and repeats.

[0134] Based on the above Figure 2 and Figure 4The image processing methods described in the corresponding embodiments are described below as device embodiments of this disclosure, which can be used to execute the method embodiments of this disclosure.

[0135] Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure. Figure 5 As shown, the device 50 includes:

[0136] The current frame image acquisition module 501 is used to acquire the current frame image and encode the current frame image to generate a first bitstream, wherein the number of bytes in the first bitstream is a preset number of bytes;

[0137] The prediction transmission time window generation module 502 is used to obtain the prediction transmission time window of the current frame image based on the first bitstream and the current prediction bandwidth.

[0138] The maximum transmission time window generation module 503 is used to obtain the maximum transmission time window based on the average encoding time, average decoding time, average display time of each frame of image, and preset maximum tolerable delay.

[0139] The first preset step execution module 504 is used to execute a first preset step if the predicted transmission time window is greater than the maximum transmission time window. The first preset step includes:

[0140] Discard the current frame image;

[0141] The next frame image is acquired and encoded to generate a second bitstream. The number of bytes in the second bitstream is the number of bytes after the number of bytes in the first bitstream is reduced according to a preset first algorithm.

[0142] The next frame image is used as the new current frame image, and the second bitstream is used as the new first bitstream;

[0143] A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is greater than the maximum transmission time window.

[0144] In one embodiment, such as Figure 6 As shown, the device 50 further includes:

[0145] The second preset step execution module 505 is used to execute a second preset step if the predicted transmission time window is less than the maximum transmission time window. The second preset step includes:

[0146] Send the first bitstream to the receiving device;

[0147] The next frame image is acquired and encoded to generate a third bitstream. The number of bytes in the third bitstream is the number of bytes after the number of bytes in the first bitstream is increased according to a preset second algorithm.

[0148] The next frame image is used as the new current frame image, and the third bitstream is used as the new first bitstream;

[0149] A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is less than the maximum transmission time window.

[0150] In one embodiment, the device 50 further includes:

[0151] The average encoding time acquisition module 506 is used for:

[0152] Acquire at least one frame of image;

[0153] Encode each frame of the at least one image and obtain the encoding time of each frame;

[0154] After encoding each frame of the at least one image, the at least one image is sent to the receiving device, and the average encoding time of each frame is obtained based on the encoding time of each frame.

[0155] In one embodiment, the device 50 further includes:

[0156] The average decoding time receiving module 507 is used to receive the average decoding time and average display time of each frame of the image sent by the receiving device. The average decoding time and the average display time are obtained by the receiving device after receiving the at least one frame of the image, decoding and displaying each frame of the at least one frame of the image.

[0157] In one embodiment, the predicted transmission time window generation module 502 is used for:

[0158] Wp = (P / B) * 1000, where Wp is the predicted transmission time window, P is the number of bytes in the first bitstream, and B is the current predicted bandwidth.

[0159] In one embodiment, the maximum transmission time window generation module 503 is used for:

[0160] Wr = Tn – E – D – S, where Wr is the maximum transmission time window, Tn is the preset maximum tolerable delay, E is the average encoding time, D is the average decoding time, and S is the average display time.

[0161] The image processing apparatus provided in this disclosure can be implemented and its technical effects can be found in the above description. Figure 2 and Figure 4 Examples are not described in detail here.

[0162] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 7 As shown, this electronic device 70 includes a processor and a memory, wherein the memory stores at least one computer instruction, which is loaded and executed by the processor to achieve [the desired result]. Figure 2 and Figure 4 The steps performed in the image processing method described in the corresponding embodiments.

[0163] Based on the above Figure 2 and Figure 4 In addition to the image processing method described in the corresponding embodiments, this disclosure also provides a computer-readable storage medium. For example, a non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, magnetic tape, a floppy disk, or an optical data storage device. This storage medium stores computer instructions for executing the above-described methods. Figure 2 and Figure 4 The image processing methods described in the corresponding embodiments will not be repeated here.

[0164] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0165] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

Claims

1. An image processing method, characterized in that, include: The current frame image is acquired and encoded to generate a first bitstream, wherein the number of bytes in the first bitstream is a preset number of bytes; The predicted transmission time window of the current frame image is obtained based on the first bitstream and the current predicted bandwidth; The maximum transmission time window is obtained based on the average encoding time, average decoding time, average display time, and preset maximum tolerable delay for each frame of image. If the predicted transmission time window is greater than the maximum transmission time window, then a first preset step is executed, the first preset step including: Discard the current frame image; The next frame image is acquired and encoded to generate a second bitstream. The number of bytes in the second bitstream is the number of bytes after the number of bytes in the first bitstream is reduced according to a preset first algorithm. The next frame image is used as the new current frame image, and the second bitstream is used as the new first bitstream; A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is greater than the maximum transmission time window. The method for obtaining the predicted transmission time window of the current frame image based on the first bitstream and the current predicted bandwidth includes: Wp=(P / B) * 1000, where Wp is the predicted transmission time window, P is the number of bytes in the first bitstream, and B is the current predicted bandwidth; The method of obtaining the maximum transmission time window based on the average encoding time, average decoding time, average display time, and preset maximum tolerable delay for each frame of image includes: Wr = Tn – E – D – S, where Wr is the maximum transmission time window, Tn is the preset maximum tolerable delay, E is the average encoding time, D is the average decoding time, and S is the average display time.

2. The method according to claim 1, characterized in that, The method further includes: If the predicted transmission time window is smaller than the maximum transmission time window, then a second preset step is executed, the second preset step including: Send the first bitstream to the receiving device; The next frame image is acquired and encoded to generate a third bitstream. The number of bytes in the third bitstream is the number of bytes after the number of bytes in the first bitstream is increased according to a preset second algorithm. The next frame image is used as the new current frame image, and the third bitstream is used as the new first bitstream; A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is less than the maximum transmission time window.

3. The method according to claim 1, characterized in that, Before acquiring the current frame image and encoding the current frame image, the method further includes: Acquire at least one frame of image; Encode each frame of the at least one image and obtain the encoding time of each frame; After encoding each frame of the at least one image, the at least one image is sent to the receiving device, and the average encoding time of each frame is obtained based on the encoding time of each frame.

4. The method according to claim 3, characterized in that, Before acquiring the current frame image and encoding the current frame image, the method further includes: The receiving end device receives the average decoding time and average display time of each frame of the image sent by the receiving end device. The average decoding time and average display time are obtained by the receiving end device after receiving the at least one frame of the image, decoding and displaying each frame of the at least one frame of the image.

5. An image processing apparatus, characterized in that, include: The current frame image acquisition module is used to acquire the current frame image and encode the current frame image to generate a first bitstream, wherein the number of bytes in the first bitstream is a preset number of bytes; The prediction transmission time window generation module is used to obtain the prediction transmission time window of the current frame image based on the first bitstream and the current prediction bandwidth. The maximum transmission time window generation module is used to obtain the maximum transmission time window based on the average encoding time, average decoding time, average display time, and preset maximum tolerable delay of each frame of image. The first preset step execution module is configured to execute a first preset step if the predicted transmission time window is greater than the maximum transmission time window. The first preset step includes: Discard the current frame image; The next frame image is acquired and encoded to generate a second bitstream. The number of bytes in the second bitstream is the number of bytes after the number of bytes in the first bitstream is reduced according to a preset first algorithm. The next frame image is used as the new current frame image, and the second bitstream is used as the new first bitstream; A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is greater than the maximum transmission time window. The predicted transmission time window generation module is used for: Wp = (P / B) * 1000, where Wp is the predicted transmission time window, P is the number of bytes in the first bitstream, and B is the current predicted bandwidth; The maximum transmission time window generation module is used for: Wr=Tn – E – D – S, where Wr is the maximum transmission time window, Tn is the preset maximum tolerable delay, E is the average encoding time, D is the average decoding time, and S is the average display time.

6. The apparatus according to claim 5, characterized in that, The device further includes: The second preset step execution module is used to execute a second preset step if the predicted transmission time window is less than the maximum transmission time window. The second preset step includes: Send the first bitstream to the receiving device; The next frame image is acquired and encoded to generate a third bitstream. The number of bytes in the third bitstream is the number of bytes after the number of bytes in the first bitstream is increased according to a preset second algorithm. The next frame image is used as the new current frame image, and the third bitstream is used as the new first bitstream; A new predicted transmission time window is obtained based on the new first bitstream and the current predicted bandwidth, and it is determined whether the new predicted transmission time window is less than the maximum transmission time window.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer instruction, which is loaded and executed by the processor to perform the steps of the image processing method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer instruction, which is loaded and executed by a processor to perform the steps in the image processing method according to any one of claims 1 to 4.

Images