Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium; the application can determine the current image processing stage, which includes an image acquisition stage, an image rendering stage or an image display stage; obtain a time-consuming parameter matched with the image processing stage, the time-consuming parameter including at least two of the following: acquisition time, rendering time and display time; based on the time-consuming parameter, determine the advance time corresponding to the next image processing stage of the image processing stage, the next image processing stage including the image rendering stage, the image display stage or the memory release stage; when it is detected that the image processing stage has been advanced for the advance time, trigger the synchronous execution of the next image processing stage. In the application, the advance time corresponding to each stage of image processing is determined, so that each stage of image processing is partially parallel. Thus, the scheme can reduce the picture delay.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In electronic devices, displaying images generally follows a pattern: each Hsync (horizontal synchronization signal) displays one line of data, and each Vsync (vertical synchronization signal) displays a complete frame of the image. Furthermore, for each frame to be displayed, the display module reads data from the frame buffer (video memory) around the time the Vsync signal arrives (different chips may have a line prefetching function). Once display is started, the image data in the frame buffer is displayed line by line, segment by segment.

[0003] However, existing technologies may suffer from significant image latency. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and storage medium that can reduce screen latency.

[0005] This application provides an image processing method, including:

[0006] Determine the current image processing stage, which may include image acquisition, image rendering, or image display.

[0007] Obtain time-consuming parameters that match the image processing stage. These time-consuming parameters include at least two of the following: acquisition time, rendering time, and display time.

[0008] Based on the time consumption parameter, the advance time corresponding to the next image processing stage is determined. The next image processing stage includes the image rendering stage, the image display stage, or the memory release stage.

[0009] When it is detected that the image processing stage has been performed ahead of schedule, the next image processing stage is triggered to be executed synchronously.

[0010] This application also provides an image processing apparatus, comprising:

[0011] The determining unit is used to determine the current image processing stage, which may include an image acquisition stage, an image rendering stage, or an image display stage.

[0012] The acquisition unit is used to acquire time-consuming parameters that match the image processing stage. The time-consuming parameters include at least two of the following: acquisition time, rendering time, and display time.

[0013] The determining unit is also used to determine the advance time corresponding to the next image processing stage based on the time consumption parameter. The next image processing stage includes an image rendering stage, an image display stage, or a memory release stage.

[0014] The triggering unit is used to trigger the synchronous execution of the next image processing stage when it is detected that the image processing stage has been advanced by a certain amount of time.

[0015] In some embodiments, determining the lead time for the next image processing stage based on a time consumption parameter includes:

[0016] Get the image resolution;

[0017] Determine the safe amount of memory read / write operations corresponding to the next image processing stage;

[0018] Based on the time consumption parameters, the safe amount of memory read / write corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined.

[0019] In some embodiments, the image processing stage is an image acquisition stage, and the next image processing stage is an image rendering stage. The time consumption parameters include acquisition time and rendering time.

[0020] Based on the time consumption parameters, the safe amount of memory read / write operations corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, including:

[0021] Determine if the acquisition time is greater than the rendering time;

[0022] When the acquisition time is greater than the rendering time, the advance time for the image rendering stage is determined based on the acquisition time, rendering time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0023] When the acquisition time is no greater than the rendering time, the lead time for the image rendering stage is determined based on the acquisition time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0024] In some embodiments, the image processing stage is the image rendering stage, and the next image processing stage is the image display stage. The time consumption parameters include rendering time and display time.

[0025] Based on the time consumption parameters, the safe amount of memory read / write operations corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, including:

[0026] Determine if the rendering time is greater than the display time;

[0027] When the rendering time is greater than the display time, the advance time for the image display stage is determined based on the rendering time, display time, resolution, and the safe amount of memory read and write corresponding to the image display stage.

[0028] When the rendering time is no greater than the display time, the lead time for the image rendering stage is determined based on the rendering time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0029] In some embodiments, the image processing stage is the image display stage, the next image processing stage is the memory release stage, and the time consumption parameters include rendering time and display time.

[0030] Based on the time consumption parameters, the safe amount of memory read / write operations corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, including:

[0031] Determine if the rendering time is greater than the display time.

[0032] When the rendering time is no greater than the display time, the advance duration of the memory release stage is determined based on the rendering time, display time, resolution, and the safe amount of memory read and write corresponding to the image display stage.

[0033] When the rendering time is greater than the display time, the advance duration of the memory release phase is determined based on the rendering time, resolution, and the safe amount of memory read and write corresponding to the image display phase.

[0034] In some embodiments, after determining the safe amount of memory read / write corresponding to the next image processing stage, the method further includes:

[0035] Get the image scaling ratio;

[0036] Based on the scaling ratio, determine the scaling factor corresponding to the safe amount of memory read and write for the next image processing stage;

[0037] Based on the scaling factor, the safe amount of memory read and write corresponding to the next image processing stage is scaled to obtain the scaled safe amount of memory read and write corresponding to the next image processing stage.

[0038] Based on the time consumption parameters, the safe amount of memory read / write operations corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, including:

[0039] Based on the time consumption parameters, the scaled safe amount of memory read / write corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined.

[0040] In some embodiments, determining the lead time for the next image processing stage based on a time consumption parameter includes:

[0041] Get the image resolution;

[0042] Based on the image resolution, obtain the safe duration for memory read / write operations corresponding to the next image processing stage;

[0043] Based on the time consumption parameters and the safe time for memory read / write operations corresponding to the next image processing stage, the advance time corresponding to the next image processing stage is determined.

[0044] In some embodiments, it also includes:

[0045] Determine the safe amount of memory read / write for each preset image processing stage;

[0046] Based on the safe amount of memory read and write corresponding to the preset image processing stage, determine the safe duration of memory read and write corresponding to each preset image processing stage at different resolutions;

[0047] Based on the image resolution, obtain the safe duration for memory reads and writes corresponding to the next image processing stage, including:

[0048] From the safe memory read / write durations corresponding to each preset image processing stage at different resolutions, select the safe memory read / write durations corresponding to the next image processing stage at the image resolution.

[0049] In some embodiments, obtaining time-consuming parameters that match the image processing stage includes:

[0050] Determine the current scene;

[0051] From the time consumption parameters corresponding to each preset image processing stage in each display scenario, determine the time consumption parameters that match the image processing stage in the current scenario.

[0052] In some embodiments, before determining the time consumption parameter matching the image processing stage in the current scene from the time consumption parameters corresponding to each preset image processing stage in each display scene, the method further includes:

[0053] Obtain multiple historical time-consuming parameters corresponding to each preset image processing stage in at least one display scenario;

[0054] Based on historical time consumption parameters, determine the time consumption parameters corresponding to each preset image processing stage in at least one display scenario.

[0055] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the image processing methods provided in this application.

[0056] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform steps in any of the image processing methods provided in this application.

[0057] In this application, the current image processing stage and the corresponding time consumption parameters are used. Then, based on the time consumption parameters, the advance duration corresponding to the next image processing stage is determined. When the advance duration of the image processing stage is detected, the next image processing stage is triggered for synchronous execution. That is, the image rendering stage is triggered before the image acquisition stage has finished executing; and / or, the image display stage is triggered before the image rendering stage has finished executing; and / or, the memory release stage is triggered before the image display stage has finished executing. Therefore, the various stages of image processing can be partially parallelized. Thus, this solution can reduce screen latency. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1a This is a flowchart illustrating the image processing method provided in this application;

[0060] Figure 1b This is a schematic diagram illustrating the lead time of the image rendering stage provided in this application;

[0061] Figure 1c This is a schematic diagram illustrating the advance duration of the image display stage provided in this application;

[0062] Figure 1d This is a schematic diagram illustrating the parallel processing of the various image processing stages provided in this application;

[0063] Figure 2 This is a schematic diagram of the structure of an image processing apparatus provided in this application;

[0064] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0065] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] This application provides an image processing method, apparatus, electronic device, and storage medium.

[0067] Specifically, the image processing device can be integrated into an electronic device, such as a terminal or server. The terminal can be a projector, smart TV, laser TV, mobile phone, tablet computer, smart Bluetooth device, laptop computer, desktop computer, etc.; the server can be a single server or a server cluster composed of multiple servers. The electronic device can run various software platforms such as Android, Linux, and embedded systems, and can also run various hardware platforms such as ARM GPU and dedicated graphics DSP. In some embodiments, the server can also be implemented in the form of a terminal. In some embodiments, the image processing device can be integrated into both the terminal and the server, with the terminal and server jointly implementing the image processing method of this application.

[0068] For example, the image processing device can be integrated into a projector, which includes a decoder, a display module, a rendering module, and a memory. The projector can determine the current image processing stage, which may include an image acquisition stage, an image rendering stage, or an image display stage; acquire time-consuming parameters matching the image processing stage, including at least two of the following: acquisition time, rendering time, and display time; based on the time-consuming parameters, determine the advance time corresponding to the next image processing stage, which may include an image rendering stage, an image display stage, or a memory release stage; and when it is detected that the image processing stage has been advanced for the specified time, trigger the synchronous execution of the next image processing stage.

[0069] In this embodiment, based on the current image processing stage of the projector, a time consumption parameter matching the current image processing stage is obtained. Then, based on the time consumption parameter, the advance duration corresponding to the next image processing stage is determined. When it is detected that the advance duration of the image processing stage has been reached, the next image processing stage is triggered for synchronous execution. That is, the image rendering stage is triggered before the image acquisition stage has finished executing; and / or, the image display stage is triggered before the image rendering stage has finished executing; and / or, the memory release stage is triggered before the image display stage has finished executing. Therefore, the various stages of image processing can be partially parallelized. Thus, this solution can reduce screen latency.

[0070] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0071] In this embodiment, an image processing method is provided, such as... Figure 1a As shown, the specific process of this image processing method can be as follows:

[0072] 110. Determine the current image processing stage. Image processing stages include image acquisition, image rendering, or image display.

[0073] The image acquisition stage can be the process by which an electronic device acquires images when a user uses it in daily life. For example, the image acquisition stage can be a decoder decoding each frame of multimedia data such as video streams or local videos; it can also be a decoder decoding each frame of a cloud game; or it can be an application in the electronic device generating each frame of the image. For example, when the electronic device is a projector, the projector can perform image translation, rotation, or trapezoidal transformation.

[0074] The image rendering stage renders images for rendering modules (such as GPUs) in electronic devices.

[0075] The image display stage allows the display module in an electronic device to display the rendered image.

[0076] In some embodiments, before determining the current image processing stage, the following steps may also be included:

[0077] 1) Obtain multiple historical time-consuming parameters corresponding to each preset image processing stage in at least one display scenario.

[0078] The preset image processing stage can include the image acquisition stage, image rendering stage, and image display stage mentioned above; it also includes a memory release stage. Specifically, the memory release stage can release the corresponding memory (frame buffer, video memory) of the display module in the electronic device to receive image data rendered in the image rendering stage.

[0079] "Multiple" can refer to two or more; for example, it can be 100, or a large amount of data such as 100,000 or 200,000.

[0080] Display scenarios can be divided according to the user's actual usage. In some embodiments, multiple platform scenarios can be first divided based on the device parameters of the electronic device, with one platform scenario corresponding to one display scenario. The device parameters are not limited and can be performance parameters of the display module, rendering module, and decoder, etc. For example, the resolution of the electronic device's display module can be 1080P, 2K, 4K, and subsequent high resolutions; the refresh rate of the display module can be 60Hz, 120Hz, 240Hz, and other subsequent high refresh rates, etc. For example, 1080P and 60Hz correspond to one display scenario, 1080P and 120Hz correspond to another display scenario, and so on.

[0081] In some embodiments, based on the platform scenario, the system can also be divided according to the user's application scenario. Each application scenario under each platform scenario corresponds to a display scenario. That is, the time consumption parameters corresponding to electronic devices with different device parameters can be different, the time consumption parameters corresponding to different application scenarios can also be different, and the time consumption parameters of electronic devices with the same device parameters can also be different in different application scenarios. For example, for an electronic device corresponding to a certain device parameter, the scenario of decoding multimedia data such as video streams can be divided into one category of application scenarios, called the ordinary decoding scenario; the scenario of real-time updated interaction such as games that are sensitive to latency can be divided into another category of application scenarios, called the sensitive scenario; and the scenario of applications generating and processing images can be divided into another category of application scenarios, called the APP scenario. Optionally, the standard decoding scenario can be further divided into multiple sub-scenes based on the source encoding format, resolution, frame rate, etc. For example, it can be divided into sub-scenes such as H.264 4K 60Hz, H.264 4K 30Hz, H.264 1080P 60Hz, H.264 1080P 30Hz, H.264 720P 60Hz, and H.264 720P 30Hz. The same applies to source encoding formats of H.265 and MPEG. Optionally, the sensitive scenario can also be divided into multiple sub-scenes based on the image resolution. That is, each sub-scene of each application scenario within each platform scenario corresponds to a separate display scenario.

[0082] Historical time consumption parameters can include historical acquisition time and historical rendering time. Specifically, acquiring multiple historical acquisition times for at least one display scenario can refer to sampling the time required for electronic devices with identical device parameters to acquire one frame of image in each application scenario, thus obtaining the historical rendering times for electronic devices with various device parameters in each application scenario. Similarly, acquiring multiple historical rendering times for at least one display scenario can refer to sampling the time required for electronic devices with identical device parameters to render one frame of image in each application scenario, thus obtaining the historical rendering times for electronic devices with various device parameters in each application scenario.

[0083] 2) Based on historical time consumption parameters, determine the time consumption parameters corresponding to each preset image processing stage in at least one display scenario.

[0084] The time consumption parameters include at least two of the following: acquisition time, rendering time, and display time. Acquisition time can be used to represent the time required to acquire one frame of an image; rendering time can be used to represent the time required for the GPU to render one frame of an image; display time can be used to represent the duration required for the display module to fully display one frame of an image. The display time can be determined by the performance parameters of the display module. For example, the display time corresponding to a refresh rate of 60Hz can be 1000 / 60ms (described below as 16.67ms).

[0085] In some embodiments, the maximum or minimum value among historical time-consuming parameters can be selected as the time-consuming parameter; alternatively, the average value of historical time-consuming parameters can be calculated to obtain the time-consuming parameter, and so on. For example, since the GPU reads the decoded image data from memory in advance, to avoid the GPU reading speed exceeding the decoder's output, it is necessary to match the GPU's fastest speed with the decoder's slowest speed, while ensuring that the image data read by the GPU does not exceed the image data decoded and output by the decoder. Therefore, in the image acquisition stage, the maximum value among historical acquisition times in normal decoding scenarios can be used as the acquisition time in normal decoding scenarios, and the minimum value among historical rendering times in normal decoding scenarios can be used as the rendering time in normal decoding scenarios; the maximum value among historical acquisition times in sensitive scenarios can be used as the acquisition time in sensitive scenarios; the minimum value among historical rendering times in sensitive scenarios can be used as the rendering time in sensitive scenarios, and so on. In the image rendering stage, the maximum value among historical rendering times in normal decoding scenarios can be used as the rendering time in normal decoding scenarios; the maximum value among historical rendering times in sensitive scenarios can be used as the rendering time in sensitive scenarios, and so on. During the image display stage, the minimum historical rendering time in a normal decoding scenario can be used as the rendering time in that scenario; the minimum historical rendering time in a sensitive scenario can be used as the rendering time in that scenario, and so on.

[0086] Optionally, the time consumption parameters corresponding to each preset image processing stage under each display scenario can be stored in the memory of electronic devices with different device parameters.

[0087] 120. Obtain the time-consuming parameters that match the image processing stage. The time-consuming parameters include at least two of the following: acquisition time, rendering time, and display time.

[0088] In some embodiments, for an electronic device with certain device parameters, the electronic device can determine the current scene; and then filter out the time consumption parameters that match the image processing stage in the current scene from the time consumption parameters corresponding to each preset image processing stage in each display scene. For example, if the current scene is a normal decoding scene and the image processing stage is an image acquisition stage, then the acquisition time and rendering time that match the image acquisition stage are filtered out. As another example, if the current scene is a sensitive scene and the image processing stage is an image rendering stage, then the rendering time and display time that match the image rendering stage are filtered out; and so on. Optionally, the time consumption parameters that match the image processing stage can be retrieved from the memory of the electronic device.

[0089] For example, when a video starts playing, it's determined that the current scene is a normal decoding scene. Obtaining the source encoding format and resolution yields a sub-scene. Then, based on the current sub-scene, the corresponding acquisition time and rendering time are read from memory. The acquisition time is assigned to value 'a', and the rendering time is assigned to value 'b'. If the source is changed subsequently, the source encoding format and resolution need to be acquired again, and then the new values ​​for 'a' and 'b' need to be acquired again.

[0090] For example, in the keystone correction scheme of a projector, the source image is obtained by taking a screenshot. Assuming the frame rate is 60Hz, since the time it takes to display one frame of an image on the screen is 16.67ms, the time to capture one frame of an image can be a fixed 16.67ms, therefore a = 16.67ms.

[0091] 130. Based on the time consumption parameter, determine the lead time corresponding to the next image processing stage. The next image processing stage includes the image rendering stage, the image display stage, or the memory release stage.

[0092] Among them, the next image processing stage after the image acquisition stage can be the image rendering stage, the next image processing stage after the image rendering stage can be the image display stage, and the next image processing stage after the image display stage can be the memory release stage.

[0093] The lead time can be used to represent the duration of processing in the current image processing stage when the next image processing stage starts before the current stage has finished. Optionally, the amount of data processed within the lead time can be called the lead amount. After determining the lead amount, the lead time can be calculated based on the corresponding image processing rate (rendering rate, acquisition rate, etc.).

[0094] In some embodiments, the lead time for the next image processing stage can be determined in real time based on a time consumption parameter, including:

[0095] 1) Obtain the resolution of the image.

[0096] Here, resolution can refer to the resolution of the image processed by the electronic device. For example, it could be the resolution of an image in a video or the resolution of an image in a cloud game.

[0097] 2) Determine the safe amount of memory read / write operations corresponding to the next image processing stage.

[0098] The safe amount of memory read / write corresponding to the next image processing stage can be used to represent the amount of data remaining to be processed in the memory corresponding to the next image processing stage when the current image processing stage ends; or, it can be used to represent the amount of data already processed by the current image processing stage when the next image processing stage starts before the current image processing stage has finished, i.e., the safe amount equals the advance amount. This safe amount may include, but is not limited to: the safe amount of memory read / write corresponding to the image rendering stage (safe amount s1), which stores the image data acquired in the image acquisition stage; the safe amount of memory read / write corresponding to the image display stage (safe amount s2), which can also be called frame buffer and can be used to store the image data rendered in the image rendering stage; and the safe amount of memory read / write corresponding to the memory release stage (safe amount s3), which can be the frame buffer corresponding to safe amount s2. In some embodiments, the safe amount can also be derived from a large amount of data based on device parameters and the scenario, as can be seen in the determination of time consumption parameters, which will not be elaborated here. Optionally, in actual use, the value of the safe amount can be fine-tuned based on the debugging of the electronic device.

[0099] In some embodiments, the scaling ratio of the image can also be obtained; based on the scaling ratio, a scaling factor corresponding to the safe amount of memory read and write corresponding to the next image processing stage can be determined; based on the scaling factor, the safe amount of memory read and write corresponding to the next image processing stage can be scaled to obtain the scaled safe amount of memory read and write corresponding to the next image processing stage.

[0100] 3) Based on the time consumption parameters, the safe amount of memory read / write corresponding to the next image processing stage, and the image resolution, determine the lead time corresponding to the next image processing stage.

[0101] In some embodiments, the lead time for the next image processing stage can be determined based on the time consumption parameter, the scaled safe amount of memory read / write corresponding to the next image processing stage, and the image resolution.

[0102] In some embodiments, the image processing stage is an image acquisition stage, and the next image processing stage is an image rendering stage. The time consumption parameters include acquisition time and rendering time. Based on the time consumption parameters, the safe amount of memory read / write corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, which may include, but is not limited to:

[0103] ① Determine whether the acquisition time is greater than the rendering time.

[0104] ② When the acquisition time is greater than the rendering time, the advance time corresponding to the image rendering stage is determined based on the acquisition time, rendering time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0105] Alternatively, the lead time for the image rendering stage can be calculated using the following formula:

[0106]

[0107] Where r represents the resolution in the vertical direction of the image; X represents the lead time corresponding to the image rendering stage; a represents the acquisition time; b represents the rendering time; and s1 represents the safe amount of memory read / write corresponding to the image rendering stage. The lead time c can then be calculated based on the lead time X, where c = X / (r / a).

[0108] Alternatively, the lead time c corresponding to the image rendering stage can also be calculated using the following formula:

[0109] c = a - b + delta1

[0110] Here, delta1 is used to represent the safe duration of memory read and write corresponding to the image rendering stage, delta1 = s1 / (r / b).

[0111] ③ When the acquisition time is not greater than the rendering time, the advance time corresponding to the image rendering stage is determined based on the acquisition time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0112] Alternatively, the lead time c corresponding to the image rendering stage can be calculated using the following formula:

[0113] c = delta1 = a × (s1 / r)

[0114] For example, taking a typical decoding scenario, where 'a' represents the acquisition time, 'b' represents the rendering time, and the lead time 'c' (in milliseconds) for the image rendering stage is calculated, refer to... Figure 1b As shown:

[0115] a) If a > b, assume the GPU and decoder complete image processing for the previous frame simultaneously. Since different GPUs may prefetch data, and in Tile mode (rendering tiling mode), the GPU will also prefetch the data from the last n rows of memory. The value of n depends on the processing platform; for example, n is 32. Therefore, a safety margin s1 needs to be reserved to ensure that the data read by the GPU does not exceed the data decoded by the decoder, preventing the reading of garbage data. Therefore, c = c' + deltal = a - b + delta1. Here, the value of delta1 corresponds to the safe processing time for the GPU to handle the safety margin s1 of data.

[0116] The principle for determining the value of the safety variable s1 is to ensure that there is a safe interval between the address where the decoder writes data to memory and the address where the GPU reads data from memory. That is, in the case of parallel image acquisition and rendering stages, the GPU reads only valid decoded data. Its value is also related to the amount of data pre-read by the decoder and GPU, and can therefore be derived from a large amount of data based on device parameters and display scenarios. For example, considering that the tiles processed by a typical GPU are 16*16 or 32*32, and given the parallelism of multi-threading, a safety variable of 128 or 256 lines of data can be reserved, ensuring that the address of the decoder's output data is at least 128 to 256 lines away from the address where the GPU reads image data.

[0117] Taking a 1080p video as an example, assuming the decoding time for one frame is 16ms and the GPU can process one frame in as little as 5ms, then a = 16ms, b = 5ms, and therefore c' = 16 - 5 = 11ms. Taking a safety margin s1 of 128 lines of data as an example, assuming the decoder writes X lines of data into memory and then the GPU starts reading data, then according to the 128-line interval requirement above, the ideal situation is that after the decoder outputs the last pixel, the GPU has exactly 128 lines of data left to process. Therefore, we can obtain the formula (1080 - X) / (1080 / 16) = (1080 - 128) / (1080 / 5), from which we can calculate the lead time X = 782.5 lines.

[0118] Where (1080 / 16) is the approximate decoding output rate (acquisition rate) of the decoder, in "lines / ms"; (1080-X) is the number of lines remaining to be decoded after the GPU starts reading and rendering the image; (1080-128) is the total number of lines of data read by the GPU when the decoder finishes decoding; and (1080 / 5) is the GPU's rendering rate, in "lines / ms". The approximate time for the decoder to complete X(782.5) lines is 782.5 / (1080 / 16) = 11.59ms, that is, c = 11.59, therefore delata1 = 0.59ms. That is, the GPU starts reading the current frame image for rendering 11.59ms after the decoder starts decoding the current frame image. In this way, even if the decoder is the slowest and the GPU is the fastest, when the decoder completes decoding the entire frame image, the GPU still has about 128 lines left to render, ensuring that the data read by the GPU will not exceed the data decoded by the decoder. Similarly, a safe amount of 64 lines can be used, resulting in a smaller delta1. However, since the GPU and decoder may not be linearly reading and writing memory, and high-performance GPUs may have high concurrency, the safe amount s1 can be greater than 128 lines to ensure better software stability.

[0119] b) If a <= b, it means the decoder's processing power is no slower than the GPU. Therefore, even if a frame of image is processed simultaneously, the GPU will not outpace the decoder. Thus, the lead time corresponding to the image rendering stage can be equal to the safety factor s1, i.e., c = delta1. The determination of s1 can refer to the derivation process in step a) above. Assuming the safety factor s1 is 128 lines, then delta1 = a * (128 / 1080), meaning that after the decoder processes 128 lines of data, it can trigger the synchronous execution of the image rendering stage.

[0120] It should be noted that delta1 can also be scaled. For example, in normal decoding scenarios and sensitive game scenarios, the image scaling ratio is determined at the initial image creation stage. Therefore, the scaling factor corresponding to delta1 remains unchanged during rendering. The most reasonable scaling factor can be found through multiple debugging and batch verifications. Optionally, the value of the scaling factor can also consider the influence of the image scaling direction during GPU rendering. Horizontal scaling usually does not need to consider its impact on the scaling factor because horizontal scaling references horizontal pixels. Even with complex interpolation algorithms, the referenced neighboring pixels are only a few pixels in the vicinity. Therefore, delta1 is usually sufficient for horizontal scaling and does not need to be enlarged. However, vertical scaling references the pixels of neighboring rows, which will inevitably affect the value of delta1. Optionally, in practical applications, the scaling factor calculation can also refer to the GPU's scaling interpolation algorithm in the scene, and then adjust a suitable scaling factor according to the actual application scenario. However, considering that delta1 is always very small, if the image scaling factor in the actual scene is also not much, such as a maximum of 2x or a minimum of 1 / 2, an appropriate factor can be selected based on the results of multiple tests. Optionally, if delta1 provides sufficient safety margin, it may not need further adjustment. However, in real-world scenarios with a large scaling ratio, such as scaling down to 1 / 5, the GPU may experience address jumps when reading rows of data at once (i.e., some rows of data may not be read). Therefore, delta1 may need to be increased. Delta1 can be fine-tuned based on the actual platform and application scenarios.

[0121] In some embodiments, the image processing stage is an image rendering stage, and the next image processing stage is an image display stage. The time consumption parameters include rendering time and display time. Based on the time consumption parameters, the safety margin, and the image resolution, the lead time corresponding to the next image processing stage is determined, which may include, but is not limited to:

[0122] ① Determine whether the rendering time is greater than the display time.

[0123] ② When the rendering time is greater than the display time, the advance time corresponding to the image display stage is determined based on the rendering time, display time, resolution, and the safe amount of memory read and write corresponding to the image display stage.

[0124] Alternatively, the lead time Y can be calculated using the following formula:

[0125]

[0126] Where r represents the vertical resolution of the image; Y represents the number of rows of image data that have been processed during the image rendering stage (advance); b represents the rendering time; h represents the display time; and s2 represents the safe amount of memory read / write operations corresponding to the image display stage. The advance time d corresponding to the image display stage can then be calculated based on this advance Y, where d = Y / (r / b).

[0127] Alternatively, the advance time d can also be calculated using the following formula:

[0128] d = d` + delta2 = b - h + delta2

[0129] Here, delta2 represents the safe duration for memory reads and writes during the display phase.

[0130] ③ When the rendering time is no greater than the display time, the lead time corresponding to the image rendering stage is determined based on the rendering time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0131] For example, taking a 60Hz display module as an example, the time it takes for the display module to fully display one frame of image is approximately 16.67ms. Calculate the lead time d (in milliseconds) corresponding to the image display stage, referring to... Figure 1c As shown:

[0132] a) If b > 16.67ms, assume the GPU and display module complete processing the previous frame image simultaneously. Since in Tile mode, the last m rows of data in memory might be read prematurely, a safety margin s2 needs to be reserved to ensure the display module reads data that does not exceed the GPU-rendered image data, preventing the reading of garbage data. Therefore, d = d` + delta2 = b - 16.67 + delta2. The value of delta2 corresponds to the processing time for the display module to process the safety margin s2 of data. It's important to note that the arrival time of Vsync is not considered above. Because the GPU is clearly in a low-performance state in this scenario, the image will definitely not reach 60Hz output. Vsync maintains a 60Hz cycle. If the Flip time is sent before Vsync arrives, the image can be displayed earlier, reducing latency. Furthermore, by displaying the current frame image processed by the GPU as quickly as possible, the current frame buffer can be freed up for the GPU to render the next new image.

[0133] The principle for determining the value of the safety margin s2 is similar to that of safety margin s1. Its principle is to ensure that there is a safe interval between the address of the data read from memory by the display module and the address of the data already rendered and written by the GPU in memory. This guarantees that in multi-threaded parallel processing, the display module reads only rendered data. Its value is related to the amount of data pre-fetched by the GPU of the electronic device, and therefore can be derived from a large amount of data based on device parameters and the scenario. For example, based on the pre-fetching amount of the display module, a safety margin of 16 / 32 lines can be reserved. The Tile mode of GPU processing, multi-threaded parallelism, and GPU cache buffering can also be considered. Because in Tile mode, the GPU renders 16 / 32 lines at a time, it is necessary to wait for all 16 / 32 lines to be rendered before pushing the address of the current frame image to the display module; otherwise, the display module may read the last line of garbage data. Considering that the GPU also has a cache (which is generally not large), the safety margin s2 can be 64 lines.

[0134] Taking a 1080p video resolution as an example, assuming the GPU processes one frame in at most 20ms, then b = 20ms, d' = 20 - 16.67 = 3.33ms. Similarly, assuming the GPU completes rendering the Y line before pushing the address of the current frame image to the display module, then according to the safety requirement of 64 lines mentioned above, the ideal situation is that after the GPU outputs the last pixel, the display module still has 64 lines of data left to read. Therefore, we can obtain the formula (1080-X) / (1080 / 20) = (1080-64) / (1080 / 16.67), from which we can calculate Y = 233.16.

[0135] Where (1080 / 20) is the GPU's rendering rate, (1080-X) is the number of lines of unprocessed data remaining on the GPU after being sent to the display module, similarly (1080-64) is the total number of lines of data read by the display module when the GPU completes its task, and (1080 / 16.67) is the display rate of the display module. The approximate time for the GPU to complete Y(233.16) line is 233.16 / (1080 / 20) = 4.32ms, i.e., d = 4.32, therefore delta2 = 4.32 - 3.33 = 0.99ms. That is, when delta2 = 0.99ms is selected, the current frame image is sent to the display module for display 4.32ms after the GPU starts rendering the current frame image. Therefore, even if the GPU's rendering speed is the slowest, and Vsync arrives immediately after the current frame image is sent to the display module, when the GPU completes the rendering of the entire frame image, the display module still has 64 lines of data left to display, ensuring that the data read by the display module does not exceed the data rendered by the GPU.

[0136] It's important to note that the above is just an example of determining the values ​​of Delta2 and d. For instance, the duration of the next Vsync can be detected, and d can be determined based on that duration. Since Vsync has a fixed period of 16.67ms, the arrival time of each Vsync can be recorded. If the GPU starts rendering the current frame image data and the next Vsync is still more than 5ms away, the address of the current frame image can be sent to the display module Flip for display immediately after the GPU starts rendering. This is because the display module will only actually read the current frame image when the Vsync arrives 5ms later. Therefore, it's equivalent to giving the display module a lead time of d = 5ms, thus ensuring that the data read by the display module will not exceed the amount of data already rendered by the GPU.

[0137] b) If b <= 1000 / 60ms, it means that the GPU processing speed is not slower than the display module. Even if the same frame of image is processed at the same time, the display module will not flip and display more data than the GPU renders. Therefore, d = delta2. The value of delta2 is determined in the same way as when a <= b in delta1. Assuming that the safe value is 64 lines, then delta2 = b * (64 / 1080). That is, when rendering 64 lines of data in the image rendering stage, the address of the current frame of image is sent to the display module.

[0138] Optionally, the value of delta2 also needs to consider the effect of image scaling. This is because image scaling will affect the amount and method of data read by the display module. Therefore, in practical applications, it is necessary to evaluate whether delta2 should be multiplied by a certain coefficient based on the scaling ratio, according to the actual effect. The points and principles to consider are similar to those for delta1, and will not be repeated here.

[0139] In some embodiments, the image processing stage is an image display stage, and the next image processing stage is a memory release stage. The time consumption parameters include rendering time and display time. Based on the time consumption parameters, the safety margin, and the image resolution, the lead time corresponding to the next image processing stage is determined, which may include, but is not limited to:

[0140] ① Determine whether the rendering time is greater than the display time.

[0141] ② When the rendering time is no greater than the display time, the advance duration corresponding to the memory release stage is determined based on the rendering time, display time, resolution, and the safe amount of memory read and write corresponding to the image display stage.

[0142] ④ When the rendering time is greater than the display time, the advance duration of the memory release stage is determined based on the rendering time, resolution, and the safe amount of memory read and write corresponding to the image display stage.

[0143] For example, taking a 60Hz display module as an example, after one vysnc cycle, the framebuffer of the current frame image needs to be released and returned to the idle frame buffer for receiving the next new frame content; therefore, the lead time e of the memory release phase can be calculated:

[0144] a) If b > 16.67ms, it means the GPU's rendering rate is slower than the display module's display rate, therefore e = delta3. Delta3, similar to delta1, is a safety margin reserved to avoid screen tearing, typically corresponding to the display module's flip display time of a few dozen lines of data (far lower than 16.67ms or b). The principle for determining the value of the safety margin s3 is similar to that of safety margins s1 and s2, and will not be elaborated here.

[0145] b) If b < 16.67ms, then similarly, e` = 16.67 - b. Considering the safety margin delta3, e = e` + delta3 = 16.67 - b + delta3. As long as it is guaranteed that after releasing the buffer, within the same frame, the memory address of the data written by the GPU will not exceed the memory address that the hardware display module is reading, it will not be elaborated further here.

[0146] It should be noted that the value of delta3 can also take into account the scaling factor of the image during GPU rendering. Therefore, in practical applications, delta3 also needs to be multiplied by the scaling factor, just as mentioned above, and will not be repeated here.

[0147] In some embodiments, after obtaining the time consumption parameters corresponding to each preset image processing stage under each display scenario, the safe duration for memory read and write and / or the corresponding advance duration for each preset image processing stage under different resolutions can be calculated and stored in the storage device.

[0148] Determining the safe duration for memory read / write operations corresponding to each preset image processing stage at different resolutions may include: determining the safe amount of memory read / write operations corresponding to each preset image processing stage; and determining the safe duration for memory read / write operations corresponding to each preset image processing stage at different resolutions based on the safe amount of memory read / write operations corresponding to each preset image processing stage. Optionally, the advance duration corresponding to each preset image processing stage may be determined based on the time consumption parameters and the safe duration for memory read / write operations corresponding to each preset image processing stage.

[0149] For example, after obtaining the acquisition time and rendering time, the value of delta1 or c can be calculated; then the value of delta1 or c can be stored in memory. After obtaining the rendering time and display time, the values ​​of delta2 and / or d can be calculated; then the values ​​of delta2 and / or d can be stored in memory. After obtaining the rendering time and display time, the values ​​of delta3 and / or e can be calculated; then the values ​​of delta3 and / or e can be stored in memory. Specific implementation methods for determining the safety duration or advance duration can refer to the foregoing implementation methods, and will not be repeated here.

[0150] Optionally, for a given display scenario, the lead time for the next image processing stage can be determined based on the time consumption parameter, which may include:

[0151] 1) Obtain the resolution of the image.

[0152] 2) Based on the image resolution, obtain the safe duration for memory read / write operations corresponding to the next image processing stage.

[0153] In some embodiments, the safe memory read / write duration corresponding to the next image processing stage at the image resolution is selected from the safe memory read / write durations corresponding to each preset image processing stage at different resolutions.

[0154] 3) Based on the time consumption parameters and the safe time for memory read and write corresponding to the next image processing stage, determine the advance time corresponding to the next image processing stage.

[0155] Optionally, for a given display scenario, the lead time corresponding to the next image processing stage can be determined based on the time consumption parameter. This can include: obtaining the image resolution; and selecting the lead time corresponding to the next image processing stage at the image resolution from the lead times corresponding to each preset image processing stage at different resolutions.

[0156] In some embodiments, when there is no prefetching in the platform (no prefetching in the GPU, no prefetching in the display module, etc.), the safety amount may not be considered; for example, the advance time corresponding to the image rendering stage can be determined directly based on the acquisition time, rendering time, and image resolution, or the advance time corresponding to the image rendering stage can be determined based on the acquisition time and resolution; the advance time corresponding to the image display stage can be determined based on the rendering time, display time, and image resolution, or the advance time corresponding to the image rendering stage can be determined based on the rendering time and resolution; and so on.

[0157] 140. When it is detected that the image processing stage has been performed ahead of schedule, the next image processing stage is triggered to be executed synchronously.

[0158] In some embodiments, when it is detected that the image acquisition phase has been performed ahead of schedule, the synchronous execution of the image rendering phase is triggered. Since the GPU renders and updates the image data to the rame buffer in advance before the image acquisition phase has finished, even if the image of a later frame times out due to a sudden GPU timeout, there will be no stuttering or other issues caused by missing vsync, thus improving smoothness.

[0159] In some embodiments, when it is detected that the image rendering stage has been advanced by a certain amount of time, the synchronous execution of the image display stage is triggered. By triggering the display module to start displaying earlier, the GPU is not waited for the entire frame of image to be rendered before being pushed to the display module; therefore, the time to display a frame of image is shortened, and image display latency is reduced.

[0160] In some embodiments, when it is detected that the image display phase has been advanced by a certain amount of time, a synchronous memory release phase is triggered. For example, an interrupt function for Vsync can be registered, and the arrival time of Vsync can be timed, denoted as t0. After time t0+e, the frame buffer corresponding to the current frame image is returned to the idle framebuffer, which can be used to receive rendering data for the next frame image. In scenarios that require continuous image rendering, the GPU can be called in advance to render new image data into the framebuffer, reducing the possibility of GPU idleness and improving GPU utilization. At the same time, because the GPU is triggered to perform image rendering processing in advance, the probability of sudden GPU anomalies causing screen stuttering or frame drops is reduced.

[0161] It should be noted that, depending on the application scenario and platform scenario, the next image processing stage can be executed in advance in all image processing stages, such as... Figure 1d The diagram illustrates the parallel processing of image processing stages in this embodiment. Alternatively, the next image processing stage can be executed earlier in some stages. This improves system smoothness without increasing other system resources such as frame buffers, while also ensuring low latency in the display.

[0162] For example, in scenarios where parallel memory read / write rules are not met, such as rendering effects like mirroring or rotation, the last line of data in memory might be read and rendered into video memory first. In such cases, the image display phase can be executed earlier during the image rendering phase, and the memory release phase can be executed earlier during the image display phase. Alternatively, if the mirroring or rotation is handled by the display module, the image rendering phase can be executed earlier during the image acquisition phase, and the memory release phase can be executed earlier during the image display phase. Furthermore, in scenarios where the platform has excellent performance and the GPU processes data quickly, the memory release phase might not be necessary during the image display phase, and latency and smoothness might be good. In such cases, to reduce software complexity, the memory release phase can be omitted during the image display phase.

[0163] In some embodiments, premature execution of the next image processing stage may cause the frame buffer queue to become full due to vsync limitations. For sensitive application scenarios, such as cloud gaming, real-time dropping of some frames can alleviate frame buffer queue congestion and reduce screen latency. For ordinary decoding scenarios, ensuring smooth playback should be prioritized, and frame dropping should be avoided unnecessarily.

[0164] As shown above, this application can determine the advance duration of the next image processing stage based on the current image processing stage and the time consumption parameters matching that stage. When the advance duration of the current image processing stage is detected, the next image processing stage is triggered for synchronous execution. That is, the image rendering stage is triggered before the image acquisition stage has finished executing; and / or, the image display stage is triggered before the image rendering stage has finished executing; and / or, the memory release stage is triggered before the image display stage has finished executing. Therefore, the various stages of image processing can be partially parallelized. Thus, this solution can reduce screen latency.

[0165] The image processing solution provided in this application can be applied to various scenarios involving image acquisition, image rendering, and image display stages. For example, taking the display of images in an app scene as an example, specifically keystone correction in a projector, the keystone correction can be implemented using HW Image Capture IP software. After HW Image Capture IP starts capturing an image, c milliseconds (the capture is not yet finished), it notifies the GPU to process the captured image data, achieving the goal of the GPU rendering the image in advance. Subsequent display and other stages can also be executed in advance. If the chip's computing power in the projector is sufficient, the delay from the start of image capture to image display can be controlled to a minimum of less than one frame. That is, taking a 60Hz output as an example, the delay of software keystone correction can be within 16.67ms. Based on the method described in the above embodiments, further detailed explanation follows.

[0166] The specific process of an image processing method is as follows:

[0167] Because the image actually captured by the HW Image capture IP is the image data output by the display module, it also needs to rely on Vsync to complete the acquisition of one frame of image. Therefore, the acquisition time is fixed at 1000 / 60≈16.67ms. At the same time, the projector chip has good performance, and the GPU keystone transformation time for one image is less than 16.67ms, which corresponds to the case of a>b in step 103 above. Therefore, the image processing method in the keystone correction scenario can include:

[0168] 201. Obtain the HW Image capture IP capture completion notification via interrupt (ISR), with a timer of t1 upon receiving the notification. Since the hardware in the projector operates based on Vsync, the completion of capture means that the next Vsync is about to begin, and a new current frame image will be displayed soon. The capture completion time for this current frame image is approximately t1 + 16.67ms.

[0169] 202. Since the HW Image capture IP sends the image to the frame buffer as a ring buffer, based on the frame buffer address of the previous frame image notified at time t1 (let's call it phy1) and the frame size of the current frame image (let's call it fsize), it can be deduced that after the next vsync starts, the capture IP will write the data of the current frame image to the address phy1 + fsize. Let phy2 = phy1 + fsize (note that the calculation of phy2 here is only an example and does not consider the situation where the ring buffer will loop back to the starting address of the buffer after reaching the highest address. This needs to be considered in actual use, but it will not be discussed in detail here).

[0170] 203. Applying the case of a>b in step 103, at time t1+c (c less than 16.67ms), the GPU is called to render the current frame image corresponding to address phy2 (at this time, the current frame image has not yet finished being captured). Assume that the data rendered by the GPU is stored in the video memory address phy3.

[0171] 204. Applying the case where b < 16.67ms in step 103, the display module is called at time t1 + c + d to execute the image display phase. The current frame image at address phy3 is pushed to the display module, waiting for Vsync to arrive before being displayed. Since c is less than 16.67ms and d is even smaller in this platform scenario, the final c + d < 16.67ms. Therefore, it can be ensured that the image display must occur before the next Vsync after time t1. Thus, the image after keystone correction can be displayed on the display module at the first Vsync after t1 + 16.67ms.

[0172] As shown above, the image rendering stage is triggered before the image capture process has finished; and the image display stage is triggered before the image rendering stage has finished. Therefore, the various stages of keystone correction can be performed in parallel. This solution can reduce screen latency.

[0173] To better implement the above methods, this application also provides an image processing apparatus, which can be specifically integrated into an electronic device. For example, in this embodiment, the method of this application will be described in detail with the image processing apparatus specifically integrated into a projector as an example.

[0174] For example, such as Figure 2 As shown, the image processing apparatus may include a determining unit 301, an acquiring unit 302, and a triggering unit 303, as follows:

[0175] The determining unit 301 is used to determine the current image processing stage, which includes an image acquisition stage, an image rendering stage, or an image display stage.

[0176] The acquisition unit 302 is used to acquire time-consuming parameters that match the image processing stage. The time-consuming parameters include at least two of the following: acquisition time, rendering time, and display time.

[0177] The determining unit 301 is also used to determine the advance time corresponding to the next image processing stage based on the time consumption parameter. The next image processing stage includes an image rendering stage, an image display stage, or a memory release stage.

[0178] The triggering unit 303 is used to trigger the synchronous execution of the next image processing stage when it is detected that the image processing stage has been advanced for a certain period of time.

[0179] In some embodiments, determining the lead time for the next image processing stage based on a time consumption parameter includes:

[0180] Get the image resolution;

[0181] Determine the safe amount of memory read / write operations corresponding to the next image processing stage;

[0182] Based on the time consumption parameters, the safe amount of memory read / write corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined.

[0183] In some embodiments, the image processing stage is an image acquisition stage, and the next image processing stage is an image rendering stage. The time consumption parameters include acquisition time and rendering time.

[0184] Based on the time consumption parameters, the safe amount of memory read / write operations corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, including:

[0185] Determine if the acquisition time is greater than the rendering time;

[0186] When the acquisition time is greater than the rendering time, the advance time for the image rendering stage is determined based on the acquisition time, rendering time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0187] When the acquisition time is no greater than the rendering time, the lead time for the image rendering stage is determined based on the acquisition time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0188] In some embodiments, the image processing stage is the image rendering stage, and the next image processing stage is the image display stage. The time consumption parameters include rendering time and display time.

[0189] Based on the time consumption parameters, the safe amount of memory read / write operations corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, including:

[0190] Determine if the rendering time is greater than the display time;

[0191] When the rendering time is greater than the display time, the advance time for the image display stage is determined based on the rendering time, display time, resolution, and the safe amount of memory read and write corresponding to the image display stage.

[0192] When the rendering time is no greater than the display time, the lead time for the image rendering stage is determined based on the rendering time, resolution, and the safe amount of memory read and write corresponding to the image rendering stage.

[0193] In some embodiments, the image processing stage is the image display stage, the next image processing stage is the memory release stage, and the time consumption parameters include rendering time and display time.

[0194] Based on the time consumption parameters, the safe amount of memory read / write operations corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, including:

[0195] Determine if the rendering time is greater than the display time.

[0196] When the rendering time is no greater than the display time, the advance duration of the memory release stage is determined based on the rendering time, display time, resolution, and the safe amount of memory read and write corresponding to the image display stage.

[0197] When the rendering time is greater than the display time, the advance duration of the memory release phase is determined based on the rendering time, resolution, and the safe amount of memory read and write corresponding to the image display phase.

[0198] In some embodiments, after determining the safe amount of memory read / write corresponding to the next image processing stage, the method further includes:

[0199] Get the image scaling ratio;

[0200] Based on the scaling ratio, determine the scaling factor corresponding to the safe amount of memory read and write for the next image processing stage;

[0201] Based on the scaling factor, the safe amount of memory read and write corresponding to the next image processing stage is scaled to obtain the scaled safe amount of memory read and write corresponding to the next image processing stage.

[0202] Based on the time consumption parameters, the safe amount of memory read / write operations corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined, including:

[0203] Based on the time consumption parameters, the scaled safe amount of memory read / write corresponding to the next image processing stage, and the image resolution, the lead time corresponding to the next image processing stage is determined.

[0204] In some embodiments, determining the lead time for the next image processing stage based on a time consumption parameter includes:

[0205] Get the image resolution;

[0206] Based on the image resolution, obtain the safe duration for memory read / write operations corresponding to the next image processing stage;

[0207] Based on the time consumption parameters and the safe time for memory read / write operations corresponding to the next image processing stage, the advance time corresponding to the next image processing stage is determined.

[0208] In some embodiments, it also includes:

[0209] Determine the safe amount of memory read / write for each preset image processing stage;

[0210] Based on the safe amount of memory read and write corresponding to the preset image processing stage, determine the safe duration of memory read and write corresponding to each preset image processing stage at different resolutions;

[0211] Based on the image resolution, obtain the safe duration for memory reads and writes corresponding to the next image processing stage, including:

[0212] From the safe memory read / write durations corresponding to each preset image processing stage at different resolutions, select the safe memory read / write durations corresponding to the next image processing stage at the image resolution.

[0213] In some embodiments, obtaining time-consuming parameters that match the image processing stage includes:

[0214] Determine the current scene;

[0215] From the time consumption parameters corresponding to each preset image processing stage in each display scenario, determine the time consumption parameters that match the image processing stage in the current scenario.

[0216] In some embodiments, before determining the time consumption parameter matching the image processing stage in the current scene from the time consumption parameters corresponding to each preset image processing stage in each display scene, the method further includes:

[0217] Obtain multiple historical time-consuming parameters corresponding to each preset image processing stage in at least one display scenario;

[0218] Based on historical time consumption parameters, determine the time consumption parameters corresponding to each preset image processing stage in at least one display scenario.

[0219] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0220] As can be seen from the above, the image processing device in this embodiment can determine the advance duration corresponding to the next image processing stage based on the current image processing stage and the time consumption parameter matching the image processing stage; then, based on the time consumption parameter, it determines the advance duration corresponding to the next image processing stage; when it is detected that the image processing stage has been advanced for the specified duration, it triggers the synchronous execution of the next image processing stage. That is, the image rendering stage is triggered in advance before the image acquisition stage has finished executing; and / or, the image display stage is triggered in advance before the image rendering stage has finished executing; and / or, the memory release stage is triggered in advance before the image display stage has finished executing; therefore, the various stages of image processing can be partially parallelized. Thus, this solution can reduce screen latency.

[0221] This application also provides an electronic device. In this embodiment, a projector will be used as an example for detailed description. For example, ... Figure 3 As shown, it illustrates the structural diagram of the electronic device involved in this application, specifically:

[0222] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0223] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0224] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0225] The electronic device also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0226] The electronic device may also include an input module 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0227] The electronic device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0228] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0229] Determine the current image processing stage, which may include image acquisition, image rendering, or image display.

[0230] Obtain time-consuming parameters that match the image processing stage. These time-consuming parameters include at least two of the following: acquisition time, rendering time, and display time.

[0231] Based on the time consumption parameter, the advance time corresponding to the next image processing stage is determined. The next image processing stage includes the image rendering stage, the image display stage, or the memory release stage.

[0232] When it is detected that the image processing stage has been performed ahead of schedule, the next image processing stage is triggered to be executed synchronously.

[0233] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0234] As shown above, the electronic device can determine the advance duration of the next image processing stage based on the current image processing stage and the corresponding time consumption parameters. When the advance duration of the current image processing stage is detected, the next image processing stage is triggered for synchronous execution. That is, the image rendering stage is triggered before the image acquisition stage has finished; and / or, the image display stage is triggered before the image rendering stage has finished; and / or, the memory release stage is triggered before the image display stage has finished. Therefore, the various stages of image processing can be partially parallelized. Thus, this solution can reduce screen latency.

[0235] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0236] Therefore, this application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image processing methods provided in this application. For example, the instructions can execute the following steps:

[0237] Determine the current image processing stage, which may include image acquisition, image rendering, or image display.

[0238] Obtain time-consuming parameters that match the image processing stage. These time-consuming parameters include at least two of the following: acquisition time, rendering time, and display time.

[0239] Based on the time consumption parameter, the advance time corresponding to the next image processing stage is determined. The next image processing stage includes the image rendering stage, the image display stage, or the memory release stage.

[0240] When it is detected that the image processing stage has been performed ahead of schedule, the next image processing stage is triggered for synchronous execution. The storage medium may include: read-only memory (ROM), random access memory (RAM), a hard disk, or an optical disk, etc.

[0241] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method provided in the above embodiments.

[0242] Since the instructions stored in the storage medium can execute the steps of any of the image processing methods provided in this application, the beneficial effects that any of the image processing methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0243] The foregoing has provided a detailed description of an image processing method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized by, The image processing method includes: Determine the current image processing stage, which may include an image acquisition stage, an image rendering stage, or an image display stage. Obtain time-consuming parameters that match the image processing stage, wherein the time-consuming parameters include at least two of the following: acquisition time, rendering time, and display time; Based on the time consumption parameter, the advance time corresponding to the next image processing stage is determined. The next image processing stage includes an image rendering stage, an image display stage, or a memory release stage. When it is detected that the image processing stage has been performed for the specified advance time, the next image processing stage is triggered to be executed synchronously. The image processing stage is the image acquisition stage, the next image processing stage is the image rendering stage, and the time consumption parameters include the acquisition time and the rendering time. The image processing stage is the image rendering stage, the next image processing stage is the image display stage, and the time consumption parameters include rendering time and display time. The image processing stage is the image display stage, the next image processing stage is the memory release stage, and the time consumption parameters include rendering time and display time. The step of determining the advance time corresponding to the next image processing stage based on the time consumption parameter includes: Get the image resolution; Determine the safe amount of memory read / write operations corresponding to the next image processing stage; Based on the time consumption parameter, the safe amount of memory read / write corresponding to the next image processing stage, and the resolution of the image, the advance time corresponding to the next image processing stage is determined.

2. The image processing method of claim 1, wherein, The image processing stage is the image acquisition stage, the next image processing stage is the image rendering stage, and the time consumption parameters include the acquisition time and the rendering time. The step of determining the advance time corresponding to the next image processing stage based on the time consumption parameter, the safe amount of memory read / write corresponding to the next image processing stage, and the resolution of the image includes: Determine whether the acquisition time is greater than the rendering time; When the acquisition time is greater than the rendering time, the advance time corresponding to the image rendering stage is determined based on the acquisition time, the rendering time, the resolution, and the safe amount of memory read and write corresponding to the image rendering stage. When the acquisition time is not greater than the rendering time, the advance duration corresponding to the image rendering stage is determined based on the acquisition time, the resolution, and the safe amount of memory read / write corresponding to the image rendering stage.

3. The image processing method of claim 1, wherein, The image processing stage is the image rendering stage, the next image processing stage is the image display stage, and the time consumption parameters include rendering time and display time. The step of determining the advance time corresponding to the next image processing stage based on the time consumption parameter, the safe amount of memory read / write corresponding to the next image processing stage, and the resolution of the image includes: Determine whether the rendering time is greater than the display time; When the rendering time is greater than the display time, the advance duration corresponding to the image display stage is determined based on the rendering time, the display time, the resolution, and the safe amount of memory read and write corresponding to the image display stage. When the rendering time is not greater than the display time, the advance duration corresponding to the image rendering stage is determined based on the rendering time, the resolution, and the safe amount of memory read / write corresponding to the image rendering stage.

4. The image processing method of claim 1, wherein, The image processing stage is the image display stage, the next image processing stage is the memory release stage, and the time consumption parameters include rendering time and display time. The step of determining the advance time corresponding to the next image processing stage based on the time consumption parameter, the safe amount of memory read / write corresponding to the next image processing stage, and the resolution of the image includes: Determine whether the rendering time is greater than the display time. When the rendering time is not greater than the display time, the advance duration corresponding to the memory release stage is determined based on the rendering time, the display time, the resolution, and the safe amount of memory read and write corresponding to the image display stage. When the rendering time is greater than the display time, the advance duration corresponding to the memory release stage is determined based on the rendering time, the resolution, and the safe amount of memory read / write corresponding to the image display stage.

5. The image processing method of claim 1, wherein, After determining the safe amount of memory read / write corresponding to the next image processing stage, the method further includes: Get the image scaling ratio; Based on the scaling ratio, determine the scaling factor corresponding to the safe amount of memory read / write corresponding to the next image processing stage; Based on the scaling factor, the safe amount of memory read and write corresponding to the next image processing stage is scaled to obtain the scaled safe amount of memory read and write corresponding to the next image processing stage. The step of determining the advance time corresponding to the next image processing stage based on the time consumption parameter, the safe amount of memory read / write corresponding to the next image processing stage, and the resolution of the image includes: Based on the time consumption parameter, the scaled safe amount of memory read / write corresponding to the next image processing stage, and the resolution of the image, the advance time corresponding to the next image processing stage is determined.

6. The image processing method of claim 1, wherein, The step of determining the advance time corresponding to the next image processing stage based on the time consumption parameter includes: Get the image resolution; Based on the resolution of the image, obtain the safe duration for memory read / write corresponding to the next image processing stage; Based on the time consumption parameter and the safe duration for memory read / write corresponding to the next image processing stage, the advance duration corresponding to the next image processing stage is determined.

7. The image processing method of claim 6, wherein, The method further includes: Determine the safe amount of memory read / write for each preset image processing stage; Based on the safe amount of memory read and write corresponding to the preset image processing stage, the safe duration of memory read and write corresponding to each preset image processing stage at different resolutions is determined. The step of obtaining the safe duration for memory read / write operations corresponding to the next image processing stage based on the image resolution includes: From the safe memory read / write durations corresponding to each preset image processing stage at different resolutions, select the safe memory read / write duration corresponding to the next image processing stage at the resolution of the image.

8. The image processing method of any one of claims 1-7, wherein, The step of obtaining the time consumption parameters matching the image processing stage includes: Determine the current scene; From the time consumption parameters corresponding to each preset image processing stage in each display scenario, determine the time consumption parameters that match the image processing stage in the current scenario.

9. The image processing method of claim 8, wherein, Before determining the time consumption parameter matching the image processing stage in the current scene from the time consumption parameters corresponding to each preset image processing stage in each display scene, the method further includes: Obtain multiple historical time-consuming parameters corresponding to each preset image processing stage in at least one display scenario; Based on the historical time consumption parameters, the time consumption parameters corresponding to each preset image processing stage in each of the at least one display scenarios are determined.

10. An image processing apparatus characterized by comprising: include: A determining unit is used to determine the current image processing stage, which includes an image acquisition stage, an image rendering stage, or an image display stage. The acquisition unit is used to acquire time-consuming parameters that match the image processing stage. The time-consuming parameters include at least two of the following: acquisition time, rendering time, and display time. The determining unit is used to determine the advance time corresponding to the next image processing stage based on the time consumption parameter, wherein the next image processing stage includes an image rendering stage, an image display stage, or a memory release stage. The triggering unit is used to trigger the synchronous execution of the next image processing stage when it is detected that the image processing stage has been performed for the specified advance time. The image processing stage is the image acquisition stage, the next image processing stage is the image rendering stage, and the time consumption parameters include the acquisition time and the rendering time. The image processing stage is the image rendering stage, the next image processing stage is the image display stage, and the time consumption parameters include rendering time and display time. The image processing stage is the image display stage, the next image processing stage is the memory release stage, and the time consumption parameters include rendering time and display time. The step of determining the advance time corresponding to the next image processing stage based on the time consumption parameter includes: Get the image resolution; Determine the safe amount of memory read / write operations corresponding to the next image processing stage; Based on the time consumption parameter, the safe amount of memory read / write corresponding to the next image processing stage, and the resolution of the image, the advance time corresponding to the next image processing stage is determined.

11. An electronic device, comprising: The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps of the image processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image processing method according to any one of claims 1 to 9.

Images