Image processing method, device, readable storage medium and program product

Abstract

Embodiments of the present disclosure provide an image processing method, device, readable storage medium and program product, the method comprising: obtaining historical frame loss information of a first processing node processing a first image stream, and historical on-screen frame rate distribution of the first image stream processed by the first processing node; determining an input frame rate of a second processing node according to the historical frame loss information and / or the historical on-screen frame rate distribution, frame loss of an original image stream collected by an image sensor based on the input frame rate; applying a first special effect to each frame of the original image stream of the original image stream after frame loss by the second processing node to obtain a second image stream; and applying a second special effect to multiple frames of the second image stream by the first processing node to obtain the second image stream for preview display.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and more particularly to an image processing method, apparatus, readable storage medium, and program product. Background Technology

[0002] Users can share content using the application, such as recording and sharing videos. When sharing recorded videos, special effects can be used to enrich the visual experience. To avoid affecting the video frame rate, these special effects can be divided into pre-processing effects and post-processing effects.

[0003] It also allows previewing video content with applied effects; however, because pre-processing and post-processing effects take different amounts of time, frame drops can easily occur when applying post-processing effects to multiple image frames. These frame drops waste resources and negatively impact the smoothness of the preview. Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, readable storage medium, and program product to reduce resource waste during previewing and improve the smoothness of the preview screen.

[0005] In a first aspect, embodiments of this disclosure provide an image processing method, comprising: acquiring historical frame loss information of a first processing node processing a first image stream, and historical screen frame rate distribution of the first image stream after processing by the first processing node; determining the input frame rate of a second processing node based on the historical frame loss information and / or the historical screen frame rate distribution, and performing frame loss on an original image stream acquired by an image sensor based on the input frame rate; applying a first effect to each frame of the original image stream after frame loss by the second processing node to obtain a second image stream; and applying a second effect to multiple frames of the second image stream by the first processing node to obtain a second image stream for preview display.

[0006] In a second aspect, embodiments of this disclosure provide an image processing device, comprising: an acquisition unit, configured to acquire historical frame loss information of a first processing node processing a first image stream, and historical screen frame rate distribution of the first image stream after processing by the first processing node; a frame rate determination and frame loss unit, configured to determine the input frame rate of a second processing node based on the historical frame loss information and / or the historical screen frame rate distribution, and to perform frame loss on the original image stream acquired by an image sensor based on the input frame rate; a first special effects processing unit, configured to apply a first special effect to each frame of the original image stream after frame loss by the second processing node to obtain a second image stream; and a second special effects processing unit, configured to apply a second special effect to multiple frames of the second image stream by the first processing node to obtain a second image stream for preview display.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0008] The memory stores computer-executed instructions;

[0009] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the image processing method as described in the first aspect and various possible designs of the first aspect.

[0010] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image processing method described in the first aspect and various possible designs of the first aspect.

[0011] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the image processing method described in the first aspect and various possible designs of the first aspect.

[0012] The image processing method, apparatus, readable storage medium, and program product provided in this disclosure acquire historical frame loss information of a first processing node processing a first image stream, and the historical screen frame rate distribution of the first image stream after processing by the first processing node; determine the input frame rate of a second processing node based on the historical frame loss information and / or the historical screen frame rate distribution; perform frame loss on the original image stream acquired by the image sensor based on the input frame rate; apply a first effect to each frame of the original image stream after frame loss by the second processing node to obtain a second image stream; and apply a second effect to multiple frames of the second image stream by the first processing node to obtain a second image stream for preview display. This achieves dynamic frame loss of the original image stream based on the historical frame loss information and / or the historical screen frame rate distribution of the first processing node, which can reduce the number of images subsequently discarded in the first processing node. Since the original image stream is frame-lossed before the second processing node, the number of images processed by the second processing node can be reduced, thereby reducing the resource occupation of the second processing node and reducing resource waste. The saved resources can be used by the first processing node to apply the first special effect to the image, improve the efficiency of applying the first special effect, increase the on-screen frame rate, and improve the preview effect. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of an image processing method in related technologies;

[0015] Figure 2 Flowchart of the image processing method provided in the embodiments of this disclosure Figure 1 ;

[0016] Figure 3 A schematic diagram illustrating the principle of the image processing method provided in this disclosure;

[0017] Figure 4 Flowchart of the image processing method provided in the embodiments of this disclosure Figure 2 ;

[0018] Figure 5 Flowchart of the image processing method provided in the embodiments of this disclosure Figure 3 ;

[0019] Figure 6 for Figure 3 The diagram shows the workflow of the frame rate control module.

[0020] Figure 7 This is a structural block diagram of an image processing apparatus provided in an embodiment of the present disclosure;

[0021] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0023] Users can use terminal devices to capture videos in real time for video content sharing. Specifically, the terminal device can use its integrated image sensor (such as a camera) to capture image sequences. The image sequence can include multiple image frames arranged chronologically according to the capture time, and then these image frames are used to generate video content. Special effects can be added to the captured image sequences to enrich the visual experience. In order not to affect the frame rate of image capture, special effects can be divided into pre-processing effects and post-processing effects. For example, pre-processing effects and post-processing effects can be processed by different threads.

[0024] Please refer to Figure 1 , Figure 1 This is a schematic diagram of an image processing method in related technologies. For example... Figure 1 As shown, an image sensor 11 can be integrated into the terminal device, and the image sensor 11 can acquire image sequences. The image sequence includes multiple image frames. For each acquired image frame, pre-processing effects 12 can be added in the first thread. Adding pre-processing effects to image frames means applying visual effects directly to each frame image. These effects can change the color, brightness, and contrast of the image, or add effects such as blurring, sharpening, stylized transformations (e.g., cartoon effects), overlaying textures, or lighting effects.

[0025] To achieve better visual effects in the generated video content, special effects can be applied to the captured images. These effects include, but are not limited to, beautification, stickers, and multi-track mixing. Since applying effects to images takes a considerable amount of time, applying all effects to every frame before previewing will result in a low image capture frame rate, leading to insufficient information in the obtained video content and poor continuity in the preview. To ensure that the image sensor's image capture frame rate is not too low, in some application scenarios, shorter-duration effects can be executed as pre-processing effects and image capture control in the first thread, while longer-duration effects can be executed as post-processing effects in the second thread. Pre-processing effects include, but are not limited to, beautification of objects in the image. Post-processing effects include, but are not limited to, adding stickers, adding audio effects, and multi-track mixing. In this disclosure, pre-processing effects are set in the second processing node, and post-processing effects are set in the first processing node.

[0026] To avoid interfering with the camera's continued image acquisition, the image sequence with applied pre-processing effects can be sent to buffer queue 14. Post-processing thread 13 extracts multiple image frames from buffer queue 14 to apply post-processing effects 15. These post-processing effects include using multiple tracks to blend the image sequence with various audio, text, and graphic effects. The image sequence with applied post-processing effects can be previewed and displayed.

[0027] In addition, video files can be generated from image sequences with applied preprocessing effects 17.

[0028] The processing time required to apply pre-processing effects to an image differs from that required to apply post-processing effects. Pre-processing effects take less time than post-processing effects. This can cause the buffer queue to overflow, resulting in dropped frames before the post-processing effects take effect. Since these dropped image frames have undergone pre-processing but are not ultimately previewed, they generate unnecessary resource consumption, consuming system resources and affecting the preview frame rate.

[0029] In this disclosure, the frame rate of the pre-processing effect is determined by the historical frame dropping information and / or the historical preview on-screen rate distribution of the first processing node. Before applying the pre-processing effect, the image stream acquired by the image sensor is subjected to frame dropping processing. By dropping frames in advance, these discarded images do not need to be processed by the pre-processing effect, which reduces the system resource consumption. The resource consumption saved can be used for the application of post-processing effects, which is conducive to improving the preview on-screen rate and thus improving the smoothness of the preset screen.

[0030] Please refer to Figure 2 , Figure 2 Flowchart of the image processing method provided in this disclosure Figure 1 ,like Figure 2 As shown, the method includes the following steps:

[0031] S201: Obtain historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after being processed by the first processing node.

[0032] In this embodiment, the entity executing the image processing method can be a terminal device, such as a terminal device equipped with an image sensor (e.g., a camera). Various applications can run on the terminal device, such as camera applications that control the camera to capture image sequences (videos) and perform various processing on the captured image sequences.

[0033] The terminal device controls the image sensor to acquire images according to user instructions, such as acquiring images of the current environment in real time according to a preset image acquisition frequency, and can also generate video content for preview based on the acquired image sequence.

[0034] In this disclosure, an image stream refers to a sequence of images arranged in chronological order. An image stream can be, for example, a sequence of images acquired by an image sensor, such as image 1, image 2, image 3, ..., image N, where N is an integer greater than 3. Alternatively, it can be multiple image sequences each assigned its own display time information.

[0035] In this embodiment, shorter-duration effects can be executed as pre-processing effects (hereinafter referred to as the first effect) and image acquisition control in the first thread, while longer-duration effects can be executed as post-processing effects (hereinafter referred to as the second effect) in the second thread. Pre-processing effects include, but are not limited to, beautifying objects in the image. Post-processing effects include, but are not limited to, adding stickers, adding audio effects, and multi-track mixing. In this disclosure, pre-processing effects are set in the second effect processing node, and post-processing effects are set in the first effect processing node.

[0036] The raw image stream acquired by the image sensor can be transmitted to the second processing node in the first thread. In the second processing node, a first effect is applied. The raw image with the first effect applied can then be input back to the first processing node, where a second effect is applied. The first thread can determine frame loss information based on one or more frames that have had the first effect applied but have not been input to the first processing node. This frame loss information may include, for example, the total number of images dropped within a preset time period, or the number of images dropped per unit time (frame loss rate).

[0037] Historical frame loss information can include the number of frames lost within a historical period or the historical frame loss rate. The number of frames lost can be an integer greater than or equal to 0, and the historical frame loss rate can be a value greater than or equal to 0 and less than 1. When the number of frames lost within a historical period is 0, or the historical frame loss rate is 0, it can be considered that no frame loss occurred.

[0038] In some implementations, obtaining the historical frame loss information of the first processing node includes:

[0039] Historical frame loss information is determined based on the number of images that applied the first special effect but were not stored in the cache queue; where the cache queue is the cache queue used by the first processing node.

[0040] A buffer queue can be set up on the first processing node to store multiple frames of images output by the second processing node. The buffer queue can output images frame by frame to the first processing node according to a first-in-first-out rule, so that the first processing node can apply the second effect to the images. However, the buffer queue has limited space, and overflow will occur when the output rate of the buffer queue is less than the input rate, resulting in frame dropping.

[0041] In these implementations, the first thread can determine the number of images that have not been input into the buffer queue to determine the frame drop information. This frame drop information may include, for example, the total number of images dropped within a preset time period, or the number of images dropped per unit time (frame drop frequency), etc.

[0042] In these implementations, since the images that enter the cache queue can be applied with a first effect by the first processing node, the historical frame loss information can be accurately obtained in real time by determining the number of historical images that were not stored in the cache queue.

[0043] In this embodiment, the frame dropping information of the image stream can be determined periodically, that is, the images acquired by the image sensor are dynamically dropped in advance to reduce resource consumption. The duration of each period can be set according to the specific application scenario. For example, the duration of each period can be 10 seconds.

[0044] The first image stream here can refer to the image stream processed by the first processing node and the second processing node within a historical period.

[0045] In addition, the first processing node can apply a second effect to the first image stream input therein, and then transmit the image with the second effect applied to the display screen for preview display.

[0046] The on-screen frame rate here refers to the number of images displayed on the screen per unit of time.

[0047] The aforementioned historical frame rate distribution can include, for example, the on-screen frame rate of the image stream at different time points within a period. For instance, within this preset time period, the on-screen frame rate D1 corresponds to the first second, D2 to the second second, D3 to the third second, ..., and the on-screen frame rate DN to the Nth second, etc. N is an integer greater than or equal to 3.

[0048] S202: Determine the input frame rate of the first processing node based on historical frame loss information and / or historical screen frame rate distribution, and perform frame loss on the raw image stream acquired by the image sensor based on the input frame rate.

[0049] After obtaining the historical frame loss information and historical screen frame rate distribution of the first image stream, the input frame rate of the first processing node in the current cycle can be determined based on the aforementioned historical frame loss information and / or historical screen frame rate distribution.

[0050] In some implementations, determining the input frame rate of the first processing node based on historical frame loss information and / or historical screen frame rate distribution includes one of the following:

[0051] The input frame rate is obtained by reducing the historical input frame rate of the first processing node based on the historical frame loss information.

[0052] Based on historical frame loss information and historical screen frame rate information, the input frame rate is obtained by adding the historical input frame rate of the first processing node.

[0053] Based on historical frame loss information and historical screen rate, the historical input frame rate of the first processing node is used as the input frame rate.

[0054] In one example, the input frame rate of the second processing node can be determined based on historical frame loss information. For instance, if the frame loss rate indicated by historical frame loss information is high, it means that the first processing node has been blocked. The historical input frame rate of the second processing node can be reduced to obtain the input frame rate of the second processing node in the current cycle. On the one hand, this can reduce the number of images input to the second processing node, thereby reducing resource consumption. On the other hand, it can alleviate the blockage of the first processing node and help improve the frame loss rate of the first processing node.

[0055] In one example, if the historical frame loss information indicates that no frames were lost, the historical frame rate distribution of the screen is relatively uniform, and many of the screens in the historical frame rate distribution have low on-screen frame rates. The historical input frame rate of the second processing node can be increased to obtain the input frame rate of the second processing node in the current cycle.

[0056] In one example, if historical frame loss information indicates no frame loss, and multiple on-screen frame rates in the historical screen rate distribution are relatively high, the historical input frame rate of the second processing node can be used as the input frame rate for the current cycle.

[0057] In these implementations, the frame rate input to the first processing node can be reduced, the frame rate input to the second processing node can be increased, or the frame rate input to the second processing node can be maintained, based on historical frame loss information, to adapt to the frame loss information of the first processing node and the preview display effect.

[0058] After obtaining the input frame rate, frames can be dropped from the raw image stream acquired by the image sensor. Illustratively, since the image acquisition frequency of the image sensor is fixed, i.e., the number of raw images acquired per second is fixed, the number of images to be dropped per second can be determined based on the input frame rate, and then a strategy for dropping frames from the raw image stream can be determined. Specifically, based on the number of images dropped per second, one or more raw images can be dropped every M frames. Here, M is an integer greater than or equal to 1 and less than the image acquisition frequency. Frames can then be dropped from the raw image stream acquired by the image sensor within the current period according to the determined frame dropping strategy.

[0059] S203: The second processing node applies the first special effect to each frame of the original image stream after frame loss to obtain the second image stream.

[0060] In step S202, after dropping frames from the original image stream, a frame-dropped original image stream is obtained. This frame-dropped original image stream is then input to the second processing node. In the second processing node, a first special effect (i.e., pre-processing effect) is applied to each frame of the original image stream to obtain the second image stream. In other words, frames are dynamically dropped before applying the first special effect to each original image in the original image stream, thereby reducing the probability that images with the first special effect applied will be discarded before the first processing node.

[0061] The second processing node can apply the first special effect to the remaining original images of each frame after frame loss, and the original image stream after passing through the second processing node is used as the second image stream.

[0062] S204: The first processing node applies a second effect to multiple frames of images in the second image stream to obtain a second image stream for preview display.

[0063] In one example, the aforementioned execution entity can input multiple frames of images from the second image stream to the first processing node frame by frame, and the first processing node can apply the second effect to each of the multiple frames of images.

[0064] It is understandable that frame drops may occur when the second effect is applied to multiple frames of the first image at the first processing node. The input frame rate of the second processing node for the next cycle can be determined based on the current frame drop information and the frame rate distribution displayed in the preview.

[0065] After obtaining the second image stream for preview display, the aforementioned execution entity can send the second image stream for preview display to the display unit for preview display.

[0066] In this embodiment, by acquiring historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after processing by the first processing node, the input frame rate of the second processing node is determined based on the historical frame loss information and / or the historical screen frame rate distribution. Frame loss is then performed on the original image stream acquired by the image sensor based on the input frame rate. This achieves dynamic frame loss of the original image stream based on the historical frame loss information and / or the historical screen frame rate distribution of the first processing node. This reduces the number of images subsequently discarded in the first processing node. Since frames are lost before the second processing node, the number of images processed by the second processing node is reduced, thereby reducing the resource consumption of the second processing node and minimizing resource waste. The saved resources can be used by the first processing node to apply a first special effect to the image, improving the efficiency of applying the first special effect, thereby increasing the on-screen frame rate and improving the preview effect.

[0067] Please refer to Figure 3 , Figure 3This is a schematic diagram illustrating the principle of the image processing method provided in this disclosure. An image sensor 301 and a display unit 306 can be configured in the terminal device. The image sensor 301 can acquire images using its inherent image acquisition frequency to obtain a raw image stream. The raw image stream output by the image sensor 301 can be input to the frame rate control module 302 in real time. The frame rate control module 302 can pre-obtain frame loss information of the first image stream processed by the first processing node in the previous cycle. For example, it can obtain the number of images that passed through the second processing node in the previous cycle but were not stored in the buffer queue 304, and determine the frame loss information of the previous cycle based on this number. For example, it can determine the frame loss rate of the previous cycle. The terminal device can also obtain the historical screen frame rate distribution of the image stream output by the first processing node 305 within one or more historical cycles. The aforementioned historical screen frame rate distribution includes the on-screen frame rates corresponding to multiple historical time points.

[0068] The frame rate control module 302 can determine the input frame rate of the second processing node based on historical frame loss information and / or historical screen frame rate distribution. Based on the input frame rate, the original image stream input to it undergoes frame loss processing to obtain an original image stream matching the input frame rate. That is, the number of original images transmitted per second to the second processing node after frame loss is equal to the aforementioned input frame rate. The original image stream after frame loss by the frame rate control module 302 is input to the second processing node 303. The second processing node 303 applies a first effect to each frame of the input original image stream and outputs a second image stream. The second image stream output by the second processing node 303 is then input to the video file generation node 307, where the second image stream is video encoded to generate a video file for storage.

[0069] The second image stream output by the second processing node 303 is input to the buffer queue 304. Specifically, the first processing node 305 can apply a second effect to the previous image input to the buffer queue and then input it to the display unit for preview. It can also retrieve the next image from the head of the buffer queue, apply a first effect, and then send it to the display unit 306 for preview display. Afterward, the image originally stored in the second position in the buffer queue is moved to the head of the queue. The second image stream output by the second processing node is input to the tail of the buffer queue for buffering according to the time sequence. When buffering the second image stream output by the second processing node into the buffer queue, there may be frame drops. The number of images dropped at this time can be recorded for determining the input frame rate to the second processing node in the next cycle.

[0070] pass Figure 3As can be seen, a frame rate control module is set between the second processing node and the image sensor. The frame rate control module determines the input frame rate of the second processing node based on the frame drop information and / or the historical screen frame rate distribution. Based on the input frame rate, the original image stream is dropped before the second processing node, thereby reducing the resource consumption of applying the first effect to the images to be dropped. The saved resources can be used by the first processing node to apply the second effect to the images input to it, which can improve the efficiency of the first processing node in applying the second effect to the images, thereby increasing the preview frame rate and improving the preview effect.

[0071] Please refer to Figure 4 , Figure 4 Flowchart of the image processing method provided in this disclosure Figure 2 ,like Figure 4 As shown, the method includes the following steps:

[0072] S401: Obtain historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after being processed by the first processing node.

[0073] In this embodiment, the entity executing the image processing method can be a terminal device. The specific implementation of step S301 can be found in the description of step S201, and will not be repeated here.

[0074] S402: In response to the historical frame loss information meeting the first preset condition, obtain the historical screen frame rate distribution within the first preset time period, and use the maximum on-screen frame rate in the historical screen frame rate distribution as the input frame rate.

[0075] Historical frame loss information can include the number of frames lost within a historical period or the historical frame loss rate. The number of frames lost can be an integer greater than or equal to 0, and the historical frame loss rate can be a value greater than or equal to 0 and less than 1. When the number of frames lost within a historical period is 0, or the historical frame loss rate is 0, it can be considered that no frame loss occurred.

[0076] The aforementioned first preset condition is a preset condition for reducing the historical input frame rate of the second processing node.

[0077] In some implementations, the first preset condition can be that the historical frame loss rate indicated by the historical frame loss information is greater than or equal to a preset frame loss rate threshold. The preset frame loss rate threshold can be, for example, 2 frames per second. Or, the number of historical frame losses can be greater than a preset number threshold.

[0078] In these implementations, if the historical frame loss rate indicated by the historical frame loss information is greater than or equal to a preset frame loss rate threshold, it can be considered that the frame rate of the original image sent to the second processing node needs to be reduced, that is, the number of frames of the original image input to the second processing node per second is reduced.

[0079] The on-screen frame rate distribution within the first preset time period can, for example, include the on-screen frame rate distribution within one or more historical periods. For instance, it could be the on-screen frame rate distribution within the previous period or the two previous periods.

[0080] To maintain a good preview effect even after reducing the frame rate input to the second processing node, the maximum on-screen frame rate can be determined from the on-screen frame rate distribution within the first preset time period. This maximum on-screen frame rate is then used as the input frame rate for the second processing node.

[0081] S403: Drop frames from the raw image stream acquired by the image sensor based on the input frame rate.

[0082] For example, the number of images to be discarded per second can be determined based on the input frame rate of the second processing node, and then a frame dropping strategy for the multiple frames of raw images acquired per second can be determined. Illustratively, the frame dropping strategy could be, for example, discarding one or more frames of raw images every M frames. M is an integer greater than or equal to 1.

[0083] S404: The second processing node applies the first special effect to each frame of the original image stream after frame loss to obtain the second image stream.

[0084] S405: The first processing node applies a second effect to multiple frames of images in the second image stream to obtain a second image stream for preview display.

[0085] For detailed implementation of steps S404 to S405, please refer to Figure 2 The descriptions of the relevant parts of steps S203 to S204 shown are not repeated here.

[0086] This embodiment describes how, when the historical frame dropping information meets the first preset condition, the maximum on-screen frame rate in the historical screen frame rate distribution within the first preset time period is used as the input frame rate of the second processing node, and then the original image stream is dropped based on the above input frame rate, thereby reducing resource waste and ensuring the smoothness of the preview effect of the image sequence with the second special effect applied.

[0087] Please refer to Figure 5 , Figure 5 Flowchart of the image processing method provided in this disclosure Figure 3 ,like Figure 5 As shown, the method includes the following steps:

[0088] S501: Obtain historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after being processed by the first processing node.

[0089] In this embodiment, the entity executing the image processing method can be a terminal device. The specific implementation of step S501 can be found in the description of step S201, and will not be repeated here.

[0090] S502: In response to the historical frame loss information meeting the second preset condition and the historical input frame rate being less than the maximum on-screen frame rate in the historical screen frame rate information, the historical input frame rate is increased according to the preset frame rate increment to obtain the input frame rate.

[0091] The second preset condition here can be a preset condition for increasing the historical input frame rate of the second processing node. Illustratively, the aforementioned second preset condition could be, for example, that the number of images discarded by the first processing node within a historical period is less than a preset threshold.

[0092] In some implementations, the second preset condition may be that the historical frame loss rate indicated by the historical frame loss information is less than a second preset frame loss rate threshold.

[0093] In these implementations, the aforementioned second preset frame drop rate threshold can be, for example, 2 frames per second.

[0094] If the historical frame drop rate is less than the second preset frame drop rate threshold, and the historical input frame rate of the second processing node is less than the maximum on-screen frame rate in the on-screen frame rate distribution, the historical input frame rate of the second processing node can be increased to obtain the input frame rate for the current period. Specifically, the sum of the preset frame rate increment and the historical input frame rate can be used as the current input frame rate of the second processing node.

[0095] The aforementioned preset frame rate increment can be set according to specific application scenarios, and there are no restrictions here. For example, the preset frame rate increment could be 2 frames per second. It can be understood that when periodically adjusting the input frame rate of the second processing node, at the next frame rate adjustment time, it can be determined whether to increase the input frame rate of the second processing node based on the obtained frame drop rate of the current period and the historical screen frame rate distribution. If it is determined to increase the input frame rate of the second processing node, the sum of the preset frame rate increment and the historical input frame rate can continue to be used as the input frame rate of the current period, and so on, until the condition for increasing the input frame rate of the second processing node is no longer met.

[0096] By increasing the input frame rate of the second processing node, the number of original image frames discarded before the second processing node can be reduced, and the frame rate of the preview display after being processed by the second and first processing nodes can be increased, thereby improving the smoothness of the preview screen.

[0097] S503: Drops frames from the raw image stream acquired by the image sensor based on the input frame rate.

[0098] S504: The second processing node applies the first special effect to each frame of the original image stream after frame loss to obtain the second image stream.

[0099] S505: The first processing node applies a second effect to multiple frames of images in the second image stream to obtain a second image stream for preview display.

[0100] For detailed implementation of steps S503 to S505, please refer to Figure 2 The descriptions of the relevant parts of steps S202 to S204 shown are not repeated here.

[0101] In this embodiment, when the historical frame loss information meets the second preset condition and the historical input frame rate of the second processing node is less than the maximum screen-on frame rate in the screen-on frame rate distribution, that is, when the historical input rate of the second processing node is too low, the input frame rate of the second processing node is increased according to the preset frame rate increment to increase the frame rate of the image frames used for preview, thereby improving the preview image quality.

[0102] exist Figure 2 In some optional implementations of the image processing method provided in the illustrated embodiment, step S202 includes:

[0103] In response to the historical frame loss information meeting the third preset condition, and the historical target frame rate being greater than or equal to the maximum on-screen frame rate in the on-screen frame rate distribution, the historical target frame rate is used as the current target frame rate.

[0104] The third preset condition includes a historical frame loss rate indicated by historical frame loss information that is less than a third preset frame loss rate threshold.

[0105] The aforementioned third preset frame drop rate threshold can be set according to specific application scenarios, and there are no restrictions here. For illustration, the third preset frame drop rate threshold can be 2 frames per second.

[0106] In these implementations, when the historical frame drop rate is low or there are no frame drops, the frame rate cannot be increased if the historical target frame rate is greater than or equal to the maximum on-screen frame rate. Furthermore, to avoid increasing the frame drop rate of the image stream input to the second processing node, the historical input frame rate of the second processing node in the previous cycle can be used as the input frame rate for the current cycle. Subsequently, frames can be dropped from the original image stream acquired in the current cycle based on this input frame rate. Then, the original image stream with dropped frames is input to the second processing node to apply the first effect, resulting in a first and second image stream; the second image stream is input to the first processing node to apply the second effect, resulting in a second image stream to be previewed. This process reduces frame drops in the image stream by the first processing node, thus reducing resource waste.

[0107] Please refer to Figure 6 , Figure 6 for Figure 3 The diagram shows the workflow of the frame rate control module. Figure 6 As shown, the image sensor acquires images at a preset image acquisition frequency to obtain a raw image stream 601. This raw image stream is input to the frame rate control module 602. The frame rate control module 602 can obtain historical frame loss information. If the historical frame loss information meets a first preset condition, the frame rate control type is determined to be a reduced frame rate. If the frame rate control module 602 determines that the frame rate control type 603 is a reduced frame rate, it can obtain the historical screen frame rate distribution. This historical screen frame rate distribution could, for example, be the on-screen frame rate for each second within the last 10 seconds. Figure 6 The maximum on-screen frame rate within the last 10 seconds is used as the input frame rate 604 for the second processing node. After setting the input frame rate of the second processing node to the maximum on-screen frame rate within the last 10 seconds, the frame rate control type can be set to hold frame rate 605.

[0108] For the frame rate control module 602, when determining that the frame rate control type 603 is to maintain the input frame rate, it can determine whether the maintenance duration of the input frame rate of the second processing node exceeds a preset duration 606. This preset duration can be, for example, 1 second. After the maintenance duration of the input frame rate exceeds the preset duration, it can determine whether the input frame rate is less than the maximum on-screen frame rate within the last 10 seconds 607. If the determination result is yes, then the frame rate control type is set to increase the frame rate 608. And during the process of maintaining the input frame rate, adaptive frame dropping 611 is performed on the original image stream according to the input frame rate.

[0109] For the frame rate control module 602, when the frame rate control type is determined to be increasing the frame rate, the sum of the historical input frame rate of the second processing node and the preset frame rate increment can be used as the input frame rate 609 of the current period. After determining the input frame rate, the frame rate control type can be set to maintaining the frame rate 610 so as to adaptively drop frames in the original image stream according to the input frame rate in the current period.

[0110] After dropping frames from the original image stream according to the input frame rate of the second processing node, the original image stream with dropped frames is output to the second effect application node 612.

[0111] Through the above process, the frame rate control module can periodically determine the input frame rate of the second processing node, adaptively drop frames in the input original image stream according to the input frame rate, and input the original image stream after frame dropping to the second effect application node to apply the first effect.

[0112] Corresponding to the image processing method in the above embodiments, Figure 7 This is a structural block diagram of an image processing apparatus provided according to embodiments of the present disclosure. For ease of explanation, only the parts relevant to embodiments of the present disclosure are shown. (Refer to...) Figure 7The device includes: an acquisition unit 701, a frame rate determination and frame dropping unit 702, a first special effects processing unit 703, and a second special effects processing unit 704.

[0113] The acquisition unit 701 is used to acquire historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after being processed by the first processing node.

[0114] The frame rate determination and frame dropping unit 702 is used to determine the input frame rate of the second processing node based on historical frame dropping information and / or historical screen frame rate distribution, and to drop frames from the original image stream acquired by the image sensor based on the input frame rate.

[0115] The first special effects processing unit 703 is used to apply the first special effects to each frame of the original image stream after frame loss by the second processing node to obtain the second image stream;

[0116] The second special effects processing unit 704 is used to apply a second special effect to multiple frames of images in the second image stream by the first processing node to obtain a second image stream for preview display.

[0117] In this embodiment, by acquiring historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after processing by the first processing node, the input frame rate of the second processing node is determined based on the historical frame loss information and / or the historical screen frame rate distribution. Frame loss is then performed on the original image stream acquired by the image sensor based on the input frame rate. This achieves dynamic frame loss of the original image stream based on the historical frame loss information and / or the historical screen frame rate distribution of the first processing node. This reduces the number of images subsequently discarded in the first processing node. Since frames are lost before the second processing node, the number of images processed by the second processing node is reduced, thereby reducing the resource consumption of the second processing node and minimizing resource waste. The saved resources can be used by the first processing node to apply a first special effect to the image, improving the efficiency of applying the first special effect, thereby increasing the on-screen frame rate and improving the preview effect.

[0118] In one embodiment of this disclosure, the frame rate determination and frame dropping unit 702 is further configured to determine the input frame rate of the second processing node based on one of the following:

[0119] The input frame rate is obtained by reducing the historical input frame rate of the second processing node based on the historical frame loss information.

[0120] Based on historical frame loss information and historical screen frame rate distribution, the input frame rate is obtained by adding the historical input frame rate of the second processing node.

[0121] Based on historical frame loss information and historical screen rate distribution, the historical input frame rate of the second processing node is used as the input frame rate.

[0122] In summary, further, the frame rate input to the first processing node can be reduced, the frame rate input to the second processing node can be increased, or the frame rate input to the second processing node can be maintained based on historical frame loss information, in order to adapt to the frame loss information of the first processing node and the preview display effect.

[0123] In one embodiment of this disclosure, the frame rate determination and frame dropping unit 702 is further configured to:

[0124] In response to the historical frame loss information meeting the first preset condition, the historical screen frame rate distribution within the first preset time period is obtained, and the maximum on-screen frame rate in the historical screen frame rate distribution is used as the input frame rate.

[0125] In summary, further, by taking the maximum on-screen frame rate in the historical screen frame rate distribution within the first preset time period as the input frame rate of the second processing node when the historical frame loss information meets the first preset condition, and then dropping frames on the original image stream based on the above input frame rate, resource waste can be reduced, and the smoothness of the preview effect of the image sequence with the second special effect can be ensured.

[0126] In one embodiment of this disclosure, the first preset condition includes:

[0127] The historical frame loss information indicates that the historical frame loss rate is greater than or equal to the first preset frame loss rate threshold.

[0128] In one embodiment of this disclosure, the frame rate determination and frame dropping unit 702 is further configured to:

[0129] In response to the historical frame loss information meeting the second preset condition, and the historical input frame rate being less than the maximum on-screen frame rate in the historical screen frame rate information, the historical input frame rate is increased according to the preset frame rate increment to obtain the input frame rate.

[0130] In summary, further, when the historical frame loss information meets the second preset condition, and the historical input frame rate of the second processing node is less than the maximum screen-on frame rate in the screen-on frame rate distribution, that is, when the historical input rate of the second processing node is too low, the input frame rate of the second processing node is increased according to the preset frame rate increment, so as to increase the frame rate of the image frames used for preview, thereby improving the preview image quality.

[0131] In one embodiment of this disclosure, the second preset condition includes:

[0132] The historical frame loss information indicates that the historical frame loss rate is less than the second preset frame loss rate threshold.

[0133] In one embodiment of this disclosure, the frame rate determination and frame dropping unit 702 is further configured to:

[0134] In response to the historical frame loss information meeting the third preset condition, and the historical input frame rate being greater than or equal to the maximum on-screen frame rate in the on-screen frame rate information, the historical input frame rate is used as the input frame rate.

[0135] The third preset condition includes a historical frame loss rate indicated by historical frame loss information that is less than a third preset frame loss rate threshold.

[0136] In summary, further, when the historical frame drop rate is low or nonexistent, the frame rate cannot be increased if the historical target frame rate is greater than or equal to the aforementioned maximum on-screen frame rate. Simultaneously, to avoid increasing the frame drop rate of the image stream input to the second processing node, the historical input frame rate of the second processing node from the previous cycle can be used as the input frame rate for the current cycle. Subsequently, frames can be dropped from the original image stream acquired in the current cycle based on this input frame rate. Then, the original image stream with dropped frames is input to the second processing node to apply the first effect, resulting in a first and second image stream; the second image stream is input to the first processing node to apply the second effect, resulting in a second image stream to be previewed. This process reduces frame drop by the first processing node, thus minimizing resource waste.

[0137] In one embodiment of this disclosure, the acquisition unit 701 is further configured to:

[0138] Historical frame loss information is determined based on the number of images that applied the first special effect but were not stored in the cache queue; where the cache queue is the cache queue used by the first processing node.

[0139] In one embodiment of this disclosure, the first processing node and the second processing node run on different threads.

[0140] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0141] To implement the above embodiments, this disclosure also provides an electronic device.

[0142] refer to Figure 8 The diagram illustrates a structural schematic of an electronic device 800 suitable for implementing embodiments of the present disclosure. The electronic device 800 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0143] like Figure 8 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0144] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0145] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0146] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0147] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0148] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0149] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0151] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0152] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0153] In a first aspect, according to one or more embodiments of the present disclosure, an image processing method is provided, comprising:

[0154] Obtain historical frame loss information of the first processing node processing the first image stream, and historical screen frame rate distribution of the first image stream after being processed by the first processing node;

[0155] The input frame rate of the second processing node is determined based on historical frame loss information and / or historical screen frame rate distribution, and frames are dropped from the raw image stream acquired by the image sensor based on the input frame rate.

[0156] The second processing node applies the first special effect to each frame of the original image stream after frame loss to obtain the second image stream;

[0157] The first processing node applies a second effect to multiple frames of images in the second image stream to obtain a second image stream for preview display.

[0158] According to one or more embodiments of this disclosure,

[0159] The input frame rate of the second processing node is determined based on historical frame loss information and / or historical screen frame rate information, including one of the following:

[0160] The input frame rate is obtained by reducing the historical input frame rate of the second processing node based on the historical frame loss information.

[0161] Based on historical frame loss information and historical screen frame rate distribution, the input frame rate is obtained by adding the historical input frame rate of the second processing node.

[0162] Based on historical frame loss information and historical screen rate distribution, the historical input frame rate of the second processing node is used as the input frame rate.

[0163] According to one or more embodiments of this disclosure, the input frame rate is obtained by reducing the historical input frame rate of the first processing node based on historical frame loss information, including:

[0164] In response to the historical frame loss information meeting the first preset condition, the historical screen frame rate distribution within the first preset time period is obtained, and the maximum on-screen frame rate in the historical screen frame rate distribution is used as the input frame rate.

[0165] According to one or more embodiments of this disclosure, the first preset condition includes:

[0166] The historical frame loss information indicates that the historical frame loss rate is greater than or equal to the first preset frame loss rate threshold.

[0167] According to one or more embodiments of this disclosure, the input frame rate is obtained by adding the historical input frame rate of the second processing node based on historical frame loss information and historical screen frame rate distribution, including:

[0168] In response to the historical frame loss information meeting the second preset condition, and the historical input frame rate being less than the maximum on-screen frame rate in the historical screen frame rate information, the historical input frame rate is increased according to the preset frame rate increment to obtain the input frame rate.

[0169] According to one or more embodiments of this disclosure, the second preset condition includes:

[0170] The historical frame loss information indicates that the historical frame loss rate is less than the second preset frame loss rate threshold.

[0171] According to one or more embodiments of this disclosure, determining a target frame rate based on historical frame loss information and / or historical screen frame rate distribution includes:

[0172] Based on historical frame loss information and historical screen rate distribution, the historical input frame rate of the second processing node is used as the input frame rate, including:

[0173] In response to the historical frame loss information meeting the third preset condition, and the historical input frame rate being greater than or equal to the maximum on-screen frame rate in the on-screen frame rate information, the historical input frame rate is used as the input frame rate.

[0174] The third preset condition includes a historical frame loss rate indicated by historical frame loss information that is less than a third preset frame loss rate threshold.

[0175] Secondly, according to one or more embodiments of the present disclosure, an image processing apparatus is provided, comprising:

[0176] The acquisition unit is used to acquire historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after being processed by the first processing node.

[0177] The frame rate determination and frame dropping unit is used to determine the input frame rate of the second processing node based on historical frame dropping information and / or historical screen frame rate distribution, and to drop frames from the raw image stream acquired by the image sensor based on the input frame rate.

[0178] The first special effects processing unit is used to apply the first special effects to each frame of the original image stream after frame loss by the second processing node to obtain the second image stream;

[0179] The second special effects processing unit is used to apply second special effects to multiple frames of images in the second image stream by the first processing node to obtain a second image stream for preview display.

[0180] According to one or more embodiments of this disclosure, the frame rate determination and frame dropping unit is further configured to determine the input frame rate of the second processing node based on one of the following:

[0181] The input frame rate is obtained by reducing the historical input frame rate of the second processing node based on the historical frame loss information.

[0182] Based on historical frame loss information and historical screen frame rate distribution, the input frame rate is obtained by adding the historical input frame rate of the second processing node.

[0183] Based on historical frame loss information and historical screen rate distribution, the historical input frame rate of the second processing node is used as the input frame rate.

[0184] According to one or more embodiments of this disclosure, the frame rate determination and frame dropping unit is further configured to:

[0185] In response to the historical frame loss information meeting the first preset condition, the historical screen frame rate distribution within the first preset time period is obtained, and the maximum on-screen frame rate in the historical screen frame rate distribution is used as the input frame rate.

[0186] According to one or more embodiments of this disclosure, the first preset condition includes:

[0187] The historical frame loss information indicates that the historical frame loss rate is greater than or equal to the first preset frame loss rate threshold.

[0188] According to one or more embodiments of this disclosure, the frame rate determination and frame dropping unit is further configured to:

[0189] In response to the historical frame loss information meeting the second preset condition, and the historical input frame rate being less than the maximum on-screen frame rate in the historical screen frame rate information, the historical input frame rate is increased according to the preset frame rate increment to obtain the input frame rate.

[0190] According to one or more embodiments of this disclosure, the second preset condition includes:

[0191] The historical frame loss information indicates that the historical frame loss rate is less than the second preset frame loss rate threshold.

[0192] According to one or more embodiments of this disclosure, the frame rate determination and frame dropping unit is further configured to:

[0193] In response to the historical frame loss information meeting the third preset condition, and the historical input frame rate being greater than or equal to the maximum on-screen frame rate in the on-screen frame rate information, the historical input frame rate is used as the input frame rate.

[0194] The third preset condition includes a historical frame loss rate indicated by historical frame loss information that is less than a third preset frame loss rate threshold.

[0195] According to one or more embodiments of this disclosure, the acquisition unit is further configured to:

[0196] Historical frame loss information is determined based on the number of images that applied the first special effect but were not stored in the cache queue; where the cache queue is the cache queue used by the first processing node.

[0197] According to one or more embodiments of this disclosure, the first processing node and the second processing node run on different threads.

[0198] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0199] The memory stores the instructions that the computer executes;

[0200] At least one processor executes computer execution instructions stored in memory, causing at least one processor to perform the image processing method as described in the first aspect above and various possible designs of the first aspect.

[0201] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, implement the image processing method described in the first aspect and various possible designs of the first aspect.

[0202] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method as described in the first aspect above and various possible designs of the first aspect.

[0203] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0204] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0205] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method, comprising: Obtain historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after being processed by the first processing node; The input frame rate of the second processing node is determined based on the historical frame loss information and / or the historical screen frame rate distribution, and frames are dropped from the original image stream acquired by the image sensor based on the input frame rate. The second processing node applies the first special effect to each frame of the original image stream after frame loss to obtain the second image stream; The first processing node applies a second effect to multiple frames of images in the second image stream to obtain a second image stream for preview display.

2. The method according to claim 1, characterized in that, Determining the input frame rate of the second processing node based on the historical frame loss information and / or the historical screen frame rate information includes one of the following: The input frame rate is obtained by reducing the historical input frame rate of the second processing node based on the historical frame loss information. Based on historical frame loss information and the historical screen frame rate distribution, the input frame rate is obtained by adding the historical input frame rate of the second processing node; Based on the historical frame loss information and the historical screen rate distribution, the historical input frame rate of the second processing node is used as the input frame rate.

3. The method according to claim 2, characterized in that, The step of reducing the historical input frame rate of the first processing node based on historical frame loss information to obtain the input frame rate includes: In response to the historical frame loss information meeting the first preset condition, the historical screen frame rate distribution within the first preset time period is obtained, and the maximum on-screen frame rate in the historical screen frame rate distribution is used as the input frame rate.

4. The method according to claim 3, characterized in that, The first preset conditions include: The historical frame loss information indicates a historical frame loss rate that is greater than or equal to a first preset frame loss rate threshold.

5. The method according to claim 2, characterized in that, The step of obtaining the input frame rate by adding the historical input frame rate of the second processing node based on historical frame loss information and the historical screen frame rate distribution includes: In response to the historical frame loss information satisfying the second preset condition, and the historical input frame rate being less than the maximum on-screen frame rate in the historical screen frame rate information, the historical input frame rate is increased by a preset frame rate increment to obtain the input frame rate.

6. The method according to claim 5, characterized in that, The second preset condition includes: The historical frame loss information indicates that the historical frame loss rate is less than the second preset frame loss rate threshold.

7. The method according to claim 2, characterized in that, The step of using the historical input frame rate of the second processing node as the input frame rate based on the historical frame loss information and the historical screen rate distribution includes: In response to the historical frame loss information satisfying the third preset condition, and the historical input frame rate being greater than or equal to the maximum on-screen frame rate in the on-screen frame rate information, the historical input frame rate is used as the input frame rate; The third preset condition includes the historical frame loss rate indicated by the historical frame loss information being less than the third preset frame loss rate threshold.

8. The method according to any one of claims 1-7, characterized in that, The step of obtaining historical frame loss information of the first processing node processing the first image stream includes: The historical frame loss information is determined based on the number of images that applied the first special effect but were not stored in the cache queue; wherein, the cache queue is the cache queue used by the first processing node.

9. An image processing apparatus, comprising: The acquisition unit is used to acquire historical frame loss information of the first processing node processing the first image stream, and the historical screen frame rate distribution of the first image stream after being processed by the first processing node. The frame rate determination and frame dropping unit is used to determine the input frame rate of the second processing node based on the historical frame dropping information and / or the historical screen frame rate distribution, and to drop frames from the original image stream acquired by the image sensor based on the input frame rate. The first special effects processing unit is used to apply the first special effects to each frame of the original image stream after frame loss by the second processing node to obtain the second image stream; The second special effects processing unit is used to apply a second special effect to multiple frames of images in the second image stream by the first processing node to obtain a second image stream for preview display.

10. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the image processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the image processing method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 8.

Images