Image processing method and related equipment

Abstract

The invention provides an image processing method and related equipment. The method comprises the following steps: acquiring a plurality of image frames to be processed; determining motion information of a first image frame in the plurality of image frames relative to a second image frame, wherein the first image frame is an adjacent image frame behind the second image frame; sampling a second result frame corresponding to the second image frame based on the motion information to obtain a first intermediate result; wherein the second result frame is an image processing result of the second image frame; and performing fusion processing based on the first intermediate result and the second result frame to obtain an image processing result of the first image frame.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image processing method and related equipment. Background Technology

[0002] Users often use image effects to achieve richer visual effects. However, existing image effects typically process the entire image data, resulting in long processing times and poor real-time performance. While processing historical image frames can improve efficiency, the resulting effects are not realistic enough and lack appeal. Summary of the Invention

[0003] This disclosure proposes an image processing method and related equipment to address, to some extent, the technical problem that the real-time performance of image special effects and the processing effect cannot be simultaneously achieved.

[0004] In a first aspect, this disclosure provides an image processing method, comprising:

[0005] Acquire multiple image frames to be processed;

[0006] Determine the motion information of a first image frame relative to a second image frame among the plurality of image frames, wherein the first image frame is an adjacent image frame located after the second image frame;

[0007] Based on the motion information, the second result frame corresponding to the second image frame is sampled to obtain a first intermediate result; wherein, the second result frame is the image processing result of the second image frame;

[0008] Based on the first intermediate result and the second result frame, a fusion process is performed to obtain the image processing result of the first image frame.

[0009] A second aspect of this disclosure provides an image processing apparatus, comprising:

[0010] The acquisition module is used to acquire multiple image frames to be processed;

[0011] A motion information module is used to determine the motion information of a first image frame relative to a second image frame among the plurality of image frames, wherein the first image frame is an adjacent image frame located after the second image frame;

[0012] The first intermediate result module is used to sample the second result frame corresponding to the second image frame based on the motion information to obtain the first intermediate result; wherein, the second result frame is the image processing result of the second image frame;

[0013] The fusion processing module is used to perform fusion processing based on the first intermediate result and the second result frame to obtain the image processing result of the first image frame.

[0014] A third aspect of this disclosure provides an electronic device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the programs including instructions for performing the method according to the first aspect.

[0015] A fourth aspect of this disclosure provides a non-volatile computer-readable storage medium containing a computer program that, when executed by one or more processors, causes the processors to perform the method described in the first aspect.

[0016] A fifth aspect of this disclosure provides a computer program product including computer program instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0017] As can be seen from the above, the image processing method and related device provided in this disclosure sample the image processing result of the historical image frame based on the motion information between adjacent historical image frames and the current image frame to obtain a first intermediate result, and then fuse the first intermediate result with the image processing result of the historical image frame, and perform image processing on the current image frame by combining the cached historical image frames and motion information. This not only enables real-time image effect processing, but also effectively improves the realism of the effects, and balances processing efficiency with image processing effect. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the image processing architecture according to an embodiment of the present disclosure.

[0020] Figure 2 This is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0021] Figure 3 This is a schematic flowchart illustrating an image processing method according to an embodiment of the present disclosure.

[0022] Figure 4 This is a schematic diagram of motion vectors according to an embodiment of the present disclosure.

[0023] Figure 5 This is a schematic diagram of a first intermediate result of an embodiment of this disclosure.

[0024] Figure 6 This is a schematic diagram of the first mask according to an embodiment of the present disclosure.

[0025] Figure 7 This is a schematic diagram of a second intermediate result of an embodiment of this disclosure.

[0026] Figure 8 This is a schematic diagram of the second mask according to an embodiment of the present disclosure.

[0027] Figure 9 This is a schematic diagram of the first result frame of an embodiment of this disclosure.

[0028] Figure 10 This is a schematic diagram of an image processing method according to an embodiment of the present disclosure.

[0029] Figure 11 This is a schematic diagram of an image processing apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0031] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0032] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0033] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0034] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0035] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0036] Figure 1 A schematic diagram of an image processing architecture according to an embodiment of the present disclosure is shown. (Reference) Figure 1 The image processing architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 can be connected via a wired or wireless network 130. The server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, security services, and CDN.

[0037] Terminal 120 can be implemented in hardware or software. For example, when terminal 120 is implemented in hardware, it can be various electronic devices with a display screen and support page display, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal 120 is implemented in software, it can be installed in the electronic devices listed above; it can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module, without specific limitations.

[0038] It should be noted that the image processing method provided in this application embodiment can be executed by the terminal 120 or by the server 110. It should be understood that... Figure 1 The number of terminals, networks, and servers shown is for illustrative purposes only and is not intended to be a limitation. Any number of terminals, networks, and servers can be used depending on implementation needs.

[0039] Figure 2 A schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of this disclosure is shown. For example... Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208, and a bus 210. The processor 202, memory 204, network module 206, and peripheral interface 208 are interconnected within the electronic device 200 via the bus 210.

[0040] Processor 202 may be a central processing unit (CPU), image processor, neural network processor (NPU), microcontroller (MCU), programmable logic device, digital signal processor (DSP), application-specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 can be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated as a single logic component. For example, such as... Figure 2 As shown, processor 202 may include multiple processors 202a, 202b and 202c.

[0041] Memory 204 can be configured to store data (e.g., instructions, computer code, etc.). Figure 2 As shown, the data stored in memory 204 may include program instructions (e.g., program instructions for implementing the image processing method of embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). Processor 202 may also access the program instructions and data stored in memory 204 and execute the program instructions to operate on the data to be processed. Memory 204 may include volatile or non-volatile storage devices. In some embodiments, memory 204 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0042] Network module 206 can be configured to provide communication with other external devices to electronic device 200 via a network. This network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the specific examples described above. In some embodiments, network module 306 may include any combination of any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc.

[0043] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to enable information input and output. For example, peripheral devices may include input devices such as keyboards, mice, touchpads, touch screens, microphones, and various sensors, as well as output devices such as displays, speakers, vibrators, and indicator lights.

[0044] Bus 210 can be configured to transfer information between various components of electronic device 200 (e.g., processor 202, memory 204, network module 206, and peripheral interface 208), such as internal buses (e.g., processor-memory bus), external buses (USB port, PCI-E bus), etc.

[0045] It should be noted that although the architecture of the above-described electronic device 200 only shows the processor 202, memory 204, network module 206, peripheral interface 208, and bus 210, in specific implementations, the architecture of the electronic device 200 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the architecture of the above-described electronic device 200 may only include the components necessary for implementing the embodiments of this disclosure, and does not necessarily include all the components shown in the figures.

[0046] Various image effects can be used to enrich image processing. Among these, glitch / scratching effects mimic the visual effects of television signal malfunctions, digital data errors, or electronic device failures. Existing glitch / scratching effects often take a complete video as input, decode it, destroy keyframes and speculative frames, and then re-encode it into a single video as output. While this method theoretically mimics video glitch / scratching and can produce a relatively realistic effect, it requires processing the entire video and cannot meet real-time processing requirements. Furthermore, existing glitch / scratching effects can also be based on post-processing methods, storing historical frames and using random noise or fixed masks to determine which frame or historical frame content is displayed at different positions on the screen. Although this method improves real-time performance, it doesn't integrate with the motion information of the original video, resulting in image effects that differ significantly from realistic video glitch / scratching. Therefore, improving the real-time performance and realism of image effects processing, and enhancing image processing effects and efficiency, have become urgent technical challenges.

[0047] In view of this, the present disclosure provides an image processing method and related apparatus. Based on the motion information between adjacent historical image frames and the current image frame, the image processing result of the historical image frame is sampled to obtain a first intermediate result. Then, the first intermediate result is fused with the image processing result of the historical image frame. Combining the cached historical image frames and motion information, image processing is performed on the current image frame. This not only enables real-time image effects processing but also effectively improves the realism of the effects, ensuring both image processing effectiveness and processing efficiency.

[0048] See Figure 3 , Figure 3 A schematic flowchart of an image processing method according to an embodiment of the present disclosure is shown. The image processing method according to an embodiment of the present disclosure can be deployed on a server or a terminal. Figure 3 In the image processing method 300, the following steps may be further included.

[0049] In step S310, multiple image frames to be processed are acquired.

[0050] Multiple image frames can form an image sequence or video. These multiple image frames can be acquired locally or via a network; there is no restriction on this.

[0051] In step S320, motion information of the first image frame relative to the second image frame among the plurality of image frames is determined, wherein the first image frame is an adjacent image frame located after the second image frame.

[0052] Here, the first image frame can refer to the current image frame being processed, and the second image frame can refer to the previous image frame. Motion information can refer to information such as the speed, displacement, and direction of motion of pixels in the first image frame relative to pixels in the second image frame. Motion information can include motion vectors and the image size used to determine those motion vectors.

[0053] In some embodiments, determining the motion information of a first image frame relative to a second image frame in the image sequence to be processed includes:

[0054] The first image frame and the second image frame are scaled to a preset size to obtain a first scaled image frame and a second scaled image frame;

[0055] The optical flow of pixels in the first scaled image frame relative to pixels in the second scaled image frame is determined to obtain the motion vector;

[0056] The motion vector and the preset size are determined as the motion information.

[0057] To accelerate and improve the efficiency of motion information estimation, the first and second image frames can be scaled to a suitable preset size S for optical flow calculation to obtain the motion direction and velocity of pixels, thus generating motion vectors. These motion vectors, along with the image size used to determine them (i.e., the preset size S), are used as motion information. Specifically, optical flow is calculated based on the first image frame (e.g., the current image frame) and the second image frame (e.g., the previous image frame, a historical image frame) to obtain the motion vectors for each pixel. These motion vectors can be stored on a 32-bit two-channel floating-point texture and passed to the GPU, such as... Figure 4 As shown, Figure 4 A schematic diagram of motion vectors according to an embodiment of the present disclosure is shown. Figure 4 In this process, the motion vector can be multiplied by A (e.g., 20) to make the display result more significant. Then, it is divided by the image size used when calculating the optical flow (i.e., the preset size S) and limited to a two-channel value between -1 and 1. This value is then added to 1 and divided by 2 to obtain a value between 0 and 1, which is used as... Figure 4 The red and green channels are set to 0, the blue channel is set to 0, and the opacity channel is set to 1. The color values ​​range from 0 to 1.

[0058] In step S330, the second result frame corresponding to the second image frame is sampled based on the motion information to obtain a first intermediate result; wherein, the second result frame is the image processing result of the second image frame.

[0059] In particular, combining motion information between adjacent image frames and the image processing results of historical image frames during image processing can improve the effectiveness and efficiency of image processing.

[0060] In some embodiments, sampling the second result frame corresponding to the second image frame based on the motion information to obtain a first intermediate result includes:

[0061] The sampling position of the second result frame is determined based on the motion information and the target position of the first intermediate result;

[0062] The pixel value corresponding to the sampling position in the second result frame is determined as the pixel value of the target position to obtain the first intermediate result.

[0063] The first intermediate result can be obtained by applying the motion vector to the second result frame (e.g., the historical image processing result corresponding to the previous image frame).

[0064] In some embodiments, determining the sampling position of the second result frame based on the motion information and the target position of the first intermediate result includes:

[0065] The sampling offset is determined based on the motion vector, the preset size, and the preset coefficient;

[0066] The position in the second result frame corresponding to the target position is determined as the initial sampling position;

[0067] The sampling position is obtained based on the initial sampling position and the sampling offset.

[0068] Specifically, after adding the motion vector, the sampling position of each pixel in the first intermediate result is: uv1 = uv0 - α × vof ÷ sof. Where uv1 is the sampling position of the second result frame, uv0 is the position in the second result frame that matches the target position of the first intermediate result, α is a user-defined coefficient used to influence the offset of the sampling position, vof is the motion vector, and sof is the image size used to calculate optical flow, usually smaller than the original size of the element image frame, and the image ratio is consistent with the original ratio during calculation. Sampling the second result frame based on the sampling position yields the first intermediate result image. Figure 5 As shown, Figure 5 A schematic diagram of a first intermediate result according to an embodiment of the present disclosure is shown.

[0069] In step S340, a fusion process is performed based on the first intermediate result and the second result frame to obtain the image processing result of the first image frame.

[0070] Specifically, at least a portion of the contents of the first intermediate result and the second result frame can be fused to obtain the image processing result of the first image frame. This image processing result of the first image frame can be the first result frame.

[0071] In some embodiments, a fusion process is performed based on the first intermediate result and the second result frame to obtain the image processing result of the first image frame, including:

[0072] The pixel values ​​of the first color portion in the first mask are updated to the corresponding pixel values ​​in the second result frame, and the pixel values ​​of the second color portion in the first mask are updated to the corresponding pixel values ​​in the first intermediate result to obtain the second intermediate result.

[0073] The image processing result of the first image frame is determined based on the second intermediate result.

[0074] The first mask can be a binarized mask that includes a first color portion and a second color portion, such as... Figure 6 As shown, Figure 6 A schematic diagram of a first mask according to an embodiment of the present disclosure is shown. It can be based on mosaic-like noise. Figure 1 Binarization yields mosaic-like noise. Figure 1 Each square in the image can have the same color, for example, a grayscale value ranging from 0 to 1. (Regarding noise...) Figure 1 Binarization is performed according to threshold 1 to obtain the first mask. Threshold 1 can range from 0 to 1 and can be adjusted by the user. The first mask can be applied to the second result frame and the first intermediate result. The first color part (e.g., black, i.e., the part with a grayscale of 0) can use the corresponding pixel value in the second result frame, and the second color part (e.g., white, i.e., the part with a grayscale of 1) can use the corresponding pixel value in the first intermediate result to obtain the second intermediate result, such as... Figure 7 As shown, Figure 7 A schematic diagram of a second intermediate result according to an embodiment of the present disclosure is shown.

[0075] In some embodiments, determining the image processing result of the first image frame based on the second intermediate result includes: determining the second intermediate result as the image processing result of the first image frame.

[0076] The second intermediate result can be directly identified as the image processing result of the first image frame, i.e., the first result frame.

[0077] In some embodiments, determining the image processing result of the first image frame based on the second intermediate result includes:

[0078] The second intermediate result and the first image frame are fused together to obtain the image processing result of the first image frame.

[0079] In some embodiments, the second intermediate result and the first image frame are fused to obtain the image processing result of the first image frame, including:

[0080] The pixel values ​​of the third color portion in the second mask are updated to the corresponding pixel values ​​in the first image frame, and the pixel values ​​of the fourth color portion in the second mask are updated to the corresponding pixel values ​​in the second intermediate result, thereby obtaining the image processing result of the first image frame.

[0081] The second mask can be a binarized mask that includes a third color portion and a fourth color portion, such as... Figure 8 As shown, Figure 8 A schematic diagram of a second mask according to an embodiment of the present disclosure is shown. It can be based on mosaic-like noise. Figure 2 Binarization yields mosaic-like noise. Figure 2 Each square in the image can have the same color, for example, a grayscale value ranging from 0 to 1. (Regarding noise...) Figure 2 Binarization is performed using threshold 2 to obtain the second mask. Threshold 2 can range from 0 to 1 and can be used as an adjustment parameter for the user to adjust.

[0082] Furthermore, the second mask can be applied to both the second intermediate result and the first image frame. The third color portion (e.g., black, i.e., the portion with a grayscale of 0) can use the corresponding pixel value in the first image frame, and the second color portion (e.g., white, i.e., the portion with a grayscale of 1) can use the corresponding pixel value in the second intermediate result to obtain the first result frame, as shown below. Figure 9 As shown, Figure 9 A schematic diagram of a first result frame according to an embodiment of the present disclosure is shown.

[0083] Furthermore, in some embodiments, the second intermediate result and the first image frame can be fused at each preset time interval to obtain the image processing result of the first image frame.

[0084] In some embodiments, method 300 further includes:

[0085] In response to determining that the first image frame is the first frame of the plurality of image frames, the first image frame is determined as the image processing result of the first image frame.

[0086] If the current image frame is the first frame of a set of multiple image frames, it is directly used as the corresponding current result frame. The current result frame and the current image frame are recorded, and image processing of the current image frame ends. If the current image frame is not the first frame of a set of multiple image frames, motion estimation can be performed based on the current image frame and the previous image frame, for example, by executing step S320.

[0087] See Figure 10 , Figure 10 A schematic diagram of an image processing method according to an embodiment of the present disclosure is shown. Figure 10As shown, multiple image frames are acquired, including image_1, image_2, ..., image_i, ..., where i is a positive integer. For each current image frame, it is recorded as a historical image frame; for example, the current image frame image_i is recorded as a historical image frame image_h_i.

[0088] In addition, for each current image frame, determine whether the current image frame is the first frame.

[0089] In response to the current image frame being the first frame, the current image frame is determined as the current result frame; for example, if the current image frame is image_1, which is the first frame of the image sequence to be processed, then image_1 is taken as the corresponding current result frame image_1'.

[0090] In response to the current image frame not being the first frame, the motion information of the current image frame is determined based on the previous result frame corresponding to the previous image frame. For example, if the current image frame is image_2, which is not the first frame of the image sequence to be processed, motion estimation can be performed based on the previous result frame image_1' corresponding to the previous image frame image_1 to obtain the motion information of the current image frame image_2. As another example, if the current image frame is image_i, which is not the first frame of the image sequence to be processed, motion estimation can be performed based on the previous result frame image_i-1' corresponding to the previous image frame image_i-1 to obtain the motion information v_info_i of the current image frame image_i. The motion information v_info_i may include the motion vector v. of _i and the image size s with respect to the motion vector of _i.

[0091] Based on the motion information of the current image frame, pixels from the previous result frame are sampled to obtain intermediate result 1. For example, for pixel pixel_j (j is a positive integer) in intermediate result 1, its position in intermediate result 1 is uv_j. The pixel value pixel_v_j of pixel pixel_j can be determined based on the motion information v_info_i of the current image frame image_i. Specifically, the position of pixel pixel_j in intermediate result 1 is uv_j, which is consistent with the position uv0 of the previous result frame image_i-1'. Based on this, the sampling position uv1 of pixel pixel_j relative to the previous result frame image_i-1' can be obtained as uv1 = uv0 - α × v of _i÷s of_i. Then the pixel value of pixel_j in intermediate result 1, pixel_v_j, is the pixel value of position uv1 in the previous result frame image_i-1'.

[0092] Intermediate result 1 can be masked to obtain intermediate result 2. Specifically, this can be done based on mosaic-like noise. Figure 1 Binarization is performed according to threshold 1 to obtain updated mask 1. The first color (e.g., black) portion of updated mask 1 is updated to the content of the previous result frame image_i-1', and the second color (e.g., white) portion of updated mask 1 is updated to the content of intermediate result 1 to obtain intermediate result 2. At this time, intermediate result 2 can be used as the current result frame image_i' of the current image frame image_i.

[0093] Furthermore, at least a portion of the intermediate result 2 can be updated to the content of the current image frame image_i at regular time intervals to obtain the current result frame image_i” of the current image frame image_i. For example, the intermediate result 2 can be masked again to obtain the current result frame image_i”. In this case, the intermediate result 2 may not be used as the current result frame of the current image frame. Specifically, it can be based on mosaic-like noise. Figure 2 Binarize according to threshold 2 to obtain updated mask 2. Update the third color (e.g., black) part in updated mask 2 with the content of intermediate result 2, and update the fourth color (e.g., white) part in updated mask 1 with the content of the current image frame image_i to obtain the current result frame image_i.

[0094] Record the current result frame image_i' or image_i" of the current image frame as a historical result frame. This historical result frame can be used to determine the sampling position of the previous result frame when sampling the intermediate result 1, and to perform masking processing 1 on the intermediate result 1.

[0095] As can be seen, the image processing method according to the embodiments of this disclosure, by combining historical input frames or historical result frames and using motion estimation and noise reduction methods, can not only simulate the fault screen distortion effect in real time, but also effectively improve the realism of the effect, ensuring both image processing effect and processing efficiency.

[0096] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0097] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] Based on the same technical concept, corresponding to any of the above embodiments, this disclosure also provides an image processing apparatus, see [link to relevant documentation]. Figure 11 The image processing apparatus includes:

[0099] The acquisition module is used to acquire multiple image frames to be processed;

[0100] A motion information module is used to determine the motion information of a first image frame relative to a second image frame among the plurality of image frames, wherein the first image frame is an adjacent image frame located after the second image frame;

[0101] The first intermediate result module is used to sample the second result frame corresponding to the second image frame based on the motion information to obtain the first intermediate result; wherein, the second result frame is the image processing result of the second image frame;

[0102] The fusion processing module is used to perform fusion processing based on the first intermediate result and the second result frame to obtain the image processing result of the first image frame.

[0103] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0104] The apparatus of the above embodiments is used to implement the corresponding image processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0105] Based on the same technical concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the image processing method as described in any of the above embodiments.

[0106] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0107] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the image processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0108] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0109] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0110] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0111] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method, comprising: Acquire multiple image frames to be processed; Determine the motion information of a first image frame relative to a second image frame among the plurality of image frames, wherein the first image frame is an adjacent image frame located after the second image frame; Based on the motion information, the second result frame corresponding to the second image frame is sampled to obtain a first intermediate result; wherein, the second result frame is the image processing result of the second image frame; Based on the first intermediate result and the second result frame, a fusion process is performed to obtain the image processing result of the first image frame.

2. The method according to claim 1, wherein, Determining the motion information of the first image frame relative to the second image frame in the image sequence to be processed includes: The first image frame and the second image frame are scaled to a preset size to obtain a first scaled image frame and a second scaled image frame; The optical flow of pixels in the first scaled image frame relative to pixels in the second scaled image frame is determined to obtain the motion vector; The motion vector and the preset size are determined as the motion information.

3. The method according to claim 2, wherein, Based on the motion information, the second result frame corresponding to the second image frame is sampled to obtain a first intermediate result, including: The sampling position of the second result frame is determined based on the motion information and the target position of the first intermediate result; The pixel value corresponding to the sampling position in the second result frame is determined as the pixel value of the target position to obtain the first intermediate result.

4. The method according to claim 3, wherein, Determining the sampling position of the second result frame based on the motion information and the target position of the first intermediate result includes: The sampling offset is determined based on the motion vector, the preset size, and the preset coefficient; The position in the second result frame corresponding to the target position is determined as the initial sampling position; The sampling position is obtained based on the initial sampling position and the sampling offset.

5. The method according to claim 4, wherein, Based on the first intermediate result and the second result frame, a fusion process is performed to obtain the image processing result of the first image frame, including: The pixel values ​​of the first color portion in the first mask are updated to the corresponding pixel values ​​in the second result frame, and the pixel values ​​of the second color portion in the first mask are updated to the corresponding pixel values ​​in the first intermediate result to obtain the second intermediate result. The image processing result of the first image frame is determined based on the second intermediate result.

6. The method according to claim 5, wherein, Determining the image processing result of the first image frame based on the second intermediate result includes: The second intermediate result is determined as the image processing result of the first image frame; or, The second intermediate result and the first image frame are fused together to obtain the image processing result of the first image frame.

7. The method according to claim 6, wherein, The second intermediate result and the first image frame are fused to obtain the image processing result of the first image frame, including: The pixel values ​​of the third color portion in the second mask are updated to the corresponding pixel values ​​in the first image frame, and the pixel values ​​of the fourth color portion in the second mask are updated to the corresponding pixel values ​​in the second intermediate result, thereby obtaining the image processing result of the first image frame.

8. The method of claim 1, further comprising: In response to determining that the first image frame is the first frame of the plurality of image frames, the first image frame is determined as the image processing result of the first image frame.

9. An image processing apparatus, comprising: The acquisition module is used to acquire multiple image frames to be processed; A motion information module is used to determine the motion information of a first image frame relative to a second image frame among the plurality of image frames, wherein the first image frame is an adjacent image frame located after the second image frame; The first intermediate result module is used to sample the second result frame corresponding to the second image frame based on the motion information to obtain the first intermediate result; wherein, the second result frame is the image processing result of the second image frame; The fusion processing module is used to perform fusion processing based on the first intermediate result and the second result frame to obtain the image processing result of the first image frame.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 8.

12. A computer program product comprising computer program instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 8.

Images