Image processing method, device, electronic device and medium

Abstract

The present application discloses an image processing method, device, electronic device and medium, belonging to the field of electronic devices. The image processing method includes: calculating the first optical flow information between images with adjacent shooting time in a continuously shot image sequence; determining at least two reference images and at least one non-reference image frame matching each reference image from the image sequence; based on the first optical flow information, calculating the second optical flow information between the matched reference image and the non-reference image; based on the second optical flow information, performing optical flow registration on each non-reference image to generate an aligned image; performing fusion processing on the matched reference image and the optical flow aligned image respectively to generate a fused image corresponding to the reference image.

Description

Technical Field

[0001] The present application belongs to the field of electronic devices, and specifically relates to an image processing method, device, electronic device, readable storage medium and computer program product. Background Art

[0002] With the development of electronic devices, electronic devices can perform high-speed continuous shooting to obtain multiple frames of images taken continuously. In order to improve image quality, each frame of the multiple frames of images taken continuously can be optimized, but the processing speed of optimizing multiple frames of images at the same time is slow, and the efficiency of generating optimized images is low. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and medium, which can solve the problem of low optimization efficiency of continuously captured multi-frame images.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Calculating first optical flow information between images with adjacent shooting times in a sequence of continuously shot images;

[0006] Determine at least two reference images and at least one non-reference image matching each reference image from the image sequence;

[0007] Based on the first optical flow information, calculating second optical flow information between the matched reference image and the non-reference image;

[0008] Based on the second optical flow information, performing optical flow registration on each non-reference image to generate an aligned image;

[0009] The matched reference image and the optical flow aligned image are fused respectively to generate a fused image corresponding to the reference image.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0011] A first optical flow information calculation module, used to calculate first optical flow information between images with adjacent shooting times in a sequence of continuously shot images;

[0012] A reference image determination module, used to determine at least two reference images and at least one non-reference image matching each reference image from an image sequence;

[0013] A second optical flow information calculation module, used to calculate second optical flow information between the matched reference image and the non-reference image based on the first optical flow information;

[0014] A registration module, used to perform optical flow registration on each non-reference image based on the second optical flow information to generate an aligned image;

[0015] The fusion module is used to fuse the matched reference image and the optical flow aligned image respectively to generate a fused image corresponding to the reference image.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method of the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method of the first aspect.

[0020] In an embodiment of the present application, first optical flow information is calculated between images with adjacent shooting times in a sequence of continuously shot images; based on the first optical flow information, second optical flow information is calculated between matching reference images and non-reference images, thereby eliminating the need to directly calculate optical flow information between any reference image and non-reference image, reducing the amount of calculation and improving image processing efficiency; based on the second optical flow information, optical flow registration is performed on each non-reference image to generate an aligned image; matching reference images and optical flow aligned images are fused separately to generate a fused image corresponding to the reference image, thereby avoiding misalignment of contents on different images that may cause distortion of the fused image generated by the final image fusion, and improving the image quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is one of the flowcharts of the image processing method provided in one embodiment of the present application;

[0022] Figure 2 This is a second flowchart of an image processing method provided in an embodiment of the present application;

[0023] Figure 3 This is a third flowchart of an image processing method provided in an embodiment of the present application;

[0024] Figure 4 This is a fourth flowchart of an image processing method provided in an embodiment of the present application;

[0025] Figure 5 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] The image processing method, device, electronic device, readable storage medium and computer program product provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0031] See also Figure 1 , an embodiment of the present application provides an image processing method, the method comprising:

[0032] S110, calculating first optical flow information between images with adjacent shooting times in a sequence of continuously shot images;

[0033] A continuously shot image sequence is an image sequence consisting of multiple frames of images taken continuously for the same scene. An image sequence refers to a sequence of images arranged in a certain order. The images in the image sequence can be arranged according to the shooting time, and images with adjacent shooting times can be understood as images with adjacent positions in the image sequence. Since the image sequence is obtained by continuous shooting, the images with adjacent shooting times in the image sequence have a high similarity, so that multiple frames of images can be selected from the image sequence for fusion processing in the future to obtain a fused image with better image quality.

[0034] The image sequence may be obtained by real-time shooting or may be obtained from a database and stored in advance, which is not limited in this application. In some embodiments, before S110, it may include: receiving a first input; and executing S110 in response to the first input. The first input may be a single click, double click, long press, sliding, etc. input made by the user in the photo preview interface.

[0035] The first optical flow information refers to the two-dimensional vector field of the motion trajectory and speed of all pixels in the image over time. When calculating the first optical flow information between adjacent images, a sparse optical flow algorithm or a dense optical flow method can be used to calculate the optical flow field. Sparse optical flow methods are such as the Lucas-Kanade optical flow method, and dense optical flow methods are such as the Gunnar Farneback optical flow method.

[0036] S120, determining at least two reference images and at least one non-reference image matching each reference image from the image sequence;

[0037] Reference images and non-reference images are selected from an image sequence. All or part of the images in the image sequence can be determined as reference images based on the shooting time of the images, image parameters of the images, etc., and non-reference images matching each reference image can be determined based on the shooting time and image parameters of the determined reference images.

[0038] Because multiple frames of images taken continuously for the same scene constitute an image sequence, there will be a certain degree of content deviation between the images in the image sequence on the basis of high similarity. Several frames of images taken before the reference image and several frames of images taken after the reference image can be selected as non-reference images matching the reference image. An image in an image sequence can be a reference image or a non-reference image relative to another reference image, that is, the reference image and the non-reference image are relative to each other, and cannot distinguish between the images in the image sequence.

[0039] For example, see Figure 2, the image sequence includes P frame images taken continuously, and the three frame images before the reference image are determined to be non-reference images. First, the i-th frame image 214 is determined to be the reference image, and the i-1-th frame image 213, the i-2-th frame image 212, and the i-3-th frame image 211 are determined to be non-reference images matching the i-th frame image 214. The i+1-th frame image 215 is determined to be the reference image, and the i-th frame image 214, the i-1-th frame image 213, and the i-2-th frame image 212 are determined to be non-reference images matching the i+1-th frame image 215. That is, the i-th frame image can be a reference image, and when the i+1-th frame image is a reference image, the i-th frame image can also be a non-reference image matching the i+1-th frame image.

[0040] S130, calculating second optical flow information between the matched reference image and the non-reference image based on the first optical flow information;

[0041] The first optical flow information can represent the optical flow change between images that are adjacent in shooting time. Therefore, if the optical flow change between adjacent images is known, the optical flow change between any adjacent or non-adjacent images can be calculated, that is, the second optical flow information is generated.

[0042] S140, performing optical flow registration on each non-reference image based on the second optical flow information to generate an aligned image;

[0043] There is a certain degree of content deviation between the images in the image sequence, so there is a content deviation between the reference image and the non-reference image. The influence of content deviation can be eliminated by performing optical flow registration on the images in the image sequence. By performing optical flow registration on different images, the positions of the pixels representing the same object on different images can be aligned, so that the misplaced contents on different images can be aligned. Among them, an image with a clarity exceeding a preset clarity threshold can be selected from the image sequence as a reference image, and an image with a clarity lower than a preset clarity threshold can be selected as a non-reference image, and the non-reference image is optically flow registered with the reference image, thereby avoiding the clear image from being registered with the unclear image, and further fusion processing is performed to obtain an unclear fused image.

[0044] S150, performing fusion processing on the matched reference image and the optical flow aligned image respectively to generate a fused image corresponding to the reference image.

[0045] The reference image and the matched registered optical flow aligned image are fused so that the fused image can obtain the features in each image. The fused image has richer content than the reference image and the optical flow aligned image, thus improving the quality of the fused image corresponding to the reference image. The fusion process can be implemented by using any of the Bayesian decision fusion method, the feature transform (SIFT) fusion method, and the fuzzy logic decision fusion method.

[0046] In the present application, the first optical flow information between images with adjacent shooting times in a continuously shot image sequence is calculated; based on the first optical flow information, the second optical flow information between the matching reference image and the non-reference image is calculated, thereby eliminating the need to directly calculate the optical flow information between any reference image and the non-reference image, reducing the amount of calculation and improving the image processing efficiency; based on the second optical flow information, optical flow registration is performed on each non-reference image to generate an aligned image; the matching reference image and the optical flow aligned image are fused separately to generate a fused image corresponding to the reference image, thereby avoiding misalignment of contents on different images that may cause distortion of the fused image generated by the final image fusion, and improving the image quality of the fused image.

[0047] In some embodiments, S130 includes:

[0048] S210, when the matched reference image and the non-reference image are images with non-adjacent shooting times, determining n first images whose shooting times are between the matched reference image and the non-reference image in the image sequence, where n is a positive integer and n≥1;

[0049] S220, calculating the sum of first optical flow information between the reference image, the n-frame first image, and images in the non-reference image that are adjacent in shooting time, to obtain second optical flow information.

[0050] The reference image and the non-reference image are non-adjacent images, which can be understood as other images between the reference image and the non-reference image in the image sequence, and the other images are the first images. All the first optical flow information between the reference image, the n-frame first image and the non-reference image is added together, and the second optical flow information obtained can represent the optical flow change between the reference image and the non-reference image.

[0051] Exemplarily, the image sequence includes P frames of images taken continuously, and the images located M frames before and N frames after the reference image are determined to be non-reference images. The i-th frame is determined to be the reference image, and the iM-th frame... to the i-1-th frame, and the i+1-th frame to the i+N-th frame are determined to be non-reference images that match the i-th frame image. First, the first optical flow information f between each image is calculated, then the second optical flow information between the iM-th frame image and the i-th frame is f(iM,iM-1)+f(iM-1,iM-2)+.....+f(i-1,i), the second optical flow information between the iM-th frame image and the i-th frame is f(iM-1,iM-2)+f(iM-2,iM-3)+.....+f(i-1,i), and so on.

[0052] By precalculating the first optical flow information between images with adjacent shooting times, when calculating the second optical flow information, the sum of the first optical flow information corresponding to n frames of first images located between the reference image and the non-reference image can be calculated, and when calculating the optical flow information between multiple reference images and non-reference images, the same first optical flow information can be used multiple times, thereby reducing the direct calculation of the optical flow information between multiple reference images and non-reference images.

[0053] In some embodiments, when the matched reference image and the non-reference image are images captured at adjacent times, the first optical flow information between the reference image and the non-reference image is set as the second optical flow information.

[0054] Exemplarily, the second optical flow information between the i-1th frame image and the i-th frame is f(i-1,i), and f(i-1,i) is the first optical flow information.

[0055] In some embodiments, S130 includes:

[0056] S310, when the i-th frame image is a reference image and the ij-th frame image is a non-reference image matching the i-th frame image, based on the first optical flow information, calculating the second optical flow information between the i-th frame image being the reference image and the ij-th frame image, where i and j are both positive integers, i>j;

[0057] S320, when the i+1th frame image is a reference image and the ijth frame image is a non-reference image matching the i+1th frame image, the second optical flow information between the i+1th frame image and the ijth frame image is calculated based on the first optical flow information between the i-th frame image and the i+1th frame image, and the second optical flow information between the i-th frame image and the ij-th frame image.

[0058] In S130, multiple reference images may be set, and the second optical flow information between each reference image and the corresponding matching non-reference image may be calculated in sequence. When the second optical flow information is calculated later, the previously calculated second optical flow information may be used to reduce the calculation.

[0059] Exemplarily, the image sequence includes P frames of images taken continuously, and the images located M frames before and N frames after the reference image are determined to be non-reference images. The i-th frame is determined to be the first reference image, and the iM-th frame... to the i-1-th frame, and the i+1-th frame to the i+N-th frame are determined to be the first non-reference image matching the i-th frame image. M<j≤1, that is, the ij-th frame image is any frame from the i-M+1-th frame... to the i-1-th frame. Based on the first optical flow information f, the second optical flow information between the ij-th frame image and the i-th frame is calculated to be f(ij,i), the second optical flow information between the i-j+1-th frame image and the i-th frame is f(i-j+1,i)....., and so on.

[0060] The i+1th frame is determined as the second reference image, and the i-M+1th frame ..... to the ith frame, and the i+2th frame to the i+1+Nth frame are determined as the second non-reference images matching the i+1th frame image. The second optical flow information f(i-j+1,i), f(i-j+2,i), etc. calculated previously can be obtained, and the second optical flow information f(i-j+1,i+1) between the i+j+1th frame image and the i+1th frame image can be obtained by calculating the sum of the second optical flow information f(i-j+1,i) and the first optical flow information f(i,i+1).

[0061] In some embodiments, S120 includes:

[0062] S410, sequentially determining images in an image sequence as reference images, and determining non-reference images matching each reference image from the image sequence;

[0063] S150 includes:

[0064] S420, performing fusion processing on the matched reference image and the optical flow aligned image respectively to generate fused images corresponding to the images in the image sequence respectively.

[0065] In this embodiment, all images in the image sequence are taken as reference images in turn and image processing is performed, so that a fused image corresponding to each image can be obtained. For example, a first input with a pressing time of 5 seconds is received; in response to the first input, 30 frames of images are collected per second, and an image sequence including 150 frames of images is obtained. The 150 frames of images are taken as reference images in turn, and the second optical flow information between each reference image and the matching non-reference image is calculated, and registration processing is performed. Each reference image is fused with the registered non-reference image to obtain 150 frames of fused images, which correspond one-to-one to the 150 frames of images taken.

[0066] By taking all images in the image sequence as reference images in turn and performing image processing, a fused image corresponding to each image can be obtained, so that all images taken continuously can be fused to provide a clearer fused image for users to browse and select.

[0067] In some embodiments, S120 includes:

[0068] S510, determining a third image as a reference image from the image sequence;

[0069] S520, determining that M consecutive images before the third image in the image sequence are non-reference images matching the third image;

[0070] S530: Set N consecutive images following the third image in the image sequence as non-reference images matching the third image.

[0071] The shooting time of the M images is earlier than the shooting time of the third image, the shooting time of the N images is later than the shooting time of the third image, and both M and N are positive integers.

[0072] The third image may be any image in the image sequence. In S120, the images in the image sequence may be determined as reference images in sequence.

[0073] In S520, when the number of images before the third image in the image sequence is greater than or equal to M images, the M consecutive images before the third image are set as non-reference images matching the third image. When the number of images before the third image in the image sequence is less than M images, all images before the third image are set as non-reference images matching the third image. In S530, when the number of images after the third image in the image sequence is greater than or equal to N images, the N consecutive images after the third image are set as non-reference images matching the third image. When the number of images after the third image in the image sequence is less than N images, all images after the third image are set as non-reference images matching the third image. M and N may be the same or different.

[0074] The shooting time of the continuous M images, the continuous N images and the third image is close, so that the content displayed in the continuous M images, the continuous N images and the third image is highly similar, thereby improving the image quality of the fused image after fusion processing. Optionally, M is 3, 4, 5 or 6. Optionally, N is 3, 4, 5 or 6.

[0075] In some other embodiments, S120 includes:

[0076] S540, determining a third image as a reference image from the image sequence;

[0077] S550, setting M images before the third image in the image sequence as non-reference images matching the third image, the M images being images with image quality greater than a preset threshold;

[0078] S560: Set N images after the third image in the image sequence as non-reference images matching the third image, and the N images are images with image quality greater than a preset threshold.

[0079] In this embodiment, the M images, the consecutive N images and the third image are not only shot at similar times, but are also screened by image quality, so that the contents displayed in the M images, the N images and the third image are highly similar and have good image quality, thereby improving the image quality of the fused image after fusion processing.

[0080] In some embodiments, the same reference image is matched with at least two non-reference images; S150 includes:

[0081] S610, extracting image features for each optical flow aligned image that matches a fourth image in at least two reference images;

[0082] S620, compensating each optical flow alignment image matched with the fourth image based on the image feature;

[0083] S630, determining a transformation matrix between each optical flow aligned image and a matching fourth image;

[0084] S640, performing image registration on each optical flow aligned image matching the fourth image based on the transformation matrix to generate a transformed image;

[0085] S650: Fusing the matched fourth image and the transformed image to generate a fused image corresponding to the reference image.

[0086] The fourth image is any one of at least two reference images. The image features of each non-reference image matching the same reference image can be extracted through a pre-trained convolution module, and then each optical flow alignment image matching the fourth image is compensated based on the image features, so that the features of each non-reference image across the reference image are compensated, the alignment accuracy is improved, and the display effect of the image after image feature compensation is improved.

[0087] Features can be extracted from the reference image and each non-reference image respectively. The extracted features can be corner points. The extracted features are matched to obtain a transformation matrix between the reference image and the non-reference image. The non-reference image is transformed according to the transformation matrix to obtain a transformed image. The transformed image and the fourth image are then combined to generate a fused image corresponding to the fourth image.

[0088] See also Figure 3, determine the i-th frame image 312 as the reference image, and determine the i-1-th frame image 311, the i+1-th frame image 313, and the i+2-th frame image 314 as non-reference images matching the i-th frame image 312. The i-1th frame image 311, the i+1th frame image 313, and the i+2th frame image 314 are optically aligned with the i-1th frame image 312 to obtain an optical flow aligned image; the optical flow aligned images corresponding to the i-1th frame image 311, the i+1th frame image 313, and the i+2th frame image 314 are used to extract image features, and the optical flow aligned images corresponding to the i-1th frame, the i+1th frame, and the i+2th frame are compensated based on the image features to achieve feature propagation; the transformation matrix between the optical flow aligned images corresponding to the i-1th frame image 311, the i+1th frame image 313, and the i+2th frame image 314 and the i-1th frame image 312 is determined, and the images are aligned through the transformation matrix to generate a transformed image; the transformed images corresponding to the i-1th frame image 311, the i+1th frame image 313, and the i+2th frame image 314 and the i-1th frame image 312 are fused to obtain a fused image 315.

[0089] In some embodiments, S110 includes:

[0090] S710, acquiring an image sequence obtained by continuous shooting, and identifying an object region and a background region of each image in the image sequence;

[0091] S720, calculating first optical flow information between object regions in images with adjacent shooting times in the image sequence;

[0092] S150 includes:

[0093] S730, fusing the object area of ​​the matched aligned image and the reference image to generate a first fused image, and splicing the first fused image and the background area of ​​the reference image to generate a fused image corresponding to the reference image.

[0094] In S710, each image in the image sequence may be processed for recognition, and an object region and a background region may be obtained. Since a moving object may be displayed differently in each image, the object region may be a region of the moving object. The background region may be a static object, scenery, etc., or a person, animal, etc. that is relatively still relative to a shooting module that shoots the image sequence.

[0095] In S720, the first optical flow information between the object areas in the images with adjacent shooting times in the image sequence is calculated, thereby eliminating the need to calculate the first optical flow information for the background area. Subsequently, the need to calculate the second optical flow information for the background area can be eliminated, and the background area of ​​the non-reference image and the background area of ​​the reference image can be aligned, thereby improving image processing efficiency.

[0096] Optionally, S130 includes:

[0097] Based on the first optical flow information, calculating second optical flow information between the object region of the matched reference image and the object region of the non-reference image;

[0098] S140 includes:

[0099] Based on the second optical flow information, optical flow registration is performed on the object region of the non-reference image to the object region of the reference image to generate an aligned image;

[0100] In S730, the object area of ​​the matched aligned image and the reference image is fused to generate a first fused image, so that the first fused image has the content of multiple frames of images. The first fused image and the background area of ​​the reference image are spliced ​​to generate a fused image corresponding to the reference image, so that the fused image directly uses the background area in the reference image, and the background area does not need to be processed by optical flow alignment, affine transformation, etc., so as to improve image processing efficiency.

[0101] For example, see Figure 4 In the continuously captured first frame image 411, second frame image 412, and third frame image 413, the vehicle moves from the left side of the image to the right side. The second frame image 412 is determined as a reference image, and the first frame image 411 and the third frame image 413 are determined as non-reference images matching the second frame image. The object region and the background region in the first frame image 411, the second frame image 412, and the third frame image 413 are identified to obtain the object region 421, the object region 422, the object region 423, and the background region 431 of the second frame image 412 respectively; the first optical flow information of the object region in the first frame image 411, the second frame image 412, and the third frame image 413 is calculated, and the second optical flow information is calculated based on the first optical flow information, and then the optical flow registration and fusion processing are performed based on the second optical flow information to generate the first fused image 432 corresponding to the second frame image 412, and the first fused image 432 is spliced ​​with the background region 431 to generate the fused image 44 corresponding to the second frame image 412. The fused image 44 not only integrates the features of the object region in the first frame image 411, the second frame image 412, and the third frame image 413, so that the object region is clear and rich in content, but also eliminates the need to perform optical flow alignment, affine transformation, and other processing on the background region, thereby improving the image processing efficiency.

[0102] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0103] See also Figure 5 , Figure 5 is a structural diagram of an image processing device provided by another embodiment of the present application. Figure 5As shown, the image processing device may include:

[0104] A first optical flow information calculation module 401, used to calculate first optical flow information between images with adjacent shooting times in a sequence of continuously shot images;

[0105] A reference image determination module 402, configured to determine at least two reference images and at least one non-reference image matching each reference image from an image sequence;

[0106] A second optical flow information calculation module 403, used to calculate second optical flow information between the matched reference image and the non-reference image based on the first optical flow information;

[0107] A registration module 404, configured to perform optical flow registration on each non-reference image based on the second optical flow information to generate an aligned image;

[0108] The fusion module 405 is used to fuse the matched reference image and the optical flow aligned image respectively to generate a fused image corresponding to the reference image.

[0109] In an embodiment of the present application, first optical flow information is calculated between images with adjacent shooting times in a sequence of continuously shot images; based on the first optical flow information, second optical flow information is calculated between matching reference images and non-reference images, thereby eliminating the need to directly calculate optical flow information between any reference image and non-reference image, reducing the amount of calculation and improving image processing efficiency; based on the second optical flow information, optical flow registration is performed on each non-reference image to generate an aligned image; matching reference images and optical flow aligned images are fused separately to generate a fused image corresponding to the reference image, thereby avoiding misalignment of contents on different images that may cause distortion of the fused image generated by the final image fusion, and improving the image quality of the fused image.

[0110] In an optional example, the second optical flow information calculation module 403 includes:

[0111] A first determining unit is used to determine, in the image sequence, n frames of first images whose shooting time is between the matched reference image and the non-reference image, when the matched reference image and the non-reference image are images whose shooting time is not adjacent, where n is a positive integer and n≥1;

[0112] The first calculation unit is used to calculate the sum of first optical flow information between the reference image, the n-frame first image and the images with adjacent shooting times among the non-reference images to obtain second optical flow information.

[0113] In an optional example, the second optical flow information calculation module 403 includes:

[0114] A second calculation unit is used to calculate, based on the first optical flow information, second optical flow information between the i-th frame image being the reference image and the ij-th frame image when the i-th frame image is the reference image and the ij-th frame image is the non-reference image matching the i-th frame image, where i and j are both positive integers, i>j;

[0115] The second calculation unit is also used to calculate the second optical flow information between the i+1th frame image and the ijth frame image based on the first optical flow information of the i-th frame image and the i+1th frame image, and the second optical flow information between the i-th frame image and the ij-th frame image, when the i+1th frame image is a reference image and the ijth frame image is a non-reference image matching the i+1th frame image.

[0116] In an optional example, the reference image determination module 402 is further used to sequentially determine images in the image sequence as reference images, and determine non-reference images matching each reference image from the image sequence;

[0117] The fusion module 405 is further used to perform fusion processing on the matched reference image and the optical flow aligned image respectively, and generate fused images corresponding to the images in the image sequence respectively.

[0118] In an optional example, the reference image determination module 402 includes:

[0119] A third determining unit, configured to determine a third image from the image sequence as a reference image;

[0120] A first setting unit, configured to determine that M consecutive images preceding a third image in the image sequence are non-reference images matching the third image;

[0121] A second setting unit, used to set N consecutive images after the third image in the image sequence as non-reference images matching the third image;

[0122] The shooting time of the M images is earlier than the shooting time of the third image, and the shooting time of the M images is later than the shooting time of the third image. Both M and N are positive integers.

[0123] In an optional example, the same reference image is matched with at least two non-reference images; how the module includes:

[0124] a feature extraction unit, configured to extract image features from each optical flow aligned image matched with a fourth image of at least two reference images;

[0125] a compensation unit, configured to compensate each optical flow aligned image matched with the fourth image based on image features;

[0126] a matrix determination unit, used to determine a transformation matrix between each optical flow aligned image and a matching fourth image;

[0127] A transformation unit, configured to perform image registration on each optical flow aligned image matched with the fourth image based on the transformation matrix to generate a transformed image;

[0128] The fusion unit is used to fuse the matched fourth image and the transformed image to generate a fused image corresponding to the reference image.

[0129] In an optional example, the first optical flow information calculation module 401 includes:

[0130] A recognition unit, used to obtain a sequence of images obtained by continuous shooting, and to recognize an object region and a background region of each image in the sequence of images;

[0131] A third calculation unit, used to calculate first optical flow information between object regions in images with adjacent shooting times in the image sequence;

[0132] The fusion module 405 includes:

[0133] A fusion unit, used for fusing the object area of ​​the matched alignment image and the reference image to generate a first fused image;

[0134] The stitching unit is used to stitch the first fused image and the background area of ​​the reference image to generate a fused image corresponding to the reference image.

[0135] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0136] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0137] The image processing device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0138] Alternatively, if Figure 6 As shown, an embodiment of the present application further provides an electronic device 100, including a processor 101 and a memory 102, wherein the memory 102 stores programs or instructions that can be executed on the processor 101, and when the program or instructions are executed by the processor 101, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0139] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0140] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0141] The electronic device 1000 includes but is not limited to components such as a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, and a processor 1210.

[0142] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1210 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0143] The processor 1210 is used to calculate the first optical flow information between images with adjacent shooting times in a sequence of continuously shot images;

[0144] The processor 1210 is configured to determine at least two reference images and at least one non-reference image matching each reference image from an image sequence;

[0145] The processor 1210 is configured to calculate second optical flow information between a matched reference image and a non-reference image based on the first optical flow information;

[0146] The processor 1210 is used to perform optical flow registration on each non-reference image based on the second optical flow information to generate an aligned image;

[0147] The processor 1210 is used to perform fusion processing on the matched reference image and the optical flow aligned image respectively, and generate a fused image corresponding to the reference image.

[0148] In an embodiment of the present application, first optical flow information is calculated between images with adjacent shooting times in a sequence of continuously shot images; based on the first optical flow information, second optical flow information is calculated between matching reference images and non-reference images, thereby eliminating the need to directly calculate optical flow information between any reference image and non-reference image, reducing the amount of calculation and improving image processing efficiency; based on the second optical flow information, optical flow registration is performed on each non-reference image to generate an aligned image; matching reference images and optical flow aligned images are fused separately to generate a fused image corresponding to the reference image, thereby avoiding misalignment of contents on different images that may cause distortion of the fused image generated by the final image fusion, and improving the image quality of the fused image.

[0149] Optionally, the processor 1210 is configured to determine, in the image sequence, n frames of first images whose shooting time is between the matched reference image and the non-reference image, when the matched reference image and the non-reference image are images whose shooting time is not adjacent, where n is a positive integer and n≥1;

[0150] The processor 1210 is configured to calculate the sum of first optical flow information between the reference image, the n-frame first image, and images of the non-reference image that are adjacent in shooting time, to obtain second optical flow information.

[0151] Optionally, the processor 1210 is configured to calculate, when the i-th frame image is a reference image and the ij-th frame image is a non-reference image matching the i-th frame image, second optical flow information between the i-th frame image being the reference image and the ij-th frame image based on the first optical flow information, where i and j are both positive integers, i>j;

[0152] Processor 1210 is used to calculate the second optical flow information between the i+1th frame image as the reference image and the ijth frame image as the matching non-reference image based on the first optical flow information between the i-th frame image and the i+1th frame image, and the second optical flow information between the i-th frame image and the ij-th frame image, when the i+1th frame image is the reference image and the ijth frame image is the non-reference image matching the i+1th frame image.

[0153] Optionally, the processor 1210 is configured to sequentially determine images in the image sequence as reference images, and determine non-reference images matching each reference image from the image sequence;

[0154] The processor 1210 is used to perform fusion processing on the matched reference image and the optical flow aligned image respectively, and generate fused images corresponding to the images in the image sequence respectively.

[0155] Optionally, the processor 1210 is configured to determine a third image from the image sequence as a reference image;

[0156] The processor 1210 is configured to set the M consecutive images preceding the third image in the image sequence as non-reference images matching the third image;

[0157] The processor 1210 is configured to determine that N consecutive images following the third image in the image sequence are non-reference images matching the third image;

[0158] The shooting time of the M images is earlier than the shooting time of the third image, the shooting time of the N images is later than the shooting time of the third image, and both M and N are positive integers.

[0159] Optionally, the same reference image is matched with at least two non-reference images; the processor 1210 is configured to extract image features from each optical flow aligned image matched with a fourth image of the at least two reference images;

[0160] The processor 1210 is configured to compensate each optical flow aligned image matched with the fourth image based on the image features;

[0161] The processor 1210 is used to determine a transformation matrix between each optical flow aligned image and a matching fourth image;

[0162] The processor 1210 is used to perform image registration on each optical flow aligned image matching the fourth image based on the transformation matrix to generate a transformed image;

[0163] The processor 1210 is configured to perform fusion processing on the matched fourth image and the transformed image to generate a fused image corresponding to the reference image.

[0164] Optionally, the processor 1210 is used to obtain an image sequence obtained by continuous shooting, and identify an object area and a background area of ​​each image in the image sequence;

[0165] The processor 1210 is used to calculate first optical flow information between object regions in images with adjacent shooting times in the image sequence;

[0166] The processor 1210 is used to perform fusion processing on the object area of ​​the matched alignment image and the reference image to generate a first fused image;

[0167] The processor 1210 is used to perform splicing processing on the first fused image and the background area of ​​the reference image to generate a fused image corresponding to the reference image.

[0168] It should be understood that in the embodiment of the present application, the input unit 1204 may include a graphics processor (Graphics Processing Unit, GPU) 12041 and a microphone 12042, and the graphics processor 12041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1206 may include a display panel 12061, and the display panel 12061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1207 includes a touch panel 12071 and at least one of other input devices 12072. The touch panel 12071 is also called a touch screen. The touch panel 12071 may include two parts: a touch detection device and a touch controller. Other input devices 12072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0169] The memory 1209 can be used to store software programs and various data. The memory 1209 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1209 may include a volatile memory or a non-volatile memory, or the memory 1209 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1209 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0170] The processor 1210 may include one or more processing units; optionally, the processor 1210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1210.

[0171] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0172] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0173] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0174] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0175] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0176] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0178] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: Calculating first optical flow information between images with adjacent shooting times in a sequence of continuously shot images; Determining at least two reference images from the image sequence, and at least one non-reference image frame matching each of the reference images; Based on the first optical flow information, calculating second optical flow information between the matched reference image and the non-reference image; Based on the second optical flow information, performing optical flow registration on each of the non-reference images to generate an aligned image; The matched reference image and the optical flow aligned image are fused respectively to generate a fused image corresponding to the reference image.

2. The image processing method according to claim 1, characterized in that: The calculating, based on the first optical flow information, second optical flow information between the matched reference image and the non-reference image comprises: In a case where the matched reference image and the non-reference image are images whose shooting times are not adjacent, determining n frames of first images whose shooting times are between the matched reference image and the non-reference image in the image sequence, where n is a positive integer and n≥1; The sum of first optical flow information between images with adjacent shooting times in the reference image, the n-frame first images, and the non-reference image is calculated to obtain second optical flow information.

3. The image processing method according to claim 2, characterized in that: The calculating, based on the first optical flow information, second optical flow information between the matched reference image and the non-reference image comprises: In a case where the i-th frame image is the reference image and the ij-th frame image is the non-reference image matching the i-th frame image, based on the first optical flow information, calculating the second optical flow information between the i-th frame image and the ij-th frame image, where i and j are both positive integers, i>j; When the i+1th frame image is the reference image and the ijth frame image is the non-reference image matching the i+1th frame image, the second optical flow information between the i+1th frame image and the ijth frame image is calculated based on the first optical flow information between the i-th frame image and the i+1th frame image, and the second optical flow information between the i-th frame image and the ij-th frame image.

4. The image processing method according to claim 1, characterized in that: The step of determining at least two reference images and at least one non-reference image matching each of the reference images from the image sequence comprises: sequentially determining images in the image sequence as reference images, and determining non-reference images matching each of the reference images from the image sequence; The fusing the matched reference image and the optical flow aligned image to generate a fused image corresponding to the reference image comprises: The matched reference image and the optical flow aligned image are fused respectively to generate fused images corresponding to the images in the image sequence.

5. The image processing method according to claim 1, characterized in that: The step of determining at least two reference images and at least one non-reference image matching each of the reference images from the image sequence comprises: Determining a third image as a reference image from the image sequence; Determine that M consecutive images in the image sequence that are located before the third image are non-reference images that match the third image; The continuous N images following the third image in the image sequence are set as non-reference images matching the third image.

6. The image processing method according to claim 1, characterized in that: The same reference image is matched with at least two of the non-reference images; The fusing the matched reference image and the optical flow aligned image to generate a fused image corresponding to the reference image comprises: extracting image features for each of the optical flow aligned images matched with a fourth image of the at least two reference images; compensating each of the optical flow aligned images matched with the fourth image based on the image features; Determining a transformation matrix between each of the optical flow aligned images and the matching fourth image; Based on the transformation matrix, performing image registration on each of the optical flow aligned images matching the fourth image to generate a transformed image; The matched fourth image and the transformed image are fused to generate a fused image corresponding to the reference image.

7. The image processing method according to claim 1, characterized in that: The calculating of first optical flow information between images in a sequence of continuously shot images with adjacent shooting times comprises: Acquire a sequence of images obtained by continuous shooting, and identify an object area and a background area of ​​each image in the sequence of images; Calculating first optical flow information between the object regions in the images in the image sequence that are adjacent in shooting time; The fusing the matched reference image and the optical flow aligned image to generate a fused image corresponding to the reference image comprises: The object area of ​​the matched optical flow aligned image and the reference image is fused to generate a first fused image, and the first fused image and the background area of ​​the reference image are spliced ​​to generate a fused image corresponding to the reference image.

8. An image processing device, characterized in that: include: A first optical flow information calculation module, used to calculate first optical flow information between images with adjacent shooting times in a sequence of continuously shot images; A reference image determination module, used to determine at least two reference images and a plurality of frames of non-reference images matching each of the reference images from the image sequence; A second optical flow information calculation module, configured to calculate second optical flow information between the matched reference image and the non-reference image based on the first optical flow information; A registration module, configured to perform optical flow registration on each of the non-reference images based on the second optical flow information to generate an aligned image; The fusion module is used to fuse the matched reference image and the optical flow aligned image respectively to generate a fused image corresponding to the reference image.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images