Image processing method, device, electronic device and storage medium
By acquiring the initial images of multiple frames and performing pixel fusion, the problem of poor quality of photos taken by smartphones is solved, image details and imaging effects are improved, and more efficient image processing is achieved.
Patent Information
- Application Number
- CN202110590685.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-05-28
AI Technical Summary
The photos taken by smartphones are of poor quality and have low detail recognition. The prior art has problems such as image information loss and poor imaging effects when generating high-resolution images.
By acquiring the initial image of multiple frames, calculating the optical flow, determining the corresponding pixel points between the reference image and the non-reference image, performing pixel fusion, generating a target image, making full use of the original pixel information of the multi-frame image to avoid information loss and error.
Improves imaging effects and image details of the generated target images, providing clearer photo quality.
Smart Images

Figure CN115409696B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art
[0002] With the development of electronic imaging and image processing technology, the camera function based on smart terminals is becoming more and more powerful, but at the same time, users have higher and higher requirements for the quality of photos taken. Due to the limitations of hardware units such as photosensitive element size, lenses and modules in terms of performance, cost, deployment space, etc., the quality of photos taken by smartphones through native camera systems is poor, resulting in poor detail recognition and low signal-to-noise ratio in the photos taken.
[0003] At present, the solution in the prior art is to optimize the target photo through multiple photos after obtaining the target photo taken by the native camera to generate a high-resolution image to improve the image quality. However, the solution in the prior art still has the problem of poor image imaging effect and low detail recognition due to the loss of image information. Summary of the invention
[0004] The embodiments of the present disclosure provide an image processing method, an apparatus, an electronic device, and a storage medium to overcome the problems of poor imaging effect and low image detail recognition when generating high-resolution images.
[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:
[0006] Acquire multiple frames of initial images, wherein the multiple frames of initial images are multiple frames of different images taken for the same scene, and the multiple frames of initial images include a reference image and a non-reference image; calculate the optical flow from the non-reference image to the reference image; determine reference pixels in the non-reference image corresponding to reference pixels in the reference image based on the optical flow from the non-reference image to the reference image; perform pixel fusion on the non-reference image and the reference image by interpolating the reference pixels in the non-reference image with the reference pixels in the reference image to generate a target image.
[0007] In a second aspect, an embodiment of the present disclosure provides an image processing device, including:
[0008] An acquisition unit, configured to acquire a plurality of frames of initial images, wherein the plurality of frames of initial images are a plurality of different frames of images taken for the same scene, and the plurality of frames of initial images include a reference image and a non-reference image;
[0009] A first determining unit, configured to calculate an optical flow from the non-reference image to the reference image;
[0010] A second determining unit, determining a reference pixel point in the non-reference image corresponding to a reference pixel point in the reference image based on an optical flow from the non-reference image to the reference image;
[0011] The generating unit is used to generate a target image by performing pixel fusion on the non-reference image and the reference image by performing interpolation calculation on the reference pixel points in the non-reference image and the reference pixel points in the reference image.
[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;
[0013] The memory stores computer-executable instructions;
[0014] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image processing method described in the first aspect and various possible designs of the first aspect.
[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method described in the first aspect and various possible designs of the first aspect is implemented.
[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method as described in the first aspect and various possible designs of the first aspect.
[0017] The image processing method, device, electronic device and storage medium provided in the present embodiment acquire multiple frames of initial images, wherein the multiple frames of initial images are multiple frames of different images taken for the same scene, and the multiple frames of initial images include a reference image and a non-reference image; calculate the optical flow from the non-reference image to the reference image; determine the reference pixel points in the non-reference image corresponding to the reference pixel points in the reference image based on the optical flow from the non-reference image to the reference image; interpolate the reference pixel points in the non-reference image with the reference pixel points in the reference image, perform pixel fusion on the non-reference image and the reference image to generate a target image. Since in the process of generating the target image, each interpolated pixel point in the target image is obtained by performing pixel fusion using the corresponding reference pixel point in the reference image and the corresponding reference pixel point in the non-reference image, the original pixel information of each frame of the multiple frames of images to be processed is fully utilized, the problem of partial image information loss and error introduction is avoided, and the imaging effect and image details of the target image finally generated are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0019] Figure 1 An application scenario diagram provided for an embodiment of the present disclosure;
[0020] Figure 2 A schematic diagram of a method for generating a high-resolution image in the prior art;
[0021] Figure 3 The image processing method provided by the present invention is shown in FIG. Figure 1 ;
[0022] Figure 4 A schematic diagram of performing interpolation calculation on a reference image provided by an embodiment of the present disclosure;
[0023] Figure 5 A schematic diagram of determining a reference pixel point provided by an embodiment of the present disclosure;
[0024] Figure 6 The image processing method provided by the present invention is shown in FIG. Figure 2 ;
[0025] Figure 7 A flowchart of a process for determining the coordinates of a reference point corresponding to a first interpolated pixel point;
[0026] Figure 8 A schematic diagram of a first interpolation pixel point and its corresponding edge pixel point provided in an embodiment of the present disclosure;
[0027] Fig. 9 A schematic diagram of a first preset range provided in an embodiment of the present disclosure;
[0028] Fig.10 A schematic diagram of a second preset range provided in an embodiment of the present disclosure;
[0029] Fig.11 A structural block diagram of an image processing device provided by an embodiment of the present disclosure;
[0030] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;
[0031] Fig.13A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0033] Figure 1 FIG. 1 is an application scenario diagram provided by an embodiment of the present disclosure. The image processing method provided by an embodiment of the present disclosure can be applied to an intelligent terminal, such as a smart phone. Figure 1 In the application scenario of shooting with a smartphone, the user uses the smartphone to generate multiple frames of low-resolution images through a single shooting operation. During this process, the smartphone displays a prompt message "Improving resolution, please do not move" to prompt the user not to move the phone, so as to collect multiple frames of continuously changing images of the same scene. Afterwards, the smartphone processes the low-resolution image through a built-in program or software, generates a high-resolution image, and displays it, so that the user can get a clearer photo with better image details.
[0034] Figure 2 A schematic diagram of a method for generating a high-resolution image in the prior art is shown in FIG. Figure 2 As shown, in the prior art, in the process of processing multiple low-resolution images to generate high-resolution images, a method usually adopted is to use one of the multiple low-resolution images as a reference frame, calculate the optical flow from the reference frame to the non-reference frame, and then align the images of other non-reference frames to the reference frame through a warp according to the optical flow calculation result and perform interpolation fusion to generate an interpolated image. However, in the process of aligning the other frame images to the reference frame, since the warp algorithm itself will introduce certain errors, after the alignment of the other frame images, the other frame images are no longer the original images, so there is a problem of loss of the original sub-pixel information, resulting in poor imaging effect and low recognition of image details in the high-resolution interpolated image generated by interpolation fusion of the aligned multiple frame images.
[0035] Figure 3 The image processing method provided by the present invention is shown in FIG. Figure 1 The method of this embodiment can be applied in a smart terminal. The image processing method includes:
[0036] Step S101 : acquiring a plurality of frames of initial images, wherein the plurality of frames of initial images are a plurality of frames of different images shot for the same scene, and the plurality of frames of initial images include a reference image and a non-reference image.
[0037] Exemplarily, the intelligent terminal is, for example, a smart phone, and one or more cameras are provided on the smart phone. Acquiring multiple frames of initial images can be to take continuously changing multiple frames of pictures in the same scene through one or more cameras of the smart phone. There is a continuously changing offset between the multiple frames of pictures. More specifically, the action of taking multiple frames of pictures through a smart phone can be triggered by a single shooting operation of the user. For example, the user presses the "take a photo" button, and the smart phone takes 10 photos in a short time (for example, within 0.5 seconds), and saves them to generate multiple frames of initial images. Since the user shakes when holding the mobile phone, there is a certain degree of continuously changing offset between the pictures taken by the smart phone. Of course, the action of taking multiple frames of pictures through a smart phone can also be triggered by multiple shooting operations of the user in a short time. The process of generating multiple frames of initial images is similar and will not be repeated here.
[0038] Furthermore, the multiple frames of initial images include reference images and non-reference images, wherein, illustratively, the reference image may be an image with better imaging effect or higher definition among the multiple frames of initial images, and the reference image and non-reference image may be determined by processing the multiple frames of initial images through a preset algorithm, such as evaluating the definition of each initial image, determining the image to be processed with the highest definition as the reference image, and determining the other images in the initial image as non-reference images. Of course, the reference image and non-reference image in the multiple frames of initial images may also be determined according to preset rules, such as determining the first frame of the multiple frames of initial images or the last frame of the multiple frames of initial images as the reference image, and the process will not be described in detail here.
[0039] Step S102: Calculate the optical flow from the non-reference image to the reference image.
[0040] Optical flow is the projection of the movement of an object in three-dimensional space onto a two-dimensional image plane. It is generated by the relative speed of the object and the camera, and reflects the movement direction and speed of the image pixels corresponding to the object in a very small time. In this embodiment, the optical flow from the non-reference image to the reference image is calculated, that is, the position mapping of each pixel in the non-reference image to the reference image is calculated through a preset optical flow algorithm, that is, for each pixel in the non-reference image, the position of the pixel in the reference image that is most similar to the pixel in the non-reference image is obtained, and the optical flow obtained is the information characterizing the position mapping. Among them, the optical flow algorithm is a prior art known to those skilled in the art, and the specific type of the optical flow algorithm is not limited this time, and it will not be described one by one.
[0041] Step S103: determining reference pixels in the non-reference image corresponding to reference pixels in the reference image based on the optical flow from the non-reference image to the reference image.
[0042] Exemplarily, after determining the optical flow from the non-reference image to the reference image, pixels related to the reference pixel in the reference image, i.e., reference pixels, are determined in other non-reference images, respectively. The reference pixel includes sub-pixel information corresponding to the reference pixel generated by the continuously changing offset between the non-reference images. Therefore, the interpolated pixel determined based on the reference pixel can better express the image details, making the visual performance of the interpolated high-resolution image more realistic.
[0043] For example, Figure 4 A schematic diagram of determining a reference pixel point provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the method for determining the reference pixel corresponding to the reference pixel can be performed by performing optical flow calculation on the reference image to obtain the optical flow from the non-reference image to the reference image, and then mapping the reference pixel to the non-reference image according to the optical flow calculation result, determining the pixel in the non-reference image that is within a preset distance from the mapping point of the reference pixel in the non-reference image, and using the pixel within the preset distance as the reference pixel. The optical flow calculation method is a prior art known to those skilled in the art and will not be described in detail here.
[0044] Step S104 , interpolating the reference pixels in the non-reference image with the reference pixels in the reference image, and performing pixel fusion on the non-reference image and the reference image to generate a target image.
[0045] Specifically, in order to generate a high-resolution target image, it is necessary to fill pixel information for each interpolated pixel in the high-resolution target image generated by interpolating the reference image, wherein the reference pixel corresponding to each interpolated pixel is one or more original pixels in other non-reference images, and the interpolated pixel is generated by pixel fusion of the reference pixel and the corresponding reference pixel, which is the interpolated pixel. The interpolated pixel contains the original sub-pixel information in different non-reference images. Therefore, the high-resolution target image composed of interpolated pixels, compared with the solution in the prior art that aligns the non-reference images by imitation transformation and then performs pixel fusion to obtain the target image, the interpolated image generated by the method provided in this embodiment makes better use of the image information in the initial image, so that the image details of the interpolated image are better.
[0046] Among them, the method of performing pixel fusion on multiple pixel points can be implemented by taking a weighted average of the pixel values of each pixel point, and the process will not be described in detail here.
[0047] In this embodiment, multiple frames of initial images are acquired, wherein the multiple frames of initial images are multiple frames of different images taken for the same scene, and the multiple frames of initial images include a reference image and a non-reference image; an optical flow from the non-reference image to the reference image is calculated; based on the optical flow from the non-reference image to the reference image, a reference pixel point in the non-reference image corresponding to a reference pixel point in the reference image is determined; and a target image is generated by interpolating the reference pixel point in the non-reference image with the reference pixel point in the reference image, pixel fusion is performed on the non-reference image and the reference image, and in the process of generating the target image, each interpolated pixel point in the target image is obtained by pixel fusion using the corresponding reference pixel point in the reference image and the corresponding reference pixel point in the non-reference image. Therefore, the original pixel information of each frame of the multiple frames of images to be processed is fully utilized, the problem of partial image information loss and error introduction is avoided, and the imaging effect and image details of the target image finally generated are improved.
[0048] Figure 5 The image processing method provided in the embodiment of the present disclosure is shown in FIG. Figure 2 In this embodiment, steps S102-S103 are refined, and the image processing method includes:
[0049] Step S201 : acquiring a plurality of frames of initial images that change continuously, wherein the plurality of frames of initial images include a reference image and a non-reference image.
[0050] Step S202, obtaining preset resolution information, and determining reference pixels in the reference image based on the resolution information and size information of the reference image, wherein the reference pixels are interpolated from original pixels in the reference image, and the number of reference pixels is greater than the number of original pixels.
[0051] Exemplarily, based on the reference image, interpolation calculation is performed on the reference image to generate a frame of high-resolution image, and reference pixel points in the reference image are determined according to the high-resolution image. Figure 6 A schematic diagram of performing interpolation calculation on a reference image provided by an embodiment of the present disclosure, such as Figure 6 As shown, exemplarily, the reference image is an image containing 4 pixels and having a resolution of 2X2. After interpolation calculation is performed on the reference image, the determined high-resolution image is an image containing 16 pixels and having a resolution of 4X4. Each pixel in the high-resolution image is a reference pixel. After that, after filling pixel information of each pixel in the high-resolution image, the target image can be generated. After interpolation calculation is performed on the reference image, the position of each reference pixel in the reference image can be determined, and the position of the reference pixel in the reference image is recorded in the interpolation image information. More specifically, the interpolation image information includes the coordinate values of each reference pixel in the interpolation image.
[0052] Among them, the interpolation calculation of the reference image can be determined according to preset resolution information, and can also be dynamically determined according to the number of frames of the initial image. There is no limitation here. After the reference pixels in the reference image are determined according to the resolution information and the size information of the reference image, the number of reference pixels is greater than the number of original pixels, which is equivalent to that after the interpolation calculation of the reference image, the resolution of the corresponding high-resolution image is greater than the resolution of the initial reference image. The specific steps of the image interpolation calculation are existing technologies known to those skilled in the art and will not be repeated here.
[0053] Step S203: Calculate the optical flow from the non-reference image to the reference image.
[0054] Step S204: determining the coordinates of a reference point in the non-reference image corresponding to the image position where the reference pixel point is located based on the optical flow from the non-reference image to the reference image.
[0055] Among them, the reference image includes multiple reference pixel points. By way of example, the optical flow calculation can be performed on each reference pixel point in a cyclic or parallel manner. The following is a detailed introduction to the process of performing optical flow calculation on one of the reference pixel points to determine the coordinates of the reference point corresponding to the image position of the reference pixel point.
[0056] Figure 7A flowchart of the process of determining the coordinates of the reference point corresponding to the image position of the reference pixel point, such as Figure 7 As shown, the process includes:
[0057] S2041, obtaining the coordinates of the reference pixel point.
[0058] S2042: Determine, in the reference image, edge pixel points corresponding to the coordinates of the reference pixel point, wherein the edge pixel point is used to represent at least one pixel point adjacent to the reference pixel point.
[0059] Exemplarily, the coordinates of the reference pixel point can be obtained through the resolution information and the size information of the reference image. The interpolated image information can be generated through the resolution information and the size information of the reference image. The interpolated image information includes the coordinates of each reference pixel point in the reference image. The implementation method of the interpolated image information has been described in detail in the above embodiments and will not be repeated here. Furthermore, in the reference image, according to the coordinates of the reference pixel point, several original pixel points closest to the coordinates, i.e., edge pixel points, are determined. Figure 8 A schematic diagram of a reference pixel point and its corresponding edge pixel point provided in an embodiment of the present disclosure, such as Figure 8 As shown, according to the coordinates of the reference pixel point O, the four original pixel points closest to it, namely A, B, C, and D, are calculated respectively, so that the original pixel points in the four reference images A, B, C, and D are determined as edge pixels. Of course, according to the change of the position of the first reference pixel point O, the number of pixel points adjacent to it may be less than 4, and no further examples will be given here.
[0060] More specifically, the specific calculation method of A, B, C, and D can be shown in formula (1):
[0061]
[0062] Wherein, a and b are the coordinate values (a, b) of the first reference pixel point O, and floor() is a floor rounding function.
[0063] S2043: Determine an offset vector of an edge pixel in the non-reference image based on the optical flow from the non-reference image to the reference image.
[0064] Exemplarily, one implementation method of determining the offset vector of edge pixels in the non-reference image is, for example, to calculate the vector information between the reference image and the non-reference image through an optical flow algorithm, wherein the vector information is used to characterize the offset vector between the pixels in the reference image and the pixels at the corresponding positions in the non-reference image; and to determine the offset vector of the edge pixels in the non-reference image based on the motion vector information. The optical flow algorithm is implemented by the optical flow function OF(), and the general expression of the optical flow calculation is shown in formula (2):
[0065] F t (x,y)=F r ((x,y)+OF(x,y)),t∈[1,N] (2)
[0066] Where (x, y) is the coordinate of the pixel in the reference image, r is the index of the reference frame, t is the index of the non-reference frame, and N is the number of frames of the non-reference image. OF() is the optical flow function, and OF(x, y) is the optical flow calculation result of the pixel. When the optical flow calculation result contains a decimal, the decimal part represents the sub-pixel position mapped to it. The mapping of the pixel in the reference image to the non-reference image can be achieved through formula (2).
[0067] Furthermore, optical flow calculation is performed on edge pixels respectively, the optical flow calculation results of each edge pixel are determined, and weighted average calculation is performed to obtain an average offset, that is, an offset vector. Exemplarily, the calculation method of the offset vector is shown in formula (3):
[0068] offset = medium(OF(A),OF(B),OF(C),OF(D)) (3)
[0069] Among them, offset is the offset vector, medium() is the weighted average calculation, OF(A), OF(B), OF(C), and OF(D) are the optical flow calculation results of edge pixels A, B, C, and D respectively.
[0070] S2044, determining the coordinates of the reference point according to the coordinates of the reference pixel point and the offset vector.
[0071] Furthermore, by performing an offset process on the coordinates of the first reference pixel point according to the offset vector, the reference coordinate point in the non-reference image can be determined. Exemplarily, the implementation method is shown in formula (4):
[0072] center1 =round((a,b)-offset) (4)
[0073] Wherein, a and b are the coordinate values (a, b) of the first reference pixel point O, round() is a rounding function, and the reference coordinate point is center1.
[0074] Step S205: All pixels in the non-reference image that are within a first preset range of the reference point coordinates are determined as reference pixels.
[0075] Fig. 9 A schematic diagram of a first preset range provided in an embodiment of the present disclosure, referring to Fig. 9Further, after determining each reference point coordinate 92 in the non-reference image according to the mapping of the reference pixel point 91 in the non-reference image and the offset vector, the pixel points in each non-reference image that are within a first preset range of the reference point coordinate 92 are determined according to the reference point coordinate 92, and the one or more pixel points are determined as reference pixel points 93. Exemplarily, the first preset range is determined according to the variance of the offset vector of the edge pixel point in the non-reference image and a preset weight coefficient.
[0076] Exemplarily, the method for determining the pixel points in the non-reference image that are within the first preset range of the reference point coordinates can be implemented by windowing, wherein the first preset range is determined by the size of the windowing radius. More specifically, the method for determining the windowing radius is shown in equations (5) and (6):
[0077] r1 = (1 -weight)*r1_th1 + (weight) *r1_th2 (5)
[0078] weight=min(variance(OF(A),OF(B),OF(C),OF(D)),weight_max) / weight_max (6)
[0080] Among them, r1 is the windowing radius, weight is the windowing weight coefficient, variance() is the variance function, min() is the minimum function, r1_th1, r1_th2 and weight_max are preset thresholds, and r1_th2>r1_th1>1, weight_max>0, r1_th1, r1_th2 and weight_max are determined by factors such as the image size, image content, texture information and other factors of the initial image, and can be obtained through experiments. The determination method is not described here.
[0081] In this embodiment, after determining the coordinates of the reference point through optical flow calculation, the first preset range is determined to obtain the reference pixel points within the first preset range. Since the determination process of the first preset range takes into account the influence of factors such as image size, image content, and texture information through parameter setting, the first preset range determined in this way can better determine the reference pixel points with better correlation with the benchmark pixel points, thereby improving the effectiveness of the reference pixel points and improving the image details and image expressiveness of the subsequently synthesized target image.
[0082] Optionally, after step S205, in order to further improve the correlation between the reference pixel and the reference pixel and improve the accuracy of subsequent pixel fusion of the non-reference image and the reference image, the method further includes:
[0083] S206: Determine the mapping coordinates of the reference pixel point in the benchmark image.
[0084] S207: Determine the reference pixel points that fall within the second preset range of the reference pixel points in the mapping coordinates as valid reference pixel points.
[0085] Fig.10 A schematic diagram of a second preset range provided in an embodiment of the present disclosure, referring to Fig.10 , for example, after determining the first preset range with the reference point coordinate as the center point, all the original pixels within the range can be used as reference pixels for subsequent pixel fusion, so as to determine the pixel value of the corresponding interpolation pixel, wherein the target image is composed of interpolation pixels, and the positions of the reference pixel and the interpolation pixel coincide. However, after the reference pixel 93 within the first preset range is mapped to the reference image, the corresponding mapping coordinate 94 will be generated. The reference pixel 91 corresponding to the mapping coordinate 94 is sometimes closer and sometimes farther. For the reference pixel 93 corresponding to the farther mapping coordinate 94, since it deviates far from the interpolation pixel, the image information represented by it is also greatly different from the real image information at the position corresponding to the reference pixel 91. Using such reference pixels for pixel fusion will cause distortion of the interpolation pixel generated in the end, affect the accuracy of pixel fusion, and further affect the image details and image effects of the target image.
[0086] Therefore, in order to further improve the correlation between reference pixels and benchmark pixels and improve the accuracy of subsequent calculation of interpolated pixels, reference pixels with higher correlation with benchmark pixels are determined as valid reference pixels, and the valid reference pixels are used for subsequent pixel fusion.
[0087] Exemplarily, the method of determining the valid reference pixel point is to determine the reference pixel point whose mapping coordinate falls within the second preset range of the reference pixel point as the valid reference pixel point, wherein the method of confirming the second preset range includes:
[0088] Obtain image texture information and / or noise information corresponding to the reference pixel point, wherein the image texture information is used to characterize the image texture structure near the reference pixel point, and the noise information is used to characterize the noise level of the initial image; determine a second preset range based on the image texture information and / or noise information corresponding to the reference pixel point.
[0089] Specifically, the method for determining the second preset range can be implemented by equation (7), equation (8) and equation (9):
[0090] r2 = (1 -weight)*r2_th1 + (weight) *r2_th2 (7)
[0091]
[0092] Vari=variance(F(A),F(B),F(C),F(D)) (9)
[0093] Among them, r2_th1, r2_th2, and scale are preset thresholds, weight is the weight coefficient, r2_th2>r2_th1>=1, scale>1, and the specific value is determined by experimental debugging. variance() is the variance function, F(A), F(B), F(C), and F(D) are the coordinates of edge pixels A, B, C, and D in the reference image, respectively, noise is the noise degree, which is the system input value, and Vari is the second preset range.
[0094] In this embodiment, the second preset range is determined by the image texture information and / or noise information corresponding to the reference pixel point, and effective reference information for subsequent fusion calculation is determined within the second preset range to further improve the image details and image effects of the generated target image.
[0095] Step S208, performing weighted calculation on the pixel values of valid reference pixels among the reference pixels corresponding to the benchmark pixel, determining the pixel values of the interpolated pixels, and constructing the target image based on the pixel values of the interpolated pixels.
[0096] Exemplarily, after determining the pixel value of the valid reference pixel point, the pixel value of the valid reference pixel point is weightedly calculated to determine the pixel value of the interpolation pixel point corresponding to the corresponding reference pixel point. Exemplarily, the method for determining the pixel value of the interpolation pixel point can be implemented by formula (10):
[0097]
[0098] Among them, O i represents the i-th interpolation pixel in the target image, P m is the pixel value of the mth valid reference pixel, W m is the weight of the mth valid reference pixel, and n is the number of valid reference pixels. Of course, it can be understood that if step S205 is not performed, that is, the fusion calculation is directly performed through the reference pixels to determine the interpolation pixel corresponding to the reference pixel, in this case P m and W m They represent the pixel value and weight of the reference pixel respectively, and n is the number of reference pixels.
[0099] The target image can be generated by determining the pixel value of each interpolation pixel in the target image through parallel processing or polling sequential processing.
[0100] In this embodiment, the implementation method of steps S201 and S203 is the same as that of the present application. Figure 3 The implementation methods of steps S101 and S102 in the illustrated embodiment are the same and will not be described in detail here.
[0101] Corresponding to the image processing method of the above embodiment, Fig.11 This is a structural block diagram of an image processing device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown.
[0102] Reference Fig.11 , the image processing device 3 comprises:
[0103] An acquisition unit 31 is used to acquire multiple frames of initial images, wherein the multiple frames of initial images are multiple frames of different images taken for the same scene, and the multiple frames of initial images include reference images and non-reference images;
[0104] A first determining unit 32, configured to calculate an optical flow from a non-reference image to a reference image;
[0105] A second determining unit 33 determines a reference pixel point in the non-reference image corresponding to a reference pixel point in the reference image based on an optical flow from the non-reference image to the reference image;
[0106] The generating unit 34 is configured to generate a target image by performing pixel fusion on the non-reference image and the reference image by performing interpolation calculation on the reference pixel points in the non-reference image and the reference pixel points in the reference image.
[0107] In one embodiment of the present disclosure, the second determination unit 33 is specifically used to: determine the reference point coordinates in the non-reference image corresponding to the image position of the reference pixel point based on the optical flow from the non-reference image to the reference image; determine all pixel points in the non-reference image that are within a first preset range of the reference point coordinates as reference pixel points.
[0108] In one embodiment of the present disclosure, when the second determination unit 33 determines the coordinates of the reference point in the non-reference image corresponding to the image position of the reference pixel point based on the optical flow from the non-reference image to the reference image, it is specifically used to: obtain the coordinates of the reference pixel point; determine the edge pixel point corresponding to the coordinates of the reference pixel point in the reference image, wherein the edge pixel point is used to characterize at least one pixel point adjacent to the reference pixel point; determine the offset vector of the edge pixel point in the non-reference image based on the optical flow from the non-reference image to the reference image; and determine the coordinates of the reference point according to the coordinates of the reference pixel point and the offset vector.
[0109] In one embodiment of the present disclosure, the second determining unit 33 is further configured to determine a first preset range of reference point coordinates according to a variance of an offset vector of an edge pixel point in a non-reference image and a preset weight coefficient.
[0110] In one embodiment of the present disclosure, when the second determination unit 33 determines the offset vector of the edge pixel point in the non-reference image based on the optical flow from the non-reference image to the reference image, it is specifically used to: calculate the vector information between the reference image and the non-reference image through the optical flow algorithm, wherein the vector information is used to characterize the offset vector between the pixel points in the reference image and the pixel points at the corresponding positions in the non-reference image; determine the offset vector of the edge pixel point in the non-reference image according to the motion vector information.
[0111] In one embodiment of the present disclosure, the second determination unit 33 is further used to: determine the mapping coordinates of the reference pixel point in the reference image; determine the reference pixel point in the mapping coordinates that falls within the second preset range of the reference pixel point as a valid reference pixel point; when the generation unit 34 performs pixel fusion on the non-reference image and the reference image to generate a target image, it is specifically used to: perform weighted calculation on the pixel values of the valid reference pixel points in the reference pixel points corresponding to the reference pixel point, and determine the pixel value of the interpolation pixel point corresponding to the reference pixel point; and construct the target image based on the pixel values of the interpolation pixel points.
[0112] In one embodiment of the present disclosure, the acquisition unit 31 is further used to: acquire image texture information and / or noise information corresponding to the reference pixel point, wherein the image texture information is used to characterize the image texture structure near the reference pixel point, and the noise information is used to characterize the noise level of the initial image; the second determination unit 33 is further used to: determine the second preset range based on the image texture information and / or noise information corresponding to the reference pixel point.
[0113] In one embodiment of the present disclosure, the acquisition unit 31 is also used to: acquire preset resolution information; determine reference pixel points in the reference image based on the resolution information and size information of the reference image, where the reference pixel points are interpolated from the original pixel points in the reference image, and the number of reference pixel points is greater than the number of original pixel points.
[0114] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.
[0115] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.12 As shown, the electronic device 4 includes at least one processor 401 and a memory 402;
[0116] Memory 402 stores computer executable instructions;
[0117] At least one processor 401 executes the computer-executable instructions stored in the memory 402, so that at least one processor 401 performs the following Figure 3-Figure 10 The screen sharing method in the illustrated embodiment.
[0118] The processor 401 and the memory 402 are connected via a bus 403 .
[0119] For related instructions, please refer to Figure 3-Figure 10 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.
[0120] refer to Fig.13 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0121] like Fig.13 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0122] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.13 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0123] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0124] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0125] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.
[0126] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.
[0127] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0128] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0129] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".
[0130] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0131] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0132] In a first aspect, according to one or more embodiments of the present disclosure, there is provided an image processing method, comprising:
[0133] Acquire multiple frames of initial images, wherein the multiple frames of initial images are multiple frames of different images taken for the same scene, and the multiple frames of initial images include a reference image and a non-reference image; calculate the optical flow from the non-reference image to the reference image; determine reference pixels in the non-reference image corresponding to reference pixels in the reference image based on the optical flow from the non-reference image to the reference image; perform pixel fusion on the non-reference image and the reference image by interpolating the reference pixels in the non-reference image with the reference pixels in the reference image to generate a target image.
[0134] According to one or more embodiments of the present disclosure, based on the optical flow from the non-reference image to the reference image, determining the reference pixel points in the non-reference image corresponding to the reference pixel points in the reference image, including: determining the reference point coordinates in the non-reference image corresponding to the image position where the reference pixel points are located based on the optical flow from the non-reference image to the reference image; and determining all pixel points in the non-reference image that are within a first preset range of the reference point coordinates as reference pixel points.
[0135] According to one or more embodiments of the present disclosure, based on the optical flow from the non-reference image to the reference image, the coordinates of the reference point in the non-reference image corresponding to the image position where the reference pixel is located are determined, including: obtaining the coordinates of the reference pixel; determining, in the reference image, an edge pixel corresponding to the coordinates of the reference pixel, wherein the edge pixel is used to characterize at least one pixel adjacent to the reference pixel; based on the optical flow from the non-reference image to the reference image, determining the offset vector of the edge pixel in the non-reference image; and determining the coordinates of the reference point according to the coordinates of the reference pixel and the offset vector.
[0136] According to one or more embodiments of the present disclosure, the method further includes: determining a first preset range of the reference point coordinates based on the variance of the offset vector of the edge pixel point in the non-reference image and a preset weight coefficient.
[0137] According to one or more embodiments of the present disclosure, based on the optical flow from the non-baseline image to the base line image, determining the offset vector of the edge pixel point in the non-baseline image includes: calculating vector information between the base line image and the non-baseline image through an optical flow algorithm, wherein the vector information is used to characterize the offset vector between the pixel point in the base line image and the pixel point at the corresponding position in the non-baseline image; and determining the offset vector of the edge pixel point in the non-baseline image based on the vector information.
[0138] According to one or more embodiments of the present disclosure, the method also includes: determining the mapping coordinates of the reference pixel point in the benchmark image; determining the reference pixel point in the mapping coordinates that falls within a second preset range of the benchmark pixel point as a valid reference pixel point; performing pixel fusion on the non-baseline image and the benchmark image to generate a target image, including: performing weighted calculation on the pixel values of the valid reference pixel points in the reference pixel points corresponding to the benchmark pixel point to determine the pixel value of the interpolation pixel point corresponding to the benchmark pixel point; and constructing the target image based on the pixel values of the interpolation pixel points.
[0139] According to one or more embodiments of the present disclosure, the method further includes: obtaining image texture information and / or noise information corresponding to the reference pixel point, wherein the image texture information is used to characterize the image texture structure near the reference pixel point, and the noise information is used to characterize the noise level of the image to be processed; and determining the second preset range based on the image texture information and / or noise information corresponding to the reference pixel point.
[0140] According to one or more embodiments of the present disclosure, before determining the reference pixel points in the non-reference image corresponding to the reference pixel points in the reference image, it also includes: obtaining preset resolution information; determining the reference pixel points in the reference image based on the resolution information and the size information of the reference image, the reference pixel points being obtained by interpolating the original pixel points in the reference image, and the number of the reference pixel points is greater than the number of the original pixel points.
[0141] In a second aspect, according to one or more embodiments of the present disclosure, there is provided an image processing apparatus, comprising:
[0142] An acquisition unit, configured to acquire a plurality of frames of initial images, wherein the plurality of frames of initial images are a plurality of different frames of images taken for the same scene, and the plurality of frames of initial images include a reference image and a non-reference image;
[0143] A first determining unit, configured to calculate an optical flow from the non-reference image to the reference image;
[0144] A second determining unit, determining a reference pixel point in the non-reference image corresponding to a reference pixel point in the reference image based on an optical flow from the non-reference image to the reference image;
[0145] The generating unit is used to generate a target image by performing pixel fusion on the non-reference image and the reference image by performing interpolation calculation on the reference pixel points in the non-reference image and the reference pixel points in the reference image.
[0146] According to one or more embodiments of the present disclosure, the second determination unit is specifically used to: determine the reference point coordinates in the non-reference image corresponding to the image position of the reference pixel point based on the optical flow from the non-reference image to the reference image; and determine all pixel points in the non-reference image that are located within a first preset range of the reference point coordinates as reference pixel points.
[0147] According to one or more embodiments of the present disclosure, when the second determination unit determines the coordinates of the reference point in the non-reference image corresponding to the image position of the reference pixel point based on the optical flow from the non-reference image to the reference image, the second determination unit is specifically used to: obtain the coordinates of the reference pixel point; determine the edge pixel point corresponding to the coordinates of the reference pixel point in the reference image, wherein the edge pixel point is used to characterize at least one pixel point adjacent to the reference pixel point; determine the offset vector of the edge pixel point in the non-reference image based on the optical flow from the non-reference image to the reference image; and determine the reference point coordinates based on the coordinates of the reference pixel point and the offset vector.
[0148] According to one or more embodiments of the present disclosure, the second determination unit is further used to determine a first preset range of reference point coordinates based on a variance of an offset vector of an edge pixel point in a non-reference image and a preset weight coefficient.
[0149] According to one or more embodiments of the present disclosure, when the second determination unit determines the offset vector of the edge pixel point in the non-baseline image based on the optical flow from the non-baseline image to the base image, it is specifically used to: calculate the vector information between the base image and the non-baseline image through an optical flow algorithm, wherein the vector information is used to characterize the offset vector between the pixel points in the base image and the pixel points at corresponding positions in the non-baseline image; and determine the offset vector of the edge pixel point in the non-baseline image based on the motion vector information.
[0150] According to one or more embodiments of the present disclosure, the second determination unit is further used to: determine the mapping coordinates of the reference pixel point in the reference image; determine the reference pixel point in the mapping coordinates that falls within a second preset range of the reference pixel point as a valid reference pixel point; when the generation unit performs pixel fusion on the non-reference image and the reference image to generate a target image, the generation unit is specifically used to: perform weighted calculation on the pixel values of the valid reference pixel points in the reference pixel points corresponding to the reference pixel point to determine the pixel value of the interpolation pixel point corresponding to the reference pixel point; and construct a target image based on the pixel values of the interpolation pixel points.
[0151] According to one or more embodiments of the present disclosure, the acquisition unit is further used to: acquire image texture information and / or noise information corresponding to the reference pixel point, wherein the image texture information is used to characterize the image texture structure near the reference pixel point, and the noise information is used to characterize the noise level of the initial image; the second determination unit is further used to: determine the second preset range based on the image texture information and / or noise information corresponding to the reference pixel point.
[0152] According to one or more embodiments of the present disclosure, the acquisition unit is also used to: acquire preset resolution information; determine reference pixel points in the reference image based on the resolution information and size information of the reference image, wherein the reference pixel points are obtained by interpolating original pixel points in the reference image, and the number of the reference pixel points is greater than the number of original pixel points.
[0153] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;
[0154] The memory stores computer-executable instructions;
[0155] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image processing method described in the first aspect and various possible designs of the first aspect.
[0156] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the image processing method described in the first aspect and various possible designs of the first aspect is implemented.
[0157] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method as described in the first aspect and various possible designs of the first aspect.
[0158] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0159] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0160] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. An image processing method, characterized in that: include: Acquire multiple frames of initial images, wherein the multiple frames of initial images are multiple frames of different images taken for the same scene, and the multiple frames of initial images include reference images and non-reference images; Calculating an optical flow from the non-reference image to the reference image; Determining, based on an optical flow from the non-reference image to the reference image, a reference pixel point in the non-reference image corresponding to a reference pixel point in the reference image; By performing interpolation calculation on the reference pixel points in the non-reference image and the reference pixel points in the reference image, pixel fusion is performed on the non-reference image and the reference image to generate a target image; Determining a reference pixel point in the non-reference image corresponding to a reference pixel point in the reference image based on an optical flow from the non-reference image to the reference image includes: Determine, based on the optical flow from the non-reference image to the reference image, the coordinates of a reference point in the non-reference image corresponding to the image position where the reference pixel point is located; Determine all pixel points in the non-reference image that are within a first preset range of the reference point coordinates as reference pixel points; The first preset range is determined based on the variance of the offset vector of the edge pixel point in the non-reference image and a preset weight coefficient, wherein the edge pixel point is used to characterize at least one pixel point adjacent to the reference pixel point.
2. The method according to claim 1, characterized in that Determining, based on the optical flow from the non-reference image to the reference image, the coordinates of a reference point in the non-reference image corresponding to the image position where the reference pixel point is located, comprises: Get the coordinates of the reference pixel point; In the reference image, determining edge pixel points corresponding to the coordinates of the reference pixel points; Determining an offset vector of the edge pixel point in the non-reference image based on an optical flow from the non-reference image to the reference image; The coordinates of the reference point are determined according to the coordinates of the reference pixel point and the offset vector.
3. The method according to claim 2, characterized in that Determining an offset vector of the edge pixel point in the non-reference image based on an optical flow from the non-reference image to the reference image includes: Calculate vector information between the reference image and the non-reference image by using an optical flow algorithm, wherein the vector information is used to represent an offset vector between a pixel point in the reference image and a pixel point at a corresponding position in the non-reference image; An offset vector of the edge pixel point in the non-reference image is determined according to the vector information.
4. The method according to claim 1, characterized in that: The method further comprises: Determining the mapping coordinates of the reference pixel point in the reference image; Determine a reference pixel point in the mapping coordinates that falls within a second preset range of the reference pixel point as a valid reference pixel point; Performing pixel fusion on the non-reference image and the reference image to generate a target image includes: Performing weighted calculation on pixel values of valid reference pixels among the reference pixels corresponding to the reference pixel, to determine the pixel value of the interpolation pixel corresponding to the reference pixel; A target image is constructed according to the pixel values of the interpolated pixels.
5. The method according to claim 4, characterized in that The method further comprises: Acquire image texture information and / or noise information corresponding to the reference pixel point, wherein the image texture information is used to characterize the image texture structure near the reference pixel point, and the noise information is used to characterize the noise level of the image to be processed; The second preset range is determined according to image texture information and / or noise information corresponding to the reference pixel point.
6. The method according to any one of claims 1 to 5, characterized in that: Before determining a reference pixel point in the non-reference image corresponding to a reference pixel point in the reference image, the method further includes: Get the preset resolution information; According to the resolution information and the size information of the reference image, reference pixel points in the reference image are determined, where the reference pixel points are obtained by interpolating original pixel points in the reference image, and the number of the reference pixel points is greater than the number of original pixel points.
7. An image processing device, characterized in that: include: An acquisition unit, configured to acquire a plurality of frames of initial images, wherein the plurality of frames of initial images are a plurality of different frames of images taken for the same scene, and the plurality of frames of initial images include a reference image and a non-reference image; A first determining unit, configured to calculate an optical flow from the non-reference image to the reference image; A second determining unit, determining a reference pixel point in the non-reference image corresponding to a reference pixel point in the reference image based on an optical flow from the non-reference image to the reference image; a generating unit, configured to perform pixel fusion on the non-reference image and the reference image by performing interpolation calculation on the reference pixel points in the non-reference image and the reference pixel points in the reference image, so as to generate a target image; The second determination unit is specifically used to: determine the reference point coordinates in the non-reference image corresponding to the image position of the reference pixel point based on the optical flow from the non-reference image to the reference image; determine all pixel points in the non-reference image that are located within a first preset range of the reference point coordinates as reference pixels; the first preset range is determined based on the variance of the offset vector of the edge pixel point in the non-reference image and a preset weight coefficient, wherein the edge pixel point is used to characterize at least one pixel point adjacent to the reference pixel point.
8. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image processing method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the image processing method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
