Image processing method, image processing device, electronic device, and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and a computer readable storage medium. The image processing method comprises obtaining a global image, a rolling shutter image and a depth image. The first coordinate of the first projection point of each pixel point in the depth image re-projected onto the global image and the second coordinate of the second projection point re-projected onto the rolling shutter image are obtained. The first motion region of the global image is obtained according to the first coordinate and the pixel value of each pixel point in the global image, and the second motion region of the rolling shutter image is obtained according to the second coordinate and the pixel value of each pixel point in the rolling shutter image. The pixel value of each pixel point in the second motion region is corrected according to the pixel value of each pixel point in the first motion region to obtain a corrected image. In the rolling shutter image, the pixel point matched with each pixel point in the motion region of the corrected image is found out, and the pixel value replacement of each pixel point in the motion region is performed to obtain a replacement image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In recent years, with the continuous development of electronic devices such as mobile phones and cameras, people's demand for images has continued to grow, and the pursuit of high-quality visual experience has also become increasingly higher. For example, users expect electronic devices to be able to obtain high-quality images in different scenarios. Current electronic devices can use inertial measurement units to correct the image content captured by the camera in the electronic device, thereby improving the imaging quality of the electronic device. However, when the object being photographed moves relative to the electronic device, the inertial measurement unit cannot effectively adjust the image content captured by the camera, resulting in poor image correction effect and affecting the imaging quality. Summary of the Invention

[0003] Embodiments of the present application provide an image processing method, an image processing device, an electronic device, and a computer-readable storage medium.

[0004] The image processing method of the embodiment of the present application is used for an electronic device, wherein the electronic device includes a global shutter camera, a rolling shutter camera and a depth camera with a common baseline, and the image processing method includes: obtaining a global image, a rolling shutter image and a depth image, wherein the global image, the rolling shutter image and the depth image are frame-synchronized, the global image is taken by the global shutter camera, the rolling shutter image is taken by the rolling shutter camera, and the depth image is taken by the depth camera, wherein the resolution of the global image is smaller than the resolution of the rolling shutter image and larger than the resolution of the depth image; obtaining the first coordinate of the first projection point of each pixel point in the depth image reprojected onto the global image, and obtaining the first coordinate of each pixel point in the depth image Reprojecting the image to the second coordinates of the second projection point on the rolling curtain image; obtaining the first motion area of ​​the global image according to the first coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the global image, and obtaining the second motion area of ​​the rolling curtain image according to the second coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the rolling curtain image; correcting the pixel value of each pixel in the second motion area according to the pixel value of each pixel in the first motion area to obtain a corrected image; finding the pixel points in the rolling curtain image that match the pixel points in the motion area of ​​the corrected image, and replacing the pixel values ​​of each pixel in the motion area to obtain a replaced image.

[0005] The image processing device of the embodiment of the present application includes a first acquisition module, a second acquisition module, a third acquisition module, a correction module and a replacement module. The first acquisition module is used to acquire a global image, a rolling shutter image and a depth image, the global image, the rolling shutter image and the depth image are frame-synchronized, the global image is taken by the global shutter camera, the rolling shutter image is taken by the rolling shutter camera, and the depth image is taken by the depth camera. The resolution of the global image is smaller than the resolution of the rolling shutter image and larger than the resolution of the depth image; the second acquisition module is used to acquire the first coordinates of the first projection point of each pixel point in the depth image reprojected onto the global image, and acquire the second coordinates of the second projection point of each pixel point in the depth image reprojected onto the rolling shutter image; the third acquisition module is used to The first motion area of ​​the global image is obtained according to the first coordinates corresponding to each pixel point in the depth image and the pixel value of each pixel point in the global image, and the second motion area of ​​the rolling image is obtained according to the second coordinates corresponding to each pixel point in the depth image and the pixel value of each pixel point in the rolling image; the correction module is used to correct the pixel value of each pixel point in the second motion area according to the pixel value of each pixel point in the first motion area to obtain a corrected image; the replacement module is used to find pixel points in the rolling image that match each pixel point in the motion area of ​​the corrected image, and replace the pixel value of each pixel point in the motion area to obtain a replacement image.

[0006] The electronic device of the embodiment of the present application includes a global shutter camera, a rolling shutter camera and a depth camera with a common baseline, the global shutter camera is used to capture a global image, the rolling shutter camera is used to capture the rolling image, and the depth camera is used to capture a depth image; one or more processors, memories and one or more computer programs. One or more of the computer programs are stored in the memory, and when the computer program is executed by the processor, the image processing method of the embodiment of the present application is implemented. The image processing method includes: acquiring a global image, a rolling image and a depth image, the global image, the rolling image and the depth image are frame-synchronized, the global image is captured by the global shutter camera, the rolling image is captured by the rolling shutter camera, the depth image is captured by the depth camera, the resolution of the global image is smaller than the resolution of the rolling image, and greater than the resolution of the depth image; acquiring the first coordinates of the first projection point where each pixel point in the depth image is reprojected onto the global image, and acquiring the second coordinates of the second projection point where each pixel point in the depth image is reprojected onto the rolling image; based on The first motion area of ​​the global image is obtained according to the first coordinates corresponding to each pixel point in the depth image and the pixel value of each pixel point in the global image, and the second motion area of ​​the rolling image is obtained according to the second coordinates corresponding to each pixel point in the depth image and the pixel value of each pixel point in the rolling image; the pixel value of each pixel point in the second motion area is corrected according to the pixel value of each pixel point in the first motion area to obtain a corrected image; in the rolling image, pixel points that match each pixel point in the motion area of ​​the corrected image are found, and the pixel values ​​of each pixel point in the motion area are replaced to obtain a replaced image.

[0007] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the program is executed by a processor, the image processing method of the embodiment of the present application is implemented. The image processing method includes: obtaining a global image, a rolling shutter image and a depth image, wherein the global image, the rolling shutter image and the depth image are frame-synchronized, the global image is taken by the global shutter camera, the rolling shutter image is taken by the rolling shutter camera, and the depth image is taken by the depth camera, the resolution of the global image is smaller than the resolution of the rolling shutter image and larger than the resolution of the depth image; obtaining the first coordinates of the first projection point where each pixel point in the depth image is reprojected onto the global image, and obtaining the second coordinates of the second projection point where each pixel point in the depth image is reprojected onto the rolling shutter image; based on The first motion area of ​​the global image is obtained according to the first coordinates corresponding to each pixel point in the depth image and the pixel value of each pixel point in the global image, and the second motion area of ​​the rolling image is obtained according to the second coordinates corresponding to each pixel point in the depth image and the pixel value of each pixel point in the rolling image; the pixel value of each pixel point in the second motion area is corrected according to the pixel value of each pixel point in the first motion area to obtain a corrected image; in the rolling image, pixel points that match each pixel point in the motion area of ​​the corrected image are found, and the pixel values ​​of each pixel point in the motion area are replaced to obtain a replaced image.

[0008] In the image processing method, image processing device, electronic device and computer-readable storage medium of the embodiments of the present application, first coordinates of a first projection point where each pixel point in the depth image is reprojected onto the global image are obtained, and second coordinates of a second projection point where each pixel point in the depth image is reprojected onto the rolling image are obtained, and a first motion area of ​​the global image is obtained based on the first coordinates and a second motion area of ​​the rolling image is obtained based on the second coordinates, and then the pixel values ​​of each pixel point in the second motion area are corrected based on the pixel values ​​of each pixel point in the first motion area to obtain a corrected image, and finally, in the rolling image, pixel points that match each pixel point in the motion area of ​​the corrected image are found, and the pixel values ​​of each pixel point in the motion area are replaced to obtain a replaced image. Thus, the entire image processing process can correct the image obtained by the electronic device when the photographed object moves relative to the electronic device, thereby ensuring the image correction effect and improving the imaging quality of the electronic device.

[0009] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0011] Figure 1 is a flowchart of an image processing method according to certain embodiments of the present application;

[0012] Figure 2 is a schematic structural diagram of an image processing device in certain embodiments of the present application;

[0013] Figure 3 is a schematic structural diagram of an electronic device according to some embodiments of the present application;

[0014] Figure 4 is a schematic structural diagram of an image processing device in certain embodiments of the present application;

[0015] Figure 5 is a flowchart of an image processing method according to certain embodiments of the present application;

[0016] Figure 6 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0017] Figure 7 is a flowchart of an image processing method according to certain embodiments of the present application;

[0018] Figure 8 is a flowchart of an image processing method according to certain embodiments of the present application;

[0019] Figure 9 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0020] Figure 10 is a flowchart of an image processing method according to certain embodiments of the present application;

[0021] Figure 11 is a flowchart of an image processing method according to certain embodiments of the present application;

[0022] Figure 12 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0023] Figure 13 is a flowchart of an image processing method according to certain embodiments of the present application;

[0024] Figure 14 is a flowchart of an image processing method according to certain embodiments of the present application;

[0025] Figure 15 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0026] Figure 16 is a flowchart of an image processing method according to certain embodiments of the present application;

[0027] Figure 17 is a flowchart of an image processing method according to certain embodiments of the present application;

[0028] Figure 18 is a flowchart of an image processing method according to certain embodiments of the present application;

[0029] Figure 19 This is a schematic diagram of the connection status between a computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0030] The following further describes the embodiments of the present application in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions. Furthermore, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are intended only to explain the embodiments of the present application and are not to be construed as limiting the present application.

[0031] In recent years, with the continuous development of electronic devices such as mobile phones and cameras, people's demand for images has continued to grow, and the pursuit of high-quality visual experience has become increasingly higher. For example, users hope that electronic devices can obtain high-quality images in different scenarios. Current electronic devices can use inertial measurement units to correct the image content captured by the camera in the electronic device, thereby improving the imaging quality of the electronic device. However, when the photographed object moves relative to the electronic device, the inertial measurement unit cannot effectively adjust the image content captured by the camera, resulting in poor image correction effect, affecting the imaging quality. In order to solve this problem, the embodiment of the present application provides an image processing method ( Figure 1 As shown), the image processing device 10 ( Figure 2 As shown), electronic device 100 ( Figure 3 ) and computer readable storage medium 200 ( Figure 19 shown).

[0032] See also Figure 1 and Figure 3 The image processing method of the embodiment of the present application is used for an electronic device 100, which includes a common-baseline global shutter camera 30, a rolling shutter camera 40, and a depth camera 50. The image processing method includes:

[0033] 01: Acquire a global image, a rolling image, and a depth image. The global image, rolling image, and depth image frames are synchronized. The global image is captured by a global shutter camera 30, the rolling image is captured by a rolling shutter camera 40, and the depth image is captured by a depth camera 50. The resolution of the global image is smaller than that of the rolling image, but larger than that of the depth image.

[0034] 03: Obtain the first coordinates of the first projection point of each pixel in the depth image reprojected onto the global image, and obtain the second coordinates of the second projection point of each pixel in the depth image reprojected onto the rolling image;

[0035] 05: Obtain a first motion region of the global image based on the first coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the global image, and obtain a second motion region of the rolling image based on the second coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the rolling image;

[0036] 07: Correcting the pixel value of each pixel in the second motion area according to the pixel value of each pixel in the first motion area to obtain a corrected image; and

[0037] 08: In the rolling shutter image, find the pixels that match the pixels in the motion area of ​​the correction image, and replace the pixel values ​​of the pixels in the motion area to obtain a replacement image.

[0038] Please combine Figure 2The above-mentioned image processing method can be applied to an image processing device 10. The image processing device 10 according to the embodiment of the present application includes a first acquisition module 11, a second acquisition module 13, a third acquisition module 15, a correction module 17, and a replacement module 18. The first acquisition module 11 is used to acquire a global image, a rolling shutter image, and a depth image. The global image, the rolling shutter image, and the depth image are frame-synchronized. The global image is captured by a global shutter camera 30, the rolling shutter image is captured by a rolling shutter camera 40, and the depth image is captured by a depth camera 50. The resolution of the global image is smaller than that of the rolling shutter image, but larger than that of the depth image. The second acquisition module 13 is used to acquire the first coordinates of the first projection point of each pixel in the depth image reprojected onto the global image, and to acquire the second coordinates of the second projection point of each pixel in the depth image reprojected onto the rolling shutter image. The third acquisition module 15 is used to acquire a first motion region of the global image based on the first coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the global image, and to acquire a second motion region of the rolling shutter image based on the second coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the rolling shutter image. The correction module 17 is configured to correct the pixel values ​​of each pixel in the second motion region based on the pixel values ​​of each pixel in the first motion region to obtain a corrected image. The replacement module 18 is configured to find pixels in the rolling image that match the pixels in the motion region of the corrected image and replace the pixel values ​​of each pixel in the motion region to obtain a replacement image.

[0039] The above-mentioned image processing method can be applied to the electronic device 100. The electronic device 100 of one embodiment of the present application includes a body 20, a global shutter camera 30, a rolling shutter camera 40, a depth camera 50, one or more processors 60, a memory 70, and one or more programs. Among them, the global shutter camera 30, the rolling shutter camera 40, the depth camera 50, the one or more processors 60 and the memory 70 are all installed in the body 20, and the global shutter camera 30, the rolling shutter camera 40 and the depth camera 50 share a common baseline. One or more programs are stored in the memory 70 and executed by the one or more processors 60. The programs include programs for executing the image processing methods in 01, 03, 05, 07 and 08. That is, the one or more processors 60 are used to: obtain a global image, a rolling image, and a depth image, wherein the global image, the rolling image, and the depth image are frame-synchronized, and the resolution of the global image is smaller than the resolution of the rolling image and larger than the resolution of the depth image; obtain first coordinates of a first projection point where each pixel in the depth image is reprojected onto the global image, and obtain second coordinates of a second projection point where each pixel in the depth image is reprojected onto the rolling image; obtain a first motion region of the global image based on the first coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the global image, and obtain a second motion region of the rolling image based on the second coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the rolling image; correct the pixel value of each pixel in the second motion region based on the pixel value of each pixel in the first motion region to obtain a corrected image; and find, in the rolling image, pixel points that match each pixel in the motion region of the corrected image, and replace the pixel values ​​of each pixel in the motion region to obtain a replaced image.

[0040] An electronic device 100 according to another embodiment of the present application may include a body 20 and an image processing device 10 according to an embodiment of the present application, wherein the image processing device 10 is installed in the body 20 .

[0041] The electronic device 100 of the embodiment of the present application includes, but is not limited to, a mobile phone, a tablet computer, a camera, a camcorder, a personal digital assistant, a wearable device, an intelligent robot, or an intelligent vehicle. The wearable device includes a smart bracelet, a smart watch, or smart glasses. The camcorder may include a charge-coupled device (CCD) camera or a complementary metal-oxide-semiconductor (CMOS) camera.

[0042] The global shutter camera 30 is used to capture a global image, the rolling shutter camera 40 is used to capture a rolling shutter image, and the depth camera 50 is used to capture a depth image. In some embodiments, the depth camera 50 may be a time-of-flight (TOF) depth camera, a structured light depth camera, a binocular depth camera, or the like. If the depth camera 50 is a time-of-flight (TOF) depth camera, it transmits pulse-modulated infrared light toward the object being photographed and receives infrared light reflected from the object. The TOF depth camera is capable of performing photoelectric conversion on the received infrared light to generate a depth image.

[0043] In certain embodiments, the global shutter camera 30, the rolling shutter camera 40, and the depth camera 50 are configured such that the image sensors of the global shutter camera 30, the rolling shutter camera 40, and the depth camera 50 are arranged along the same baseline in the electronic device 100 (i.e., the image sensors of the global shutter camera 30, the rolling shutter camera 40, and the depth camera 50 are aligned with each other), and the optical axes of the lenses in the global shutter camera 30, the rolling shutter camera 40, and the depth camera 50 are parallel to each other and perpendicular to the same baseline. Thus, the global shutter camera 30, the rolling shutter camera 40, and the depth camera 50 share a common baseline, thereby ensuring consistency in images captured by different cameras of the same scene, thereby improving the accuracy of subsequent image processing.

[0044] Since the global shutter camera 30, rolling shutter camera 40, and depth camera 50 are three independent devices, the local times set in the three devices are not necessarily synchronized. For example, at the same time, the local time in the global shutter camera 30 is 21:58, the local time in the rolling shutter camera 40 is 21:59, and the local time in the depth camera 50 is 22:00. As a result, the data collected by the global shutter camera 30, rolling shutter camera 40, and depth camera 50 cannot be correlated. Therefore, please refer to Figure 4In the embodiment of the present application, the global image, rolling image and depth image frames are synchronized. Specifically, the global shutter camera 30, the rolling shutter camera 40 and the depth camera 50 can be connected by a hardware synchronization line to ensure the synchronization of the global image, the rolling image and the depth image frames, that is, the global image, the rolling image and the depth image taken at the same time are for the same scene. This can avoid the problem that the data collected by the global shutter camera 30, the rolling shutter camera 40 and the depth camera 50 cannot be associated due to the local time being out of synchronization within the global shutter camera 30, the rolling shutter camera 40 and the depth camera 50, thereby ensuring the accuracy of subsequent data processing. It can be understood that the global shutter camera 30, the rolling shutter camera 40, the depth camera 50 and the IMU can also be connected by a hardware synchronization line to ensure the accuracy of data association and improve the effect of subsequent image correction.

[0045] In some embodiments, the electronic device 100 can directly obtain the global image, rolling image, and depth image in real time from the global shutter camera 30, the rolling shutter camera 40, and the depth camera 50. The global image is captured by the global shutter camera 30, the rolling image is captured by the rolling shutter camera 40, and the depth image is captured by the depth camera 50. In other embodiments, the electronic device 100 can obtain the global image, rolling image, and depth image from other devices, such as a server, a cloud, or other electronic devices.

[0046] Resolution refers to the amount of information stored in an image, which is the number of pixels per inch of the image. The unit of resolution is PPI (Pixels Per Inch), commonly called pixels per inch. In some embodiments, the resolution of the global image is lower than the resolution of the rolling image and higher than the resolution of the depth image. This allows the high-resolution image (rolling image) to be adjusted using the low-resolution image (global image), thereby improving the image quality captured by the electronic device 100. Compared to a case where the resolution of the global shutter camera 30 is the same as that of the rolling shutter camera 40, the global shutter camera 30 in this embodiment is less expensive, thereby reducing the production cost of the electronic device 100.

[0047] In some embodiments, before the electronic device 100 takes a picture, it is necessary to calibrate the electronic device 100. Calibration methods include, but are not limited to, traditional camera calibration methods, active vision camera calibration methods, or camera self-calibration methods. In the embodiments of the present application, the Zhang calibration method, a traditional camera calibration method, may be used for calibration.

[0048] The calibration content includes the internal parameters of the global shutter camera 30, the rolling shutter camera 40, and the depth camera 50, the external parameters from the depth camera 50 to the global shutter camera 30, and the external parameters from the depth camera 50 to the rolling shutter camera 40. In the embodiment of the present application, matrices are used to represent the internal parameters of the global shutter camera 30 and the rolling shutter camera 40, and rotation matrices and translation vectors are used to represent the external parameters from the depth camera 50 to the global shutter camera 30 and the external parameters from the depth camera 50 to the rolling shutter camera 40.

[0049] Specifically, the intrinsic parameter matrices representing the global shutter camera 30 and the rolling shutter camera 40 can be expressed as follows:

[0050]

[0051] For the global shutter camera 30 and the rolling shutter camera 40 , fx and fy are the intrinsic focal lengths, x0 and y0 are the principal point coordinates (relative to the imaging plane), and s is the coordinate axis tilt parameter, which can generally be set to 0.

[0052] The rotation matrix looks like this:

[0053]

[0054] The translation vector is as follows:

[0055]

[0056] In the image processing method, image processing device 10, electronic device 100 and computer-readable storage medium of the embodiments of the present application, first coordinates of a first projection point where each pixel point in the depth image is reprojected onto the global image are obtained, and second coordinates of a second projection point where each pixel point in the depth image is reprojected onto the rolling image are obtained, and a first motion area of ​​the global image is obtained based on the first coordinates and a second motion area of ​​the rolling image is obtained based on the second coordinates. Then, according to the pixel values ​​of each pixel point in the first motion area, the pixel values ​​of each pixel point in the second motion area are corrected to obtain a corrected image. Finally, in the rolling image, pixel points that match each pixel point in the motion area of ​​the corrected image are found, and the pixel values ​​of each pixel point in the motion area are replaced to obtain a replaced image. Thus, the entire image processing process can correct the image acquired by the electronic device 100 when the photographed object moves relative to the electronic device 100, thereby ensuring the image correction effect and improving the imaging quality of the electronic device 100.

[0057] See also Figure 5 and Figure 6In some embodiments, 02: obtaining first coordinates of a first projection point where each pixel point in the depth image is reprojected onto the global image, and obtaining second coordinates of a second projection point where each pixel point in the depth image is reprojected onto the rolling image, includes:

[0058] 021: Convert the coordinates of each pixel in the depth image into point cloud coordinates according to the principal point coordinates of the depth camera 50, the coordinates of the pixel in the depth image, the depth value of the pixel in the depth image, and the intrinsic reference focal length of the depth camera 50;

[0059] 023: Convert the point cloud coordinates to the first coordinate system of the global shutter camera 30 and the second coordinate system of the rolling shutter camera 40, respectively, to obtain first image coordinates corresponding to the point cloud coordinates in the first coordinate system and second image coordinates corresponding to the point cloud coordinates in the second coordinate system; and

[0060] 025: Obtain the first coordinates of the first projection point on the global image according to the first image coordinates and the intrinsic parameter matrix of the global shutter camera 30 , and obtain the second coordinates of the second projection point on the rolling shutter image according to the second image coordinates and the intrinsic parameter matrix of the rolling shutter camera 40 .

[0061] Please combine Figure 2 The second acquisition module 13 is further used to: convert the coordinates of each pixel in the depth image into point cloud coordinates according to the principal point coordinates of the depth camera 50, the coordinates of the pixel in the depth image, the depth value of the pixel in the depth image, and the intrinsic reference focal length of the depth camera 50; convert the point cloud coordinates into the first coordinate system of the global shutter camera 30 and the second coordinate system of the rolling shutter camera 40, respectively, to obtain first image coordinates corresponding to the point cloud coordinates in the first coordinate system, and second image coordinates corresponding to the point cloud coordinates in the second coordinate system; and obtain first coordinates of the first projection point on the global image according to the first image coordinates and the intrinsic reference matrix of the global shutter camera 30, and obtain second coordinates of the second projection point on the rolling image according to the second image coordinates and the intrinsic reference matrix of the rolling shutter camera 40.

[0062] Please combine Figure 3, the program in the electronic device 100 of the embodiment of the present application includes a program for executing the image processing method in 021, 023 and 025. That is, the one or more processors 60 are used to convert the coordinates of each pixel in the depth image into point cloud coordinates according to the principal point coordinates of the depth camera 50, the coordinates of the pixel in the depth image, the depth value of the pixel in the depth image and the intrinsic focal length of the depth camera 50; convert the point cloud coordinates into the first coordinate system of the global shutter camera 30 and the second coordinate system of the rolling shutter camera 40 respectively, so as to obtain the first image coordinates corresponding to the point cloud coordinates in the first coordinate system and the second image coordinates corresponding to the point cloud coordinates in the second coordinate system; and obtain the first coordinates of the first projection point on the global image according to the first image coordinates and the intrinsic parameter matrix of the global shutter camera 30, and obtain the second coordinates of the second projection point on the rolling image according to the second image coordinates and the intrinsic parameter matrix of the rolling shutter camera 40.

[0063] The depth image includes multiple pixels, which can be converted into point cloud coordinates after coordinate conversion. Specifically, the formula for converting the coordinates of the pixel points in the depth image into point cloud coordinates is as follows:

[0064]

[0065] Among them, x w 、y w 、z w is the point cloud coordinate, u and v are the coordinates of the pixel points in the depth image, and Z c is the depth value of the (u, v) coordinate corresponding to the depth image and the pixel point, f x 、f y is the intrinsic focal length of the depth camera 50 , and u0 and v0 are the principal point coordinates of the depth camera 50 .

[0066] The conversion formulas for converting the point cloud coordinates into the coordinates in the first coordinate system of the global shutter camera 30 and the point cloud coordinates into the coordinates in the second coordinate system of the rolling shutter camera 40 are as follows:

[0067]

[0068] In the case where the point cloud coordinates are converted into the coordinates of the first coordinate system of the global shutter camera 30, x rgb 、y rgb 、z rgb is the first image coordinate of the point cloud coordinate in the first coordinate system of the global shutter camera 30, T is the translation vector in the external parameters from the depth camera 50 to the global shutter camera 30, R is the rotation matrix in the external parameters from the depth camera 50 to the global shutter camera 30, x w 、y w 、z wis the point cloud coordinate. When the point cloud coordinate is converted into the coordinate of the second coordinate system of the rolling shutter camera 40, xr gb 、y rgb 、z rgb is the second image coordinate of the point cloud coordinate in the second coordinate system of the rolling shutter camera 40, T is the translation vector in the external parameter from the depth camera 50 to the rolling shutter camera 40, R is the rotation matrix in the external parameter from the depth camera 50 to the rolling shutter camera 40, x w 、y w 、z w is the point cloud coordinate.

[0069] The formulas for obtaining the first coordinates of the first projection point on the global image based on the first image coordinates and the intrinsic parameter matrix of the global shutter camera 30, and obtaining the second coordinates of the second projection point on the rolling shutter image based on the second image coordinates and the intrinsic parameter matrix of the rolling shutter camera 40 are as follows:

[0070]

[0071] Among them, a is the proportional factor, which is a known constant. rgb 、y rgb 、z rgb is the first image coordinate of the point cloud coordinate in the first coordinate system of the global shutter camera 30, a is the first scale factor, K is the internal parameter matrix of the global shutter camera 30, u rgb 、v rgb is the first coordinate of the first projection point on the global image; rgb 、y rgb 、z rgb Wherein, is the second image coordinate of the point cloud coordinate in the second coordinate system of the rolling shutter camera 40; a is the second scale factor, which may be the same as or different from the first scale factor; K is the intrinsic parameter matrix of the rolling shutter camera 40, which may be the same as or different from the intrinsic parameter matrix of the global shutter camera 30; u rgb 、v rgb is the second coordinate of the second projection point on the rolling image.

[0072] Please combine Figure 6In this embodiment, the above formula is used to reproject each pixel in the depth image onto the global image to obtain multiple first projection points on the global image, and each pixel in the depth image is reprojected onto the rolling curtain image to obtain multiple second projection points on the rolling curtain image, and the first coordinates corresponding to the first projection points and the second coordinates corresponding to the second projection points are obtained. As a result, each pixel in the depth image corresponds to a first projection point and a second projection point. For example, pixel D1 in the depth image may correspond to the first projection point P3 in the global image, and pixel D1 in the depth image may also correspond to the second projection point P3' in the rolling curtain image; for another example, pixel D2 in the depth image may correspond to the first projection point P8 in the global image, and pixel D2 in the depth image may also correspond to the second projection point P11' in the rolling curtain image. As a result, the pixels in the depth image can provide corresponding references for the pixels in the global image and the rolling curtain image, thereby facilitating subsequent processing of the global image and the rolling curtain image.

[0073] See also Figure 7 In some embodiments, 03: obtaining a first motion region of the global image based on first coordinates corresponding to each pixel in the depth image and a pixel value of each pixel in the global image, and obtaining a second motion region of the rolling image based on second coordinates corresponding to each pixel in the depth image and a pixel value of each pixel in the rolling image, includes:

[0074] 031: Traverse each pixel point in the depth image, obtain the first pixel value of the first projection point in the global image according to the first coordinate of the corresponding first projection point, and obtain the second pixel value of the second projection point in the rolling image according to the second coordinate of the corresponding second projection point;

[0075] 033: if the difference between the first pixel value and the second pixel value is greater than a preset first threshold, confirming the pixel in the depth image as a moving pixel; and

[0076] 035: Expand the first projection point corresponding to the moving pixel point in the global image to obtain a first moving area, and expand the second projection point corresponding to the moving pixel point in the rolling image to obtain a second moving area.

[0077] Please combine Figure 2The third acquisition module 15 is further used to: traverse each pixel point in the depth image, obtain a first pixel value of the first projection point in the global image according to the first coordinate of the corresponding first projection point, and obtain a second pixel value of the second projection point in the rolling image according to the second coordinate of the corresponding second projection point; when the difference between the first pixel value and the second pixel value is greater than a preset first threshold, confirm the pixel point in the depth image as a moving pixel point; and expand the first projection point corresponding to the moving pixel point in the global image to obtain a first moving area, and expand the second projection point corresponding to the moving pixel point in the rolling image to obtain a second moving area.

[0078] Please combine Figure 3 , the program in the electronic device 100 of the embodiment of the present application includes a program for executing the image processing method in 031, 033 and 035. That is, the one or more processors 60 are used to traverse each pixel point in the depth image, obtain the first pixel value of the first projection point in the global image according to the first coordinate of the corresponding first projection point, and obtain the second pixel value of the second projection point in the rolling image according to the second coordinate of the corresponding second projection point; when the difference between the first pixel value and the second pixel value is greater than a preset first threshold, the pixel point in the depth image is confirmed as a moving pixel point; and the first projection point corresponding to the moving pixel point in the global image is expanded to obtain a first moving area, and the second projection point corresponding to the moving pixel point in the rolling image is expanded to obtain a second moving area.

[0079] Among them, see Figure 6 , each pixel in the depth image corresponds to a first projection point and a second projection point, obtain the first pixel value of the first projection point corresponding to a pixel in the depth image, and the second pixel value of the second projection point corresponding to the same pixel, and when the difference between the first pixel value and the second pixel value is greater than the preset first threshold, the pixel in the depth image is confirmed as a moving pixel. The above processing is performed on each pixel in the depth image to determine all moving pixels in the depth image. For example, if the first projection point P12 in the global image and the second projection point P25' in the rolling shutter image both correspond to the pixel D4 in the depth image, and the difference between the first projection point P12 and the second projection point P25' is greater than the preset first threshold, the pixel D4 in the depth image can be regarded as a moving pixel.

[0080] In some embodiments, the first pixel value may be any one of the RGB three-channel color values, and the second pixel value may be any one of the RGB three-channel color values. For example, when the first pixel value is R1 in (R1, G1, B1), and the second pixel value is R2 in (R2, G2, B2), if the absolute value of the difference between the first pixel value and the second pixel value (|R1-R2|) is greater than a preset first threshold, then the pixel corresponding to the first projection point corresponding to the first pixel value in the depth image is a moving pixel; in other words, the pixel corresponding to the second projection point corresponding to the second pixel value in the depth image is a moving pixel. For another example, when the first pixel value is G1 in (R1, G1, B1), and the second pixel value is G2 in (R2, G2, B2), if the absolute value of the difference between the first pixel value and the second pixel value (|G1-G2|) is greater than a preset first threshold, then the pixel corresponding to the first projection point corresponding to the first pixel value in the depth image is a moving pixel. For example, when the first pixel value is B1 in (R1, G1, B1), and the second pixel value is B2 in (R2, G2, B2), if the absolute value of the difference between the first pixel value and the second pixel value (|B1-B2|) is greater than a preset first threshold, then the pixel corresponding to the first projection point corresponding to the first pixel value in the depth image is a moving pixel. For the three channels of R, G, and B, each color channel has a corresponding first threshold, and the three first thresholds corresponding to the three channels can be the same or different. In addition, as long as one channel meets the conditions given in the example above, it can be determined that the pixel corresponding to the second projection point corresponding to the second pixel value in the depth image is a moving pixel.

[0081] In other embodiments, the first pixel value may be a composite value Y1 of the three-channel color values ​​of R, G, and B, for example, Y1 = a1R1 + b1G1 + c1B1. Correspondingly, the second pixel value may be a composite value Y2 = a2R2 + b2G2 + c2B2 of the three-channel color values ​​of R, G, and B. When the absolute value of the difference between the first pixel value and the second pixel value is greater than a preset first threshold, the pixel corresponding to the first projection point corresponding to the first pixel value in the depth image is a moving pixel; in other words, the pixel corresponding to the second projection point corresponding to the second pixel value in the depth image is a moving pixel.

[0082] In some embodiments, the first threshold value may be a threshold value set by the image processing device 10 or the electronic device 100 before leaving the factory. The first threshold value is stored in the memory 70 and can be subsequently called by the processor 60. The first threshold value may be obtained in such a manner that, under a certain threshold value, all the pixel points corresponding to the distorted area in the rolling image seen by the user can be determined as the second projection points corresponding to the moving pixel points, and the threshold value is the first threshold value. For example, if the user observes that the pixel points corresponding to the distorted area in the rolling image are P25', P27', and P33', at this time, if the difference between the first pixel value (the pixel value of the pixel point corresponding to the pixel point P25', P27', and P33' in the global image) and the second pixel value (the pixel value of the pixel point P25', P27', and P33' in the rolling image) is greater than a certain threshold value, the threshold value can be used as the first threshold value.

[0083] Please combine Figure 9 In some embodiments, the expansion process may be an expansion process, that is, the first projection point ( Figure 9 The first projection points P12, P14 and P18 corresponding to the moving pixel points in the rolling image are expanded to obtain the first motion area, and the second projection points ( Figure 9 The second projection points P25', P27' and P33') corresponding to the moving pixel points in the rolling shutter image are expanded to obtain the second motion area. The expansion process refers to the morphological operation performed on the original image using the structural element, that is, the first projection points corresponding to the moving pixel points in the global image are expanded using the structural element, so that the expanded first projection points can form the first motion area, and the expanded first projection points can fill the first motion area, and the second projection points corresponding to the moving pixel points in the rolling shutter image are expanded using the structural element, so that the expanded second projection points can form the second motion area, and the expanded second projection points can fill the second motion area. It should be noted that, in some embodiments, the expansion process may include horizontal expansion, vertical expansion or omnidirectional expansion, etc., which are not limited here. The number of first motion areas formed by the expansion process may be one or more, and the number of second motion areas may also be one or more.

[0084] In other embodiments, the dilation process may be a convex hull process, that is, a convex hull process is performed on the first projection points corresponding to the moving pixel points in the global image to obtain a first motion region, and a convex hull process is performed on the second projection points corresponding to the moving pixel points in the rolling image to obtain a second motion region. The convex hull is the boundary of a convex set consisting of all given points, and the given points may be the first projection points corresponding to the moving pixel points or the second projection points corresponding to the moving pixel points. In some embodiments, the convex hull algorithm may be a Graham scanning method, a Jarvis stepping method, or an Andrew algorithm, etc., without limitation herein.

[0085] Specifically, see Figure 8 In some embodiments, 035: performing expansion processing on first projection points corresponding to the moving pixel points in the global image to obtain a first motion region, and performing expansion processing on second projection points corresponding to the moving pixel points in the rolling image to obtain a second motion region, includes:

[0086] 0351: Determine a first dilation amount of a first projection point corresponding to a moving pixel point in the global image and a second dilation amount of a second projection point corresponding to the moving pixel point in the rolling image; and

[0087] 0353: The area formed by dilating the first projection point corresponding to the moving pixel point in the global image according to the first dilation amount is the first moving area, and the area formed by dilating the second projection point corresponding to the moving pixel point in the rolling image according to the second dilation amount is the second moving area.

[0088] Please combine Figure 2 The third acquisition module 15 is further configured to determine a first dilation amount for a first projection point corresponding to a moving pixel point in the global image and a second dilation amount for a second projection point corresponding to the moving pixel point in the rolling image; an area formed by dilating the first projection point corresponding to the moving pixel point in the global image according to the first dilation amount is defined as a first motion area, and an area formed by dilating the second projection point corresponding to the moving pixel point in the rolling image according to the second dilation amount is defined as a second motion area.

[0089] Please combine Figure 3The program in the electronic device 100 according to the embodiment of the present application includes a program for executing the image processing methods in 0351 and 0353. That is, the one or more processors 60 are configured to determine a first dilation amount for a first projection point corresponding to a moving pixel in the global image and a second dilation amount for a second projection point corresponding to the moving pixel in the rolling image; and a region formed by dilating the first projection point corresponding to the moving pixel in the global image according to the first dilation amount as a first motion region, and a region formed by dilating the second projection point corresponding to the moving pixel in the rolling image according to the second dilation amount as a second motion region.

[0090] Please combine Figure 9 In a first motion region obtained by dilating the first projection point corresponding to the moving pixel point in the global image according to the first dilation amount, the first projection point can fill the first motion region; and in a second motion region obtained by dilating the second projection point corresponding to the moving pixel point in the rolling image according to the second dilation amount, the second projection point can fill the second motion region. This ensures that the acquired motion region is more accurate, thereby ensuring the accuracy of subsequent image correction and improving the quality of the obtained target image.

[0091] More specifically, see Figure 10 In some embodiments, 0351: determining a first dilation amount of a first projection point corresponding to a moving pixel point in a global image and a second dilation amount of a second projection point corresponding to a moving pixel point in a rolling image includes:

[0092] 03511: Obtain a first dilation amount according to the resolution difference between the global shutter camera 30 and the depth camera 50;

[0093] 03513: Obtain a second dilation amount according to the resolution difference between the rolling shutter camera 40 and the depth camera 50.

[0094] Please combine Figure 2 The third acquisition module 15 is further configured to: acquire a first dilation value according to a resolution difference between the global shutter camera 30 and the depth camera 50 ; and acquire a second dilation value according to a resolution difference between the rolling shutter camera 40 and the depth camera 50 .

[0095] Please combine Figure 3 The program in the electronic device 100 according to the embodiment of the present application includes a program for executing the image processing methods in 03511 and 03513. That is, the one or more processors 60 are configured to obtain a first dilation value based on the resolution difference between the global shutter camera 30 and the depth camera 50. And obtain a second dilation value based on the resolution difference between the rolling shutter camera 40 and the depth camera 50.

[0096] In some embodiments, if the resolution of the global shutter camera 30 is four times the resolution of the depth camera 50 (twice the length and twice the width), the first dilation amount may be 1*(2-1), i.e., the area formed by dilating the first projection points corresponding to the moving pixels in the global image according to the first dilation amount is the first moving area. It is understood that the method for obtaining the second dilation amount is essentially the same as the method for obtaining the first dilation amount and is not further described here. The first projected points after dilation can fill the first moving area; the second projected points after dilation can fill the second moving area. In other embodiments, the first dilation amount may be determined based on the number and spacing of the first projected points corresponding to the moving pixels in the global image. It is only necessary to ensure that after the area formed by dilating the first projected points corresponding to the moving pixels in the global image according to the first dilation amount is the first moving area, the first projected points after dilation can fill the first moving area. Correspondingly, the second expansion amount can be determined based on the number and intervals of the second projection points corresponding to the moving pixel points in the rolling image. It is only necessary to ensure that after the area formed by expanding the second projection points corresponding to the moving pixel points in the rolling image according to the second expansion amount is the second moving area, the expanded second projection points can fill the second moving area.

[0097] See also Figure 11 In some embodiments, 07: correcting the pixel value of each pixel point in the second motion area according to the pixel value of each pixel point in the first motion area to obtain a corrected image, including:

[0098] 071: In the first motion area, obtain mapping coordinates corresponding to pixel points in the second motion area;

[0099] 073: When the mapping coordinates are all integer values, the pixel value of the pixel point in the second motion area is replaced by the pixel value of the pixel point under the mapping coordinates;

[0100] 075: When at least one of the mapped coordinates is a decimal value, the pixel value of the pixel point under the mapped coordinate in the first motion area is obtained by interpolation, and the pixel value of the pixel point in the second motion area is updated to the interpolated pixel value.

[0101] Please combine Figure 2 The correction module 17 is further configured to obtain, in the first motion region, mapping coordinates corresponding to pixels in the second motion region. If the mapping coordinates are all integer values, the pixel value of the pixel in the second motion region is replaced by the pixel value of the pixel at the mapping coordinates. If at least one of the mapping coordinates is a decimal value, the pixel value of the pixel at the mapping coordinate in the first motion region is obtained by interpolation, and the pixel value of the pixel in the second motion region is updated to the interpolated pixel value.

[0102] Please combine Figure 3 , the program in the electronic device 100 of the embodiment of the present application includes a program for executing the image processing method in 071, 073 and 075. That is, one or more processors 60 are used to obtain the mapping coordinates corresponding to the pixel points in the second motion area in the first motion area. When the mapping coordinates are all integer values, the pixel value of the pixel point in the second motion area is replaced with the pixel value of the pixel point under the mapping coordinates. When at least one of the mapping coordinates is a decimal value, the pixel value of the pixel point under the mapping coordinates in the first motion area is obtained by interpolation, and the pixel value of the pixel point in the second motion area is updated to the pixel value obtained by interpolation.

[0103] Specifically, please combine Figure 12 In some embodiments, the pixel points in the second motion area are mapped to the first motion area, thereby obtaining the mapped pixel points in the first motion area corresponding to the pixel points in the second motion area, and obtaining the mapped coordinates of the mapped pixel points in the first motion area. The mapped coordinates may all be integer values; or the mapped coordinates may all be decimal values; or one of the mapped coordinates may be an integer and the other may be a decimal. In the case where the mapped coordinates are all integer values, the pixel points in the second motion area are replaced with the mapped pixel points in the first motion area; in the case where at least one of the mapped coordinates is a decimal value, the pixel value of the mapped pixel point in the first motion area is obtained by interpolation, and the pixel value of the pixel point in the second motion area is updated to the pixel value obtained by interpolation. Traverse each pixel point in the second motion area to replace the pixel value of each pixel point in the second motion area with the pixel value of the mapped pixel point under the mapped coordinates.

[0104] In some embodiments, the interpolation algorithm may be nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation, etc., without limitation herein. For example, when the interpolation algorithm is bilinear interpolation, when at least one of the mapped coordinates is a decimal value, the pixel value of the pixel point at the mapped coordinate in the first motion region is obtained by bilinear interpolation, and the pixel value of the pixel point in the second motion region is updated to the interpolated pixel value.

[0105] Specifically, see Figure 13 In some embodiments, 075: obtaining pixel values ​​of pixel points at the mapped coordinates in the first motion region by interpolation includes:

[0106] 0751: Obtain the four test pixels in the first motion area that are closest to the pixel at the mapped coordinates;

[0107] 0753: Perform weighted processing on the pixel values ​​of the four test pixels according to preset weights to obtain the pixel value of the pixel at the mapping coordinate in the first motion area.

[0108] Please combine Figure 2 The correction module 17 is further configured to obtain four test pixels in the first motion region that are closest to the pixel at the mapped coordinates, and perform weighted processing on the pixel values ​​of the four test pixels according to preset weights to obtain the pixel value of the pixel at the mapped coordinates in the first motion region.

[0109] Please combine Figure 3 The program in the electronic device 100 according to the embodiment of the present application includes a program for executing the image processing methods in 0751 and 0753. That is, the one or more processors 60 are configured to obtain four test pixels in the first motion region that are closest to the pixel at the mapped coordinates. The pixel values ​​of the four test pixels are weighted according to a preset weight to obtain the pixel value of the pixel at the mapped coordinates in the first motion region.

[0110] When at least one of the mapping coordinates is a decimal value, obtain the four test pixels closest to the pixel point (mapping pixel point) under the mapping coordinate in the first motion area, and perform weighted processing on the pixel values ​​of the four test pixels according to the preset weights to obtain the pixel value of the mapping pixel point under the mapping coordinate in the first motion area. Figure 12 In some embodiments, if in the first motion region, the mapped pixel point of the mapped coordinate corresponding to the pixel point P25' in the second motion region is P112, and the mapped pixel point P112 is located between the pixel point P12 and the pixel P18 in the global image, then the four test pixels closest to the mapped pixel point P112 in the first motion region may be P12, P( Figure 12 The pixel points P between the pixel points P12 and P14 of the global image), P( Figure 12The pixel points P) and P18 to the left of the pixel point P18 in the global image of the image are weighted according to preset weights to obtain the pixel value of the mapped pixel point P112, and the pixel value of the pixel point P25' in the second motion area is updated to the pixel value of the mapped pixel point P112 obtained by interpolation. Specifically, in some embodiments, if the coordinates of the pixel point in the second motion area are (4, 4), the mapping coordinates corresponding to the pixel point with coordinates (4, 4) in the first motion area are (2.04, 2.04). Therefore, it is necessary to find the four test pixels ((1, 1), (2, 1), (1, 2), and (2, 2)) closest to the pixel point with the mapping coordinates (2.04, 2.04) in the first motion area, and weight the pixel values ​​of the four test pixels according to preset weights to obtain the pixel value of the pixel point with the mapping coordinates in the first motion area.

[0111] The pixel values ​​of the four test pixels are weighted according to the preset weights to obtain the pixel value of the pixel at the mapping coordinate in the first motion area. The formula is as follows:

[0112] S=d1+d2+d3+d4

[0113]

[0114] Among them, d1, d2, d3 and d4 are the Euler distances from the four test pixels to the mapping coordinates, p1, p2, p3 and p4 are the pixel values ​​of the four test pixels, and T is the pixel value of the pixel under the mapping coordinates in the first motion area.

[0115] See also Figure 14 and Figure 15 In some embodiments, 08: finding, in the rolling image, pixel points that match each pixel point in the motion region of the correction image, and replacing the pixel values ​​of each pixel point in the motion region to output a replacement image, including:

[0116] 081: In the current rolling shutter image of the nth frame, search for pixel points that match the pixel points in the motion area of ​​the correction image, and replace the pixel values ​​of the pixel points in the motion area to output a replacement image.

[0117] Please combine Figure 2 The replacement module 18 is further configured to search for pixel points that match the pixel points in the motion region of the correction image in the current rolling shutter image of the nth frame, and replace the pixel values ​​of the pixel points in the motion region to output a replacement image.

[0118] Please combine Figure 3The program in the electronic device 100 according to the embodiment of the present application includes a program for executing the image processing method in 081. That is, the one or more processors 60 are configured to search for pixels in the current rolling shutter image of the nth frame that match the pixels in the motion region of the correction image, and replace the pixel values ​​of the pixels in the motion region to output a replacement image.

[0119] Because the resolution of the global image is lower than that of the rolling image, the resolution of the motion region of the corrected image obtained in step 07 is lower, resulting in poor image quality. Therefore, it is necessary to find pixels in the rolling image that match the pixels in the motion region of the corrected image and replace the pixel values ​​of each pixel in the motion region to obtain a replacement image with better image quality.

[0120] In some embodiments, a convolutional neural network model can be used to search for pixel points that match the pixel points in the motion area of ​​the corrected image in the current n-th frame of the rolling shutter image. After finding the matching pixel points, the pixel values ​​of the pixel points in the motion area of ​​the corrected image are replaced to obtain a replacement image.

[0121] Specifically, see Figure 16 In some embodiments, 081: searching for pixel points that match each pixel point in the motion region of the corrected image, and replacing pixel values ​​of each pixel point in the motion region to output a replacement image, includes:

[0122] 0811: Acquire a first image block centered on each pixel point in the motion area of ​​the corrected image;

[0123] 0813: Acquire multiple second image blocks centered on all pixels in the rolling image;

[0124] 0815: Matching the first image block and the plurality of second image blocks; and

[0125] 0817: Update the pixel value of the pixel point corresponding to the second image block whose matching result is greater than the preset second threshold to the pixel value of the pixel point corresponding to the first image block.

[0126] Please combine Figure 2 The replacement module 18 is further configured to: obtain a first image block centered on each pixel in the motion region of the corrected image; obtain multiple second image blocks centered on all pixels in the rolling image; match the first image block with the multiple second image blocks; and update the pixel values ​​corresponding to the pixels in the second image blocks whose matching results are greater than a preset second threshold to the pixel values ​​corresponding to the pixels in the first image blocks.

[0127] Please combine Figure 3The program in the electronic device 100 according to the embodiment of the present application includes a program for executing the image processing methods in 0811, 0813, 0815, and 0817. That is, the one or more processors 60 are configured to obtain a first image block centered on each pixel in the motion region of the corrected image; obtain multiple second image blocks centered on all pixels in the rolling image; match the first image block with the multiple second image blocks; and update the pixel values ​​corresponding to the pixels in the second image blocks whose matching results are greater than a preset second threshold to the pixel values ​​corresponding to the pixels in the first image block.

[0128] Specifically, multiple m*m image blocks are constructed as the second image blocks with all the pixels in the current n-th frame as the center, and then an m*m image block is constructed as the first image block with each pixel in the motion area of ​​the corrected image as the center. The first image block is matched with the multiple second image blocks. If the match is successful, the pixel values ​​of the pixels corresponding to the second image blocks in the motion area of ​​the corrected image are updated to the pixel values ​​of the pixels corresponding to the first image blocks in the rolling image. For example, please combine Figure 15 , construct multiple 3*3 (m=3) image blocks as second image blocks with all pixels in the current n-th frame as the center, construct a 3*3 image block as the first image block with pixel point P112 in the motion area of ​​the corrected image as the center, match the first image block with multiple second image blocks, and if the matching result of the first image block centered on pixel point P112 and the second image block centered on pixel point P25' is greater than a preset second threshold, update the pixel value of pixel point P112 in the motion area of ​​the corrected image to the pixel value of pixel point P25' in the current n-th frame rolling image.

[0129] It should be noted that, in some embodiments, if all pixels in the motion region of the corrected image can be found in the current rolling image of the nth frame, the replacement image output in step 081 is the target image.

[0130] The calculation formula for matching the first image block and the plurality of second image blocks is as follows:

[0131]

[0132] Wherein, μ1 is the mean value of the pixels in the first image block, μ2 is the mean value of the pixels in the second image block, x in I1(x) is the coordinate of the pixel in the first image block, x in I2(x) is the coordinate of the pixel in the second image block, I1(x) is the pixel value of the pixel in the first image block, and I2(x) is the pixel value of the pixel in the second image block.

[0133] In other embodiments, feature similarity can be used to search for pixels in the current n-th frame of the rolling shutter image that match each pixel in the motion region of the corrected image. After finding the matching pixels, the pixel values ​​of each pixel in the motion region of the corrected image are replaced to obtain a replacement image. Specifically, multiple second image blocks centered around all pixels in the current n-th frame of the rolling shutter image are obtained, and a first image block centered around each pixel in the motion region of the corrected image is obtained. Feature points in the first image block are extracted and matched with feature points in the second image block. When the similarity between the feature points in the first image block and the feature points in the second image block is greater than a preset second threshold, the pixel values ​​corresponding to the pixel blocks in the motion region of the corrected image that have successfully matched are updated to the pixel values ​​corresponding to the pixel blocks in the rolling shutter image.

[0134] In some embodiments, the second threshold may be a threshold set by the image processing apparatus 10 or the electronic device 100 before shipment. The second threshold may be obtained by taking as the second threshold a threshold at which pixels in the motion region of the corrected image are more likely to be updated to pixels in the current n-th frame of the rolling shutter image.

[0135] See also Figure 17 In some embodiments, if no pixel points matching a preset number of pixel points in the motion region of the corrected image are found in the rolling image of the current n-th frame, 08: finding pixel points in the rolling image matching each pixel point in the motion region of the corrected image, and replacing pixel values ​​of each pixel point in the motion region to output a replacement image, further comprising:

[0136] 083: In the rolling shutter image of the i-th frame, find pixel points that match the pixel points in the motion area of ​​the corrected image, and replace the pixel values ​​of the pixel points in the motion area to obtain a replacement image, where i<n.

[0137] Please combine Figure 2 The replacement module 18 is further used to: in the rolling shutter image of the i-th frame, find pixel points that match the pixel points in the motion area of ​​the corrected image, and replace the pixel values ​​of the pixel points in the motion area to output a replacement image, where i<n.

[0138] Please combine Figure 3 The program in the electronic device 100 according to the embodiment of the present application includes a program for executing the image processing method in 083. That is, the one or more processors 60 are configured to search for pixels in the rolling shutter image of the i-th frame that match the pixels in the motion region of the correction image, and replace the pixel values ​​of the pixels in the motion region to output a replacement image, where i<n.

[0139] Specifically, in some embodiments, i may be a value such as n-1, n-2, n-3, or n-4 that satisfies i<n. That is, if no pixel points matching the preset number of pixel points in the motion region of the corrected image are found in the current rolling image of the nth frame, then the rolling image of the i(n-1)th frame may search for pixel points matching each pixel point in the motion region of the corrected image, and the pixel values ​​of each pixel point in the motion region may be replaced to output a replacement image. If no pixel points matching the preset number of pixel points in the motion region of the corrected image are found in the rolling image of the n-1th frame (where i=n-1), then the rolling image of the n-2th frame (where i=n-2) may search for pixel points matching each pixel point in the motion region of the corrected image, and the pixel values ​​of each pixel point in the motion region may be replaced to output a replacement image. The operation of step 083 is substantially the same as that of step 081 and will not be described in detail herein.

[0140] It should be noted that in some embodiments, the preset number may be any value in the range (a, b), where a is the minimum number of updated pixels in the motion region of the corrected image sufficient to interpolate the pixels in the motion region of the corrected image; and b is the total number of pixels in the motion region of the corrected image. Specifically, if the number of pixels found in the current rolling image of the nth frame that match the pixels in the motion region of the corrected image is less than a, then the rolling image of the ith frame may be searched for pixels that match the pixels in the motion region of the corrected image, and the pixel values ​​of the pixels in the motion region may be replaced to output a replacement image, where i < n. If the number of pixel points that match the pixel points in the motion area of ​​the corrected image in the rolling image of the i-th (i=nx, (x≥1, x is an integer)) frame is b, then the replacement image obtained in step 083 can be directly output as the target image; if the number of pixel points that match the pixel points in the motion area of ​​the corrected image in the rolling image of the i-th (i=nx, (x≥1, x is an integer)) frame is (a, b), then the replacement image obtained in step 083 can be directly output or further processed as described below.

[0141] That is, see Figure 18 In some embodiments, if the number of pixels in the rolling shutter image of the current nth frame that match the pixels in the motion region of the corrected image is within (a, b), then the image processing method may further include:

[0142] 09: Using the pixel values ​​of the pixels after successful matching and replacement, the pixel values ​​of the pixels that have not been matched and replaced in the motion area of ​​the corrected image are interpolated and filled to obtain the target image.

[0143] Please combine Figure 2 The interpolation module 19 is used to use the pixel values ​​of the pixels after successful matching and replacement to interpolate and fill the pixel values ​​of the pixels that are not matched and replaced in the motion area of ​​the corrected image to obtain the target image.

[0144] Please combine Figure 3 The program in the electronic device 100 of the embodiment of the present application includes a program for executing the image processing method in 09. That is, the one or more processors 60 are used to use the pixel values ​​of the pixel points after successful matching and replacement to interpolate and fill the pixel values ​​of the pixel points that have not been matched and replaced in the motion area of ​​the corrected image to obtain the target image.

[0145] Because the global shutter camera 30 and the rolling shutter camera 40 are not completely overlapped, the global image obtained by the global shutter camera 30 is different from the rolling shutter image obtained by the rolling shutter camera 40. Furthermore, due to motion blur, it may not be possible to find all the pixel points that match the pixels in the motion region of the corrected image in the current n-th frame rolling shutter image and the i-th frame rolling shutter image. Therefore, it is necessary to interpolate and fill the pixel values ​​of the unmatched pixel points in the motion region of the corrected image based on the pixel values ​​of the successfully matched pixel points to obtain the target image. It should be noted that in some embodiments, the interpolation algorithm may be nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc., which are not limited here.

[0146] If, in step 081, pixel points matching a preset number of pixel points in the motion region of the corrected image are found in the rolling shutter image of the current frame n, then step 09 can be performed directly after step 081. That is, the pixel values ​​of the successfully matched and replaced pixels are used to interpolate and fill the pixel values ​​of the unmatched and replaced pixels in the motion region of the corrected image to obtain the target image, which can be directly output. This can improve the efficiency of image processing to a certain extent.

[0147] In addition, in some embodiments, if at least one pixel point that matches each pixel point in the motion area of ​​the corrected image is found in the current rolling image of the nth frame, but the pixel points in the motion area of ​​the corrected image are not completely updated, it is also possible to find pixel points that match each pixel point in the motion area of ​​the corrected image in the i-th frame rolling image, and replace the pixel values ​​of each pixel point in the motion area, where i<n. In some embodiments, if at least one pixel matching each pixel in the motion region of the corrected image is still found in the rolling image of the i-th (i=n-1) frame, but the pixels in the motion region of the corrected image have not been completely updated, then the search for pixel matching each pixel in the motion region of the corrected image may continue in the rolling image of the i-th (i=n-2) frame until all the pixels in the motion region of the corrected image are found in the rolling image of the i-th (i=nx, (x≥1, x is an integer)) frame. The resulting replacement image may then be directly output as the target image. Alternatively, if no pixel corresponding to the pixel in the motion region of the corrected image is found in the rolling image of the i-th (i=nx, (x≥1, x is an integer)) frame, the replacement image obtained at this time may be directly output. Among them, if no pixel points corresponding to the pixel points in the motion area of ​​the corrected image can be found in the rolling shutter image of the i-th (i=nx, (x≥1, x is an integer)) frame, the pixel values ​​of the pixel points after successful matching and replacement can be used to interpolate and fill the pixel values ​​of the pixel points in the motion area of ​​the corrected image that have not been matched and replaced to obtain the target image, which can be directly output.

[0148] See also Figure 1 and Figure 19 The present application also provides a computer-readable storage medium 200 on which a computer program 210 is stored. When the computer program 210 is executed by one or more processors 220, the image processing method of any embodiment described above is implemented.

[0149] For example, when the program 210 is executed by the processor 220, the following image processing method is implemented:

[0150] 01: Acquire a global image, a rolling image, and a depth image. The global image, rolling image, and depth image frames are synchronized. The global image is captured by a global shutter camera 30, the rolling image is captured by a rolling shutter camera 40, and the depth image is captured by a depth camera 50. The resolution of the global image is smaller than that of the rolling image, but larger than that of the depth image.

[0151] 03: Obtain the first coordinates of the first projection point of each pixel in the depth image reprojected onto the global image, and obtain the second coordinates of the second projection point of each pixel in the depth image reprojected onto the rolling image;

[0152] 05: Obtain a first motion region of the global image based on the first coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the global image, and obtain a second motion region of the rolling image based on the second coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the rolling image;

[0153] 07: Correcting the pixel value of each pixel in the second motion area according to the pixel value of each pixel in the first motion area to obtain a corrected image;

[0154] 08: In the rolling shutter image, find the pixels that match the pixels in the motion area of ​​the correction image, and replace the pixel values ​​of the pixels in the motion area to obtain a replacement image.

[0155] For another example, when the program 210 is executed by the processor 220, the following image processing method is implemented:

[0156] 021: Convert the coordinates of each pixel in the depth image into point cloud coordinates according to the principal point coordinates of the depth camera 50, the coordinates of the pixel in the depth image, the depth value of the pixel in the depth image, and the intrinsic reference focal length of the depth camera 50;

[0157] 023: Convert the point cloud coordinates to the first coordinate system of the global shutter camera 30 and the second coordinate system of the rolling shutter camera 40, respectively, to obtain first image coordinates corresponding to the point cloud coordinates in the first coordinate system and second image coordinates corresponding to the point cloud coordinates in the second coordinate system;

[0158] 025: Obtain the first coordinates of the first projection point on the global image according to the first image coordinates and the intrinsic parameter matrix of the global shutter camera 30 , and obtain the second coordinates of the second projection point on the rolling shutter image according to the second image coordinates and the intrinsic parameter matrix of the rolling shutter camera 40 .

[0159] For another example, when program 210 is executed by processor 220, it can also implement the image processing methods in 031, 033, 035, 0351, 0353, 03511, 03513, 071, 073, 075, 0751, 0753, 081, 0811, 0813, 0815, 0817, 083 and 09.

[0160] It should be noted that the explanations of the image processing method and the image processing device 10 in the aforementioned embodiments are also applicable to the computer-readable storage medium 200 in the embodiments of the present application, and will not be elaborated here.

[0161] In the non-volatile computer-readable storage medium 200 in the present application, the first coordinates of the first projection point where each pixel point in the depth image is reprojected onto the global image are obtained, and the second coordinates of the second projection point where each pixel point in the depth image is reprojected onto the rolling image are obtained, and the first motion area of ​​the global image is obtained according to the first coordinates and the second motion area of ​​the rolling image is obtained according to the second coordinates. Then, the pixel values ​​of each pixel point in the second motion area are corrected according to the pixel values ​​of each pixel point in the first motion area to obtain a corrected image. Finally, in the rolling image, the pixel points that match the pixel points in the motion area of ​​the corrected image are found, and the pixel values ​​of each pixel point in the motion area are replaced to obtain a replaced image. Therefore, the entire image processing process can correct the image acquired by the electronic device 100 when the photographed object moves relative to the electronic device 100, ensure the image correction effect, and improve the imaging quality of the electronic device 100.

[0162] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0163] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0164] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner as necessary, and then stored in a computer memory.

[0165] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0166] Those skilled in the art will appreciate that all or part of the steps carried out in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, the various functional units in the various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk or an optical disk, etc.

[0167] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are illustrative and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. An image processing method for an electronic device comprising a common-baseline global shutter camera, a rolling shutter camera, and a depth camera, characterized in that: include: Acquire a global image, a rolling image, and a depth image, wherein the global image, the rolling image, and the depth image are frame-synchronized, the global image is captured by the global shutter camera, the rolling image is captured by the rolling shutter camera, and the depth image is captured by the depth camera, and the resolution of the global image is smaller than the resolution of the rolling image and greater than the resolution of the depth image; Obtaining first coordinates of a first projection point on the global image where each pixel in the depth image is reprojected, and obtaining second coordinates of a second projection point on the rolling image where each pixel in the depth image is reprojected; Obtaining a first motion region of the global image according to the first coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the global image, and obtaining a second motion region of the rolling image according to the second coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the rolling image; Correcting the pixel value of each pixel point in the second motion area according to the pixel value of each pixel point in the first motion area to obtain a corrected image; and In the rolling shutter image, pixel points that match the pixel points in the motion area of ​​the corrected image are found, and pixel values ​​of the pixel points in the motion area are replaced to obtain a replacement image.

2. The image processing method according to claim 1, wherein: The obtaining of first coordinates of a first projection point where each pixel point in the depth image is reprojected onto the global image, and obtaining second coordinates of a second projection point where each pixel point in the depth image is reprojected onto the rolling image, include: Converting the coordinates of each pixel in the depth image into point cloud coordinates according to the principal point coordinates of the depth camera, the coordinates of the pixel in the depth image, the depth value of the pixel in the depth image, and the intrinsic reference focal length of the depth camera; Converting the point cloud coordinates into a first coordinate system of the global shutter camera and a second coordinate system of the rolling shutter camera, respectively, to obtain first image coordinates corresponding to the point cloud coordinates in the first coordinate system and second image coordinates corresponding to the point cloud coordinates in the second coordinate system; and The first coordinates of the first projection point on the global image are obtained according to the first image coordinates and the intrinsic parameter matrix of the global shutter camera, and the second coordinates of the second projection point on the rolling shutter image are obtained according to the second image coordinates and the intrinsic parameter matrix of the rolling shutter camera.

3. The image processing method according to claim 1, wherein: The obtaining, according to the first coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the global image, of the first motion region of the global image, and obtaining, according to the second coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the rolling image, the second motion region of the rolling image, includes: Traversing each pixel in the depth image, obtaining a first pixel value of the first projection point in the global image according to the first coordinate of the corresponding first projection point, and obtaining a second pixel value of the second projection point in the rolling image according to the second coordinate of the corresponding second projection point; If the difference between the first pixel value and the second pixel value is greater than a preset first threshold, identifying the pixel in the depth image as a moving pixel; and The first projection points corresponding to the moving pixel points in the global image are expanded to obtain the first moving area, and the second projection points corresponding to the moving pixel points in the rolling image are expanded to obtain the second moving area.

4. The image processing method according to claim 3, wherein: The step of expanding the first projection points corresponding to the moving pixel points in the global image to obtain the first motion region, and expanding the second projection points corresponding to the moving pixel points in the rolling image to obtain the second motion region, includes: determining a first dilation amount of the first projection point corresponding to the moving pixel point in the global image and a second dilation amount of the second projection point corresponding to the moving pixel point in the rolling image; and An area formed by dilating the first projection point corresponding to the moving pixel point in the global image according to the first dilation amount is the first moving area, and an area formed by dilating the second projection point corresponding to the moving pixel point in the rolling image according to the second dilation amount is the second moving area.

5. The image processing method according to claim 4, characterized in that The determining a first dilation quantity of the first projection point corresponding to the moving pixel point in the global image and a second dilation quantity of the second projection point corresponding to the moving pixel point in the rolling image includes: Obtaining the first dilation amount according to a resolution difference between the global shutter camera and the depth camera; and The second dilation amount is obtained according to a resolution difference between the rolling shutter camera and the depth camera.

6. The image processing method according to claim 1, wherein: Correcting the pixel value of each pixel point in the second motion area according to the pixel value of each pixel point in the first motion area to obtain a corrected image includes: In the first motion area, obtaining mapping coordinates corresponding to pixel points in the second motion area; When the mapping coordinates are all integer values, the pixel value of the pixel point in the second motion area is replaced by the pixel value of the pixel point under the mapping coordinates; and When at least one of the mapping coordinates is a decimal value, the pixel value of the pixel point under the mapping coordinate in the first motion area is obtained by interpolation, and the pixel value of the pixel point in the second motion area is updated to the pixel value obtained by interpolation.

7. The image processing method according to claim 6, characterized in that: Obtaining pixel values ​​of pixel points at the mapping coordinates in the first motion area by interpolation includes: Obtain four test pixels in the first motion area that are closest to the pixel point at the mapping coordinates; and The pixel values ​​of the four test pixels are weighted according to preset weights to obtain the pixel value of the pixel at the mapping coordinate in the first motion area.

8. The image processing method according to claim 1, wherein: The step of finding, in the rolling image, pixel points that match the pixel points in the motion region of the corrected image, and replacing the pixel values ​​of the pixel points in the motion region to output a replacement image comprises: In the rolling shutter image of the current n-th frame, pixel points matching the pixel points in the motion area of ​​the corrected image are searched, and pixel values ​​of the pixel points in the motion area are replaced to output a replacement image.

9. The image processing method according to claim 8, characterized in that: If no pixel points matching a preset number of pixel points in the motion region of the corrected image are found in the rolling shutter image of the current n-th frame, the method further includes: In the rolling shutter image of the i-th frame, pixel points matching the pixel points in the motion area of ​​the corrected image are searched, and pixel values ​​of the pixel points in the motion area are replaced to output a replacement image, where i<n.

10. The image processing method according to claim 8 or 9, characterized in that: The searching for pixel points that match the pixel points in the motion region of the corrected image and replacing the pixel values ​​of the pixel points in the motion region to output a replacement image includes: Acquire a first image block centered on each pixel point in the motion area of ​​the corrected image; Acquire a plurality of second image blocks centered on all pixels in the rolling shutter image; matching the first image block with a plurality of the second image blocks; and The pixel value of the pixel point corresponding to the second image block whose matching result is greater than a preset second threshold is updated to the pixel value of the pixel point corresponding to the first image block.

11. The image processing method according to claim 8, wherein: Also includes: The pixel values ​​of the pixels that have been successfully matched and replaced are used to interpolate and fill the pixel values ​​of the pixels that have not been matched and replaced in the motion area of ​​the corrected image to obtain a target image.

12. An image processing device, characterized in that: For use in an electronic device, the electronic device including a common-baseline global shutter camera, a rolling shutter camera, and a depth camera, including: a first acquisition module, configured to acquire a global image, a rolling image, and a depth image, wherein the global image, the rolling image, and the depth image are frame-synchronized, the global image is captured by the global shutter camera, the rolling image is captured by the rolling shutter camera, and the depth image is captured by the depth camera, and the resolution of the global image is smaller than the resolution of the rolling image and larger than the resolution of the depth image; a second acquisition module, configured to acquire first coordinates of a first projection point where each pixel point in the depth image is reprojected onto the global image, and to acquire second coordinates of a second projection point where each pixel point in the depth image is reprojected onto the rolling image; a third acquisition module, configured to acquire a first motion region of the global image based on the first coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the global image, and to acquire a second motion region of the rolling image based on the second coordinates corresponding to each pixel in the depth image and the pixel value of each pixel in the rolling image; a correction module, configured to correct the pixel value of each pixel point in the second motion area according to the pixel value of each pixel point in the first motion area to obtain a corrected image; and The replacement module is used to find pixel points in the rolling shutter image that match the pixel points in the motion area of ​​the corrected image, and replace the pixel values ​​of the pixel points in the motion area to obtain a replacement image.

13. An electronic device, characterized in that: include: A global shutter camera, a rolling shutter camera, and a depth camera with a common baseline, wherein the global shutter camera is used to capture a global image, the rolling shutter camera is used to capture the rolling image, and the depth camera is used to capture a depth image; One or more processors, memory; and One or more computer programs, wherein one or more of the computer programs are stored in the memory, and when the computer programs are executed by the processor, the image processing method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method according to any one of claims 1 to 11 is implemented.

Images