Image correction method, device, system, and electronic equipment

Abstract

The present invention discloses an image correction method, device, system, and electronic device. The image correction method comprises: obtaining a visible light image and a depth image captured of a target object, transforming them to form a base image pair, wherein the base image pair comprises a first image and a second image; correcting the base image pair using a preset correction mode to obtain a plurality of correction parameters; and aligning and correcting the base image pair based on each correction parameter to obtain a target image pair. The present invention solves the technical problems in related technologies of being unable to achieve dynamic correction between two different cameras, having low adaptability to different environments, resulting in poor image alignment, and easily affecting user interest.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image correction method, device and system, and electronic equipment. Background Art

[0002] In related technologies, with the continuous improvement of hardware technology, depth imaging has become more and more accurate, and applications based on depth information have also developed rapidly. Currently, common depth imaging methods are mainly divided into three types: binocular stereo imaging, structured light imaging, and time of flight imaging (ToF).

[0003] Binocular stereo imaging requires simultaneous image capture using two RGB cameras, followed by binocular matching and triangulation to determine depth information. This approach offers advantages such as low cost, low power consumption, and relatively high image resolution. However, because depth is calculated entirely through algorithms, it requires high computing resources, suffers from poor real-time performance, and is sensitive to the imaging environment.

[0004] Structured light imaging uses a camera to emit a laser pattern (speckle or dot pattern). When the object being measured reflects this pattern, the camera captures the reflected pattern and calculates the size of the speckle or dot pattern, thereby measuring the distance between the object and the camera. The main advantage of structured light imaging is that it is not affected by the texture of the object. However, the laser speckle pattern is obscured in strong sunlight, making it unsuitable for outdoor use.

[0005] ToF cameras use the time difference between the transmitted and reflected signals to directly obtain depth information of the measured point. Their advantages include high real-time performance and immunity to changes in lighting and object texture. However, these cameras generally have low image resolution, large modules, and high hardware costs.

[0006] When using various terminal devices, it is necessary to align the relative positions of the depth camera and the visible light camera. Currently, in order to reduce the shaking during shooting, many visible light cameras use optical image stabilization (OIS) and autofocus (AF) to improve the clarity of the captured image. These mechanisms will cause the relative position of the two cameras to change, and the focal length and optical center of the camera to change. In addition, the falling of the device, the asynchronous frames between the cameras, or the different frame rates will also cause the relative position of the two cameras to change. When these changes occur, the intrinsic and extrinsic parameters of the two cameras have changed. If the calibrated parameters are still used, the alignment accuracy between the depth image and the visible light image will be affected, and the effect of subsequent algorithms that rely on depth information will be affected. In many cases, there is no environment for recalibration, and dynamic correction cannot be achieved, resulting in poor image alignment and affecting user interest.

[0007] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0008] Embodiments of the present invention provide an image correction method, device, system, and electronic device to at least solve the technical problems in the related art that dynamic correction between two different cameras cannot be achieved, the adaptability to the environment is low, resulting in poor image alignment effect, and easily affecting user interest in use.

[0009] According to one aspect of an embodiment of the present invention, there is provided an image correction method, comprising: acquiring a visible light image and a depth image taken of a target object, and forming a basic image pair after transformation, wherein the basic image pair comprises a first image and a second image; performing correction processing on the basic image pair using a preset correction mode to obtain a plurality of correction parameters; and performing alignment correction on the basic image pair based on each of the correction parameters to obtain a target image pair.

[0010] Optionally, the step of correcting the basic image pair using a preset correction mode to obtain a plurality of correction parameters includes: scaling the basic image pair to a preset resolution, and performing pyramid correction processing to obtain the plurality of correction parameters.

[0011] Optionally, the step of obtaining a visible light image and a depth image taken of the target object and forming a basic image pair after transformation includes: based on preset calibration parameters, transforming the above-mentioned depth image to the image coordinate system of the above-mentioned visible light image, and adjusting it to obtain a preliminary aligned depth map with the same resolution as the above-mentioned visible light image, wherein the above-mentioned visible light image and the above-mentioned preliminary aligned depth map are combined to form the above-mentioned basic image pair, the above-mentioned first image is the above-mentioned visible light image, and the above-mentioned second image is the said preliminary aligned depth map.

[0012] Optionally, the step of correcting the above-mentioned basic image pair using a preset correction mode to obtain multiple correction parameters also includes: determining the target translation parameters and target scaling coefficients between the above-mentioned first image and the above-mentioned second image; and determining multiple correction parameters based on the above-mentioned target translation parameters and the above-mentioned target scaling coefficients.

[0013] Optionally, before correcting the above-mentioned basic image pair using a preset correction mode to obtain multiple correction parameters, the above-mentioned image correction method also includes: preprocessing the preliminary aligned depth map in the above-mentioned basic image pair to obtain the above-mentioned first image; and filtering the visible light image in the above-mentioned basic image pair to obtain the above-mentioned second image.

[0014] Optionally, the step of determining the target translation parameters and target scaling coefficients between the first image and the second image includes: calculating the target translation parameters of the first image relative to the second image, and translating the first image based on the target translation parameters to obtain a third image; selecting multiple scaling coefficients, and scaling the third image with each scaling coefficient, and calculating the image matching score between the third image and the second image; and taking the scaling coefficient corresponding to the smallest score among the multiple image matching scores as the target scaling coefficient.

[0015] Optionally, the step of determining the target translation parameters and target scaling coefficients between the first image and the second image includes: calculating the target translation parameters of the first image relative to the second image, and translating the first image based on the target translation parameters to obtain a fourth image; selecting multiple scaling coefficients, and scaling the fourth image with each scaling coefficient, and calculating the image matching score between the fourth image and the second image; adjusting the scaling coefficient until the change in the image matching score is less than a first threshold value, and the scaling coefficient corresponding to the image matching score is used as the target scaling coefficient.

[0016] Optionally, the step of determining the target translation parameters and target scaling coefficient between the first image and the second image includes: selecting multiple scaling coefficients, and scaling the first image with each of the scaling coefficients; sliding the first image scaled based on each of the scaling coefficients on the second image, and calculating the image matching score between it and the second image; and taking the scaling coefficient and translation amount corresponding to the smallest score among the multiple image matching scores as the target scaling coefficient and the target translation parameters.

[0017] Optionally, the step of preprocessing the preliminary aligned depth map in the above-mentioned basic image pair to obtain the above-mentioned first image includes: mapping the depth value of each pixel point in the preliminary aligned depth map in the above-mentioned basic image pair to a preset pixel range; and / or adjusting the image contrast of the above-mentioned preliminary aligned depth map to obtain the above-mentioned first image.

[0018] Optionally, the step of determining the target translation parameters and target scaling coefficients between the first image and the second image includes: extracting image features of the first image to obtain a first feature subset, wherein the first feature subset includes a first distance image, a first boundary direction map and first mask information; extracting image features of the second image to obtain a second feature subset, wherein the second feature subset includes a second distance image and a second boundary direction map; and calculating the target translation parameters of the first image relative to the second image based on the first feature subset and the second feature subset.

[0019] Optionally, the step of extracting the image features of the above-mentioned first image to obtain a first feature subset includes: extracting all boundary pixel points of each target object in the above-mentioned first image to obtain a first edge image; performing inversion processing on the above-mentioned first edge image to obtain a second edge image; extracting the contour of the above-mentioned first edge image to obtain a first contour array, and calculating the pixel direction corresponding to each pixel point based on the above-mentioned first contour array to obtain a first contour direction array; performing a preset distance transformation processing on the above-mentioned second edge image based on a first preset distance threshold to obtain the above-mentioned first distance image; calculating the first boundary direction map corresponding to the boundaries of each target object in the above-mentioned second edge image based on the above-mentioned first contour direction array; and determining the above-mentioned first feature subset based on the above-mentioned first distance image and the above-mentioned first boundary direction map.

[0020] Optionally, based on a first preset distance threshold, the second edge image is subjected to a preset distance transformation process to obtain a first distance image, comprising: determining the first mask information based on the first preset distance threshold, wherein the first mask information is used to shield part of the edge information in the second image; and adding the first mask information to the first feature subset.

[0021] Optionally, the step of extracting image features of the second image to obtain a second feature subset includes: extracting all boundary pixel points of each target object in the second image to obtain a third edge image; using the first mask information to delete the contours in the third edge image; performing inversion processing on the deleted third edge image to obtain a fourth edge image; extracting contours from the fourth edge image to obtain a second contour array, and calculating the pixel direction corresponding to each pixel based on the second contour array to obtain a second contour direction array; performing a preset distance transformation on the fourth edge image based on a second preset distance threshold to obtain the second distance image; calculating the second boundary direction map corresponding to the boundaries of each target object in the fourth edge image based on the second contour direction array; and obtaining the second feature subset based on the second distance image and the second boundary direction map.

[0022] Optionally, the step of calculating the target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset includes: using a first judgment condition to extract contour pixel points whose pixel distance between the first distance image and the second distance image is less than a first distance threshold, to obtain a first contour pixel point set participating in image matching; using a second judgment condition to extract contour pixel points whose pixel distance between the first boundary direction map and the second boundary direction map is less than a second distance threshold, to obtain a second contour pixel point set participating in image matching; based on the first contour pixel point set and the second contour pixel point set, determining a chamfer distance score, a direction map distance, and an image adjustment factor between the first image and the second image, wherein the image adjustment factor is used to adjust the chamfer distance score and the direction map distance ratio; sliding the second image on the first image, and inputting the chamfer distance score, the direction map distance, and the image adjustment factor into a first preset formula to calculate the image sliding score; determining a target sliding position corresponding to the minimum score among all image sliding scores; and determining a target translation parameter based on the target sliding position.

[0023] Optionally, the step of scaling the base image pair to a preset resolution and performing pyramid correction processing to obtain the multiple correction parameters includes: obtaining an alignment accuracy value of a terminal application, and determining multiple correction resolutions based on the alignment accuracy value and the resolution of the base image pair, wherein the multiple correction resolutions include at least a preset resolution, which is the minimum resolution among the multiple correction resolutions; scaling the base image pair to the preset resolution and performing pyramid correction processing until the alignment accuracy value is met, thereby obtaining the multiple correction parameters.

[0024] Optionally, the above-mentioned image correction method also includes: determining the image alignment requirement accuracy of the terminal application at the target image resolution; step S1, judging whether the current alignment accuracy image corresponding to the above-mentioned target image pair reaches the above-mentioned image alignment requirement accuracy at the above-mentioned first image resolution; step S2, if it is determined that the current alignment accuracy image corresponding to the above-mentioned target image pair does not reach the above-mentioned image alignment requirement accuracy, adjusting the image resolution to the second image resolution, wherein the resolution value of the above-mentioned second image resolution is higher than the above-mentioned first image resolution; step S3, executing the step of correcting the above-mentioned basic image pair using a preset correction mode to obtain multiple correction parameters; step S4, executing the step of aligning and correcting the above-mentioned basic image pair based on each of the above-mentioned correction parameters to obtain the target image pair; repeating steps S1 to S4 until the current alignment accuracy image reaches the above-mentioned image alignment requirement accuracy.

[0025] Optionally, the above-mentioned image correction method also includes: comparing the image resolution of the above-mentioned visible light image and the image resolution of the above-mentioned depth image to obtain a comparison result with the minimum resolution; calculating a correction number threshold based on the image resolution obtained from the comparison result and the initial set maximum resolution of the correction processing; during the alignment correction process, if the number of image corrections reaches the above-mentioned correction number threshold, the correction processing is stopped.

[0026] According to another aspect of an embodiment of the present invention, an image correction device is also provided, including: an acquisition unit, used to acquire a visible light image and a depth image taken of a target object, and form a basic image pair after transformation, wherein the above-mentioned basic image pair includes a first image and a second image; a first correction unit, used to correct the above-mentioned basic image pair using a preset correction mode to obtain multiple correction parameters; and a second correction unit, used to align and correct the above-mentioned basic image pair based on each of the above-mentioned correction parameters to obtain a target image pair.

[0027] Optionally, the first correction unit includes: a first correction module, configured to scale the basic image pair to a preset resolution, and perform pyramid correction processing to obtain the plurality of correction parameters.

[0028] Optionally, the acquisition unit includes: a first transformation module, used to transform the depth image into the image coordinate system of the visible light image based on preset calibration parameters, and adjust it to obtain a preliminary aligned depth map with the same resolution as the visible light image, wherein the visible light image and the preliminary aligned depth map are combined to form the basic image pair, the first image is the visible light image, and the second image is the preliminary aligned depth map.

[0029] Optionally, the first correction unit further includes: a first determination module for determining a target translation parameter and a target scaling factor between the first image and the second image; and a second determination module for determining a plurality of correction parameters based on the target translation parameter and the target scaling factor.

[0030] Optionally, the above-mentioned image correction device also includes: a first processing unit, which is used to pre-process the preliminary aligned depth map in the above-mentioned basic image pair before correcting the above-mentioned basic image pair using a preset correction mode to obtain multiple correction parameters, so as to obtain the above-mentioned first image; and a second processing unit, which is used to filter the visible light image in the above-mentioned basic image pair to obtain the above-mentioned second image.

[0031] Optionally, the above-mentioned first determination module includes: a first calculation module, used to calculate the above-mentioned target translation parameters of the above-mentioned first image relative to the above-mentioned second image, and translate the above-mentioned first image based on the above-mentioned target translation parameters to obtain a third image; a first scaling module, used to select multiple scaling coefficients, and scale the above-mentioned third image with each of the above-mentioned scaling coefficients respectively, and calculate the image matching score between the above-mentioned third image and the above-mentioned second image; a second determination module, used to use the scaling coefficient corresponding to the smallest score among the multiple image matching scores as the target scaling coefficient.

[0032] Optionally, the above-mentioned first determination module also includes: a second calculation module, used to calculate the above-mentioned target translation parameters of the above-mentioned first image relative to the above-mentioned second image, and translate the above-mentioned first image based on the above-mentioned target translation parameters to obtain a fourth image; a second scaling module, used to select multiple scaling coefficients, and scale the above-mentioned fourth image with each of the above-mentioned scaling coefficients respectively, and calculate the image matching score between the above-mentioned fourth image and the above-mentioned second image; a third determination module, used to adjust the above-mentioned scaling coefficient until the score change in the above-mentioned image matching score is less than the first threshold value, and the scaling coefficient corresponding to the above-mentioned image matching score is used as the target scaling coefficient.

[0033] Optionally, the above-mentioned first determination module also includes: a third scaling module, used to select multiple scaling factors, and scale the above-mentioned first image with each of the above-mentioned scaling factors respectively; a third calculation module, used to slide the above-mentioned first image scaled based on each of the above-mentioned scaling factors on the above-mentioned second image, and calculate the image matching score between it and the above-mentioned second image; a fourth determination module, used to use the scaling factor and translation amount corresponding to the smallest score among the multiple image matching scores as the above-mentioned target scaling factor and the above-mentioned target translation parameter.

[0034] Optionally, the above-mentioned first processing unit includes: a first mapping module, used to map the depth value of each pixel point in the preliminary aligned depth map in the above-mentioned basic image pair to a preset pixel range; and / or, a first adjustment module, used to adjust the image contrast of the above-mentioned preliminary aligned depth map to obtain the above-mentioned first image.

[0035] Optionally, the above-mentioned first determination module also includes: a first extraction module, used to extract image features of the above-mentioned first image to obtain a first feature subset, wherein the above-mentioned first feature subset includes a first distance image, a first boundary direction map and first mask information; a second extraction module, used to extract image features of the above-mentioned second image to obtain a second feature subset, wherein the above-mentioned second feature subset includes a second distance image and a second boundary direction map; a fourth calculation module, used to calculate the target translation parameters of the above-mentioned first image relative to the above-mentioned second image based on the above-mentioned first feature subset and the above-mentioned second feature subset.

[0036] Optionally, the above-mentioned first extraction module includes: a first extraction submodule, used to extract all boundary pixel points of each target object in the above-mentioned first image to obtain a first edge image; a first inversion submodule, used to perform inversion processing on the above-mentioned first edge image to obtain a second edge image; a second extraction submodule, used to extract the contour of the above-mentioned first edge image to obtain a first contour array, and calculate the pixel direction corresponding to each pixel point based on the above-mentioned first contour array to obtain a first contour direction array; a first transformation submodule, used to perform a preset distance transformation processing on the above-mentioned second edge image based on a first preset distance threshold to obtain a first distance image; a first calculation submodule, used to calculate the first boundary direction map corresponding to the boundaries of each target object in the above-mentioned second edge image based on the above-mentioned first contour direction array; a first determination submodule, used to determine the first feature subset based on the above-mentioned first distance image and the above-mentioned first boundary direction map.

[0037] Optionally, the above-mentioned first transformation submodule includes: a second determination submodule, used to determine the first mask information based on the above-mentioned first preset distance threshold, wherein the above-mentioned first mask information is used to shield part of the edge information in the above-mentioned second image; and an adding submodule, used to add the above-mentioned first mask information to the above-mentioned first feature subset.

[0038] Optionally, the above-mentioned second extraction module includes: a second extraction submodule, used to extract all boundary pixel points of each target object in the above-mentioned second image to obtain a third edge image; a deletion submodule, used to use the above-mentioned first mask information to delete the contours in the above-mentioned third edge image; a second inversion submodule, used to invert the above-mentioned third edge image after deletion to obtain a fourth edge image; a second calculation submodule, used to extract the contours of the above-mentioned fourth edge image to obtain a second contour array, and calculate the pixel direction corresponding to each pixel point based on the above-mentioned second contour array to obtain a second contour direction array; a second transformation submodule, used to perform a preset distance transformation on the above-mentioned fourth edge image based on a second preset distance threshold to obtain a second distance image; a third calculation submodule, used to calculate the second boundary direction map corresponding to the boundaries of each target object in the above-mentioned fourth edge image based on the above-mentioned second contour direction array; a third determination submodule, used to obtain a second feature subset based on the above-mentioned second distance image and the above-mentioned second boundary direction map.

[0039] Optionally, the fourth calculation module includes: a third extraction submodule, configured to extract, using a first judgment condition, contour pixels from the first range image and the second range image whose pixel distance is less than a first distance threshold, to obtain a first contour pixel set participating in image matching; a fourth extraction submodule, configured to extract, using a second judgment condition, contour pixels from the first boundary direction map and the second boundary direction map whose pixel distance is less than a second distance threshold, to obtain a second contour pixel set participating in image matching; a fifth determination submodule, configured to determine, based on the first contour pixel set and the second contour pixel set, a chamfer distance score, a direction map distance, and an image adjustment factor between the first image and the second image, wherein the image adjustment factor is used to adjust the chamfer distance score and the direction map distance ratio; a fourth calculation submodule, configured to slide the second image on the first image, and input the chamfer distance score, the direction map distance, and the image adjustment factor into a first preset formula to calculate an image sliding score; a sixth determination submodule, configured to determine a target sliding position corresponding to the minimum score among all image sliding scores; and a seventh determination submodule, configured to determine a target translation parameter based on the target sliding position.

[0040] Optionally, the above-mentioned first correction module includes: a first acquisition submodule, used to obtain the alignment accuracy value of the terminal application, and determine multiple correction resolutions based on the above-mentioned alignment accuracy value and the resolution of the above-mentioned basic image pair, wherein the above-mentioned multiple correction resolutions at least include: a preset resolution, and the above-mentioned preset resolution is the minimum resolution among the multiple correction resolutions; a first correction submodule, used to scale the above-mentioned basic image pair to the above-mentioned preset resolution, and perform pyramid correction processing until the above-mentioned alignment accuracy value is met, thereby obtaining the above-mentioned multiple correction parameters.

[0041] Optionally, the above-mentioned image correction device also includes: a determination unit, used to determine the image alignment requirement accuracy of the terminal application at the target image resolution; a first judgment unit, used to execute step S1, to determine whether the current alignment accuracy image corresponding to the above-mentioned target image pair reaches the above-mentioned image alignment requirement accuracy at the above-mentioned first image resolution; a first adjustment unit, used to execute step S2, if it is determined that the current alignment accuracy image corresponding to the above-mentioned target image pair does not reach the above-mentioned image alignment requirement accuracy, adjust the image resolution to the second image resolution, wherein the resolution value of the above-mentioned second image resolution is higher than the above-mentioned first image resolution; a first execution unit, used to execute step S3, execute the step of correcting the above-mentioned basic image pair using a preset correction mode to obtain multiple correction parameters; a second execution unit, used to execute step S4, execute the step of aligning and correcting the above-mentioned basic image pair based on each of the above-mentioned correction parameters to obtain the target image pair; repeat steps S1 to S4 until the current alignment accuracy image reaches the above-mentioned image alignment requirement accuracy.

[0042] Optionally, the above-mentioned image correction device also includes: a comparison unit, used to compare the image resolution of the above-mentioned visible light image and the image resolution of the above-mentioned depth image to obtain a comparison result with the minimum resolution; a calculation unit, used to calculate a correction number threshold based on the image resolution obtained by the comparison result and the initial set maximum resolution of the correction processing; and a stopping unit, used to stop the correction processing if the number of image corrections reaches the above-mentioned correction number threshold during the alignment correction process.

[0043] According to another aspect of an embodiment of the present invention, an image correction system is also provided, including: a first image capture device for capturing a visible light image of a target object; a second image capture device for capturing a depth image of the target object; a correction device for acquiring the visible light image and the depth image captured of the target object, and forming a basic image pair after change, wherein the basic image pair includes a first image and a second image; correcting the basic image pair using a preset correction mode to obtain a plurality of correction parameters; aligning and correcting the basic image pair based on each of the correction parameters to obtain a target image pair; and a result output device for outputting the aligned target image pair to a preset terminal display interface.

[0044] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned image correction methods by executing the above-mentioned executable instructions.

[0045] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned image correction methods.

[0046] In an embodiment of the present invention, a visible light image and a depth image captured of a target object are acquired and transformed to form a base image pair, wherein the base image pair comprises a first image and a second image. A preset correction mode is used to correct the base image pair to obtain multiple correction parameters. Based on each correction parameter, the base image pair is aligned and corrected to obtain a target image pair. In this embodiment, alignment operations can be performed on images captured by multiple cameras to achieve dynamic correction. The correction environment is simple, and alignment correction can be completed using images captured by the device. This solves the technical problems in related technologies such as the inability to achieve dynamic correction between two different cameras, low adaptability to the environment, poor image alignment results, and a tendency to affect user interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0048] Figure 1 is a flow chart of an optional image correction method according to an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of an optional preliminary alignment depth map according to an embodiment of the present invention;

[0050] Figure 3 is an optional superimposed image of the aligned depth image and the visible light image according to an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of an optional image correction device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0053] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, coefficient, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0054] To facilitate those skilled in the art to understand the present invention, some terms involved in the embodiments of the present invention are explained below:

[0055] RGB, Red Green Blue, a color standard, also refers to common color images in this article;

[0056] RGB-D, Red Green Blue-Depth, color-depth map;

[0057] ToF, Time of flight, flight time;

[0058] OIS, Optical Image Stabilization, optical image stabilization;

[0059] AF, Automatic Focus, autofocus;

[0060] FF, Fixed-focus, fixed focus;

[0061] VGA resolution: 640*480 resolution.

[0062] The following embodiments of the present invention are applicable to scenarios including, but not limited to, three-dimensional reconstruction, autonomous driving, facial recognition, object measurement, 3D modeling, background blurring, visible light image and infrared image fusion, visible light image and depth image fusion, VR glasses, and vehicle-mounted imaging equipment. To address image discrepancies caused by complex shooting environments and efficiently perform image alignment, alignment parameters are calculated based on images acquired by multiple cameras. The present invention is applicable to image capture devices that can only provide visible light and infrared images, or only provide visible light and depth images, or can provide visible light, infrared, and depth images and include depth cameras. There are no specific requirements for the type of depth camera; it can be a ToF camera, an infrared camera, a structured light camera, and / or a binocular depth camera.

[0063] The present invention simplifies the calibration process, eliminating the need for a specific environment or specific shooting pattern. Dynamic calibration can be achieved by simply aligning the visible light image and depth image based on pre-set calibration parameters. For camera devices without OIS, the present invention only requires performing dynamic alignment correction at regular intervals to achieve image alignment. Furthermore, for calibration processes requiring high alignment accuracy but less real-time performance, the present invention can perform alignment correction at high resolution, which can then be used for 3D modeling, background blurring, and the fusion of visible light and infrared images.

[0064] The present invention can also be used for the detection and matching of certain objects, with common applications including gesture detection and pedestrian detection. Calibration errors in mobile phone cameras caused by external forces such as drops can be corrected using the present invention for regular automatic calibration or post-sale calibration of terminal devices equipped with depth cameras. For VR glasses and in-vehicle imaging equipment, alignment errors between visible light images and depth images caused by vibration can also be corrected using the present invention. The present invention is described below with reference to various embodiments.

[0065] Example 1

[0066] According to an embodiment of the present invention, an embodiment of an image correction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0067] An embodiment of the present invention provides an image correction method that can be applied to an image correction system, the image correction system comprising: a first image capture device capable of capturing depth images / infrared images (illustrated in the embodiment of the present invention as a depth camera) and a second image capture device capable of capturing visible light images (illustrated in the embodiment of the present invention as a visible light camera). Due to the OIS mechanism, AF mechanism, device drops, and frame asynchrony or frame rate differences between the image capture devices, camera intrinsic and extrinsic parameters can change. Aligning the images of the two cameras using preset calibration parameters results in errors, primarily manifested as translation and scaling issues in the initially aligned image pairs. This is unacceptable for image processing techniques such as three-dimensional reconstruction and background blurring, which require stringent precision. Therefore, when the relative positions or parameters of the image capture devices change, further correction processing is required on the visible light image and depth image aligned using the preset calibration parameters to reduce alignment errors. The embodiment of the present invention provides an image correction method with high alignment accuracy, fast processing speed, real-time capability, and applicability to most practical scenarios.

[0068] The embodiments of the present invention can improve the practicality of image alignment and can perform alignment operations on images captured by multiple cameras. Calibration parameters can be calculated based on visible light images and infrared images, or based on visible light images and depth images. This method is applicable to various devices equipped with ToF depth cameras or structured light depth cameras. The images captured by these devices often have significant texture differences from visible light images, making conventional key point matching solutions unfeasible. However, the technical solution provided by the embodiments of the present invention can still achieve relatively accurate alignment results.

[0069] Figure 1 is a flow chart of an optional image correction method according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0070] Step S102, obtaining a visible light image and a depth image taken of a target object, and transforming them into a basic image pair, wherein the basic image pair includes a first image and a second image;

[0071] Step S104, performing correction processing on the basic image pair using a preset correction mode to obtain a plurality of correction parameters;

[0072] Step S106 , performing alignment correction on the above-mentioned basic image pair based on each correction parameter to obtain a target image pair.

[0073] Through the above steps, a visible light image and a depth image captured of the target object can be obtained, and after transformation, a base image pair can be formed, wherein the base image pair includes a first image and a second image. The base image pair is corrected using a preset correction mode to obtain multiple correction parameters. The base image pair is then aligned and corrected based on each correction parameter to obtain a target image pair. In this embodiment, alignment operations can be performed on images captured by multiple cameras to achieve dynamic correction. The correction environment is simple and alignment correction can be completed using images actually captured by the device, thereby resolving the technical problems in related technologies such as the inability to achieve dynamic correction between two different cameras, low adaptability to the environment, poor image alignment results, and a tendency to affect user interest.

[0074] The following is a detailed description of the above implementation steps.

[0075] Step S102 : obtaining a visible light image and a depth image taken of the target object, and transforming them into a basic image pair, wherein the basic image pair includes a first image and a second image.

[0076] The image capture device used in embodiments of the present invention may include a depth camera and a visible light camera. The depth camera can obtain a depth image and a corresponding infrared image, while the visible light camera obtains a visible light image. In embodiments of the present invention, the depth camera does not need to obtain both depth and infrared images simultaneously. Embodiments of the present invention are applicable to devices that can provide both infrared and depth images, as well as devices that can only provide depth images or infrared images.

[0077] As an optional embodiment of the present invention, a visible light image and a depth image taken of a target object are obtained, and a basic image pair is formed after transformation, including: based on preset calibration parameters, the depth image is transformed into an image coordinate system of the visible light image, and adjusted to obtain a preliminary aligned depth map with the same resolution as the visible light image, wherein the visible light image and the preliminary aligned depth map are combined to form a basic image pair, the first image is the visible light image, and the second image is the preliminary aligned depth map.

[0078] The preset calibration parameters described above are based on the initial calibration of the depth camera and visible light camera, such as the factory calibration parameters. Typically, the resolution of the depth map is smaller than that of the visible light image. After transforming the depth map to the visible light image coordinate system, it can be adjusted using traditional interpolation algorithms or deep learning super-resolution models to obtain a preliminary aligned depth map with the same resolution as the visible light image, facilitating subsequent image alignment and correction.

[0079] Optionally, before correcting the basic image pair using a preset correction mode to obtain multiple correction parameters, the image correction method also includes: preprocessing the preliminary aligned depth map in the basic image pair to obtain a first image; and filtering the visible light image in the basic image pair to obtain a second image.

[0080] Optionally, the step of preprocessing the preliminary aligned depth map in the base image pair to obtain the first image includes: mapping the depth value of each pixel in the preliminary aligned depth map in the base image pair to a preset pixel range; and / or adjusting the image contrast of the preliminary aligned depth map to obtain the first image.

[0081] In an embodiment of the present invention, a visible light image and a depth image are obtained from a visible light camera and a depth camera, respectively. The depth image is transformed into a visible light image coordinate system according to preset calibration parameters to obtain a preliminary aligned depth map with the same resolution as the visible light image. The preliminary aligned depth map is preprocessed to map the depth value of each pixel in the preliminary aligned depth map to a preset pixel range [0, 255]. In addition, in response to existing overexposure problems, the image contrast is adjusted to restore lost details and enhance texture information weakened by overexposure, so as to more effectively perform subsequent alignment correction processing. The preprocessed preliminary aligned depth map is recorded as a first image or template image. Another visible light image is filtered to remove high-frequency noise signals that appear due to sampling when the resolution of the visible light image changes. The obtained image is recorded as a second image or a matching image.

[0082] In practical applications, such as three-dimensional reconstruction, it is usually necessary to align visible light images and depth images. The smaller the alignment error, the more beneficial it is to the subsequent algorithm effect. The goal of the embodiment of the present invention is to reduce the error between the visible light image and the initial aligned depth map. In the embodiment of the present invention, the correction parameters can be calculated based on the visible light image and the depth map, and can also be calculated based on the visible light image and the infrared image. In the embodiment of the present invention, the implementation process is described using the calculation of correction parameters for visible light images and depth images as an example. This implementation process is also applicable to the calculation of correction parameters for visible light images and infrared images.

[0083] Figure 2 is a schematic diagram of an optional preliminary alignment depth map according to an embodiment of the present invention, such as Figure 2 As shown, the preliminary aligned depth map contains the extracted contour and edge information of the main objects.

[0084] Figure 3 is an optional superposition image of the aligned depth image and the visible light image according to an embodiment of the present invention, such as Figure 3 As shown in Figure 2, after adding the visible light image, it is necessary to align the visible light image with the offset part of the object in the preliminary alignment depth map. Figure 3 The offset difference of the image pairs further aligned in the process is very small, and pixel-by-pixel alignment is basically possible.

[0085] The image correction scheme or dynamic correction scheme involved in the embodiments of the present invention is applicable to calibrated devices, that is, the initial camera intrinsic and extrinsic parameters are known. Because factors such as the OIS mechanism, the AF mechanism, and device drops can affect camera intrinsic and inter-camera extrinsic parameters, depth maps and visible light images aligned using known calibration parameters may have misalignment, primarily manifesting as translation and scaling issues in the initially aligned image pair. The embodiments of the present invention correct these translation and scaling issues based on the input visible light and depth images.

[0086] Step S104 : Correcting the basic image pair using a preset correction mode to obtain a plurality of correction parameters.

[0087] As an optional embodiment of the present invention, a preset correction mode is used to perform correction processing on the basic image pair to obtain multiple correction parameters, including: determining the target translation parameters and target scaling coefficients between the first image and the second image; and determining multiple correction parameters based on the target translation parameters and the target scaling coefficients.

[0088] Optionally, the step of determining the target translation parameters and target scaling coefficients between the first image and the second image includes: extracting image features of the first image to obtain a first feature subset, wherein the first feature subset includes a first distance image, a first boundary direction map, and first mask information; extracting image features of the second image to obtain a second feature subset, wherein the second feature subset includes a second distance image and a second boundary direction map; and calculating the target translation parameters of the first image relative to the second image based on the first feature subset and the second feature subset.

[0089] Next, feature extraction is performed on the first image and the second image obtained after preprocessing.

[0090] First, the embodiment of the present invention extracts image features of a first image to obtain a first feature subset.

[0091] Optionally, the step of extracting image features of the first image to obtain a first feature subset includes: extracting all boundary pixel points of each target object in the first image to obtain a first edge image; performing inversion processing on the first edge image to obtain a second edge image; extracting contours from the first edge image to obtain a first contour array, and calculating the pixel direction corresponding to each pixel based on the first contour array to obtain a first contour direction array; performing a preset distance transformation processing on the second edge image based on a first preset distance threshold to obtain a first distance image; calculating a first boundary direction map corresponding to the boundaries of each target object in the second edge image based on the first contour direction array; and determining the first feature subset based on the first distance image and the first boundary direction map.

[0092] When extracting image features of the first image, extracting all boundary pixel points of each target object in the first image to obtain a first edge image refers to extracting the main edges in the first image. The edge refers to the boundary pixel point of at least one object in the image. The present invention does not specifically limit the algorithm for extracting the edge, for example, based on gradient difference detection, based on depth difference detection and based on deep learning edge detection methods.

[0093] A first contour array is obtained by extracting contours from the first edge image. The first contour array records contour delineation information and the positional relationship between the pixels included in each contour as a multidimensional array. Therefore, based on the first contour array, the pixel direction corresponding to each pixel can be calculated to obtain a first contour direction array. Calculating a first boundary direction map corresponding to each target object boundary in the second edge image based on the first contour direction array can include: recording the contour delineation information and the direction values ​​of each pixel included in the contour in the first contour direction array, mapping the information contained in the first contour direction array to the second edge image, calculating the direction corresponding to each target object boundary in the second edge image, and saving it as the first boundary direction map.

[0094] In an embodiment of the present invention, the step of performing a preset distance transformation on the second edge image based on a first preset distance threshold to obtain a first distance image includes: determining first mask information based on the first preset distance threshold, wherein the first mask information is used to shield part of the edge information in the second image; and adding the first mask information to the first feature subset.

[0095] Based on a first preset distance threshold, a Euclidean distance transform is performed on the second edge image to obtain a first distance image. This also generates a mask to remove redundant edge information from the second image / query image. The mask is used to filter and block regions in the second image or matching image from processing. By using the mask information generated by the distance transform of a simple contour image to process the second image with complex contours, redundant contours can be removed, reducing the computational complexity of image processing.

[0096] For example, extracting features of the first image / template image includes:

[0097] Step 1: Extract the main edges of the first image and record the obtained edge image as C1, where the grayscale value of the edge pixel position is 255 and the grayscale value of the non-edge pixel position is 0;

[0098] Step 2: Invert the edge image C1 to obtain the edge image E1, where the grayscale value of the edge pixel position is 0 and the grayscale value of the non-edge pixel position is 255;

[0099] Step 3: Extract the contour in C1, denoted as V1;

[0100] Step 4: Based on a preset distance threshold, perform a Euclidean distance transform on the edge image E1 to obtain a distance image DT1 and a mask to remove redundant edge information in the second image / query image.

[0101] Step 5: Calculate the orientation map OM1 of the edge in edge image E1. First, calculate the orientation of the pixel on the contour V1, denoted as O1. Based on the mapping of O1 to the edge image E1, calculate the orientation of the contour and save it as the orientation map OM1.

[0102] Then, the embodiment of the present invention extracts image features of the second image to obtain a second feature subset.

[0103] Optionally, the step of extracting image features of the second image to obtain a second feature subset includes: extracting all boundary pixel points of each target object in the second image to obtain a third edge image; using the first mask information to delete the contours in the third edge image; performing inversion processing on the deleted third edge image to obtain a fourth edge image; extracting contours from the fourth edge image to obtain a second contour array, and calculating the pixel direction corresponding to each pixel based on the second contour array to obtain a second contour direction array; performing a preset distance transformation on the fourth edge image based on a second preset distance threshold to obtain a second distance image; calculating a second boundary direction map corresponding to the boundaries of each target object in the fourth edge image based on the second contour direction array; and obtaining a second feature subset based on the second distance image and the second boundary direction map.

[0104] The primary edges of each object in the second image are extracted to obtain a third edge image. The edges in the third edge image are then deleted using mask information. The processed edge image is then inverted to obtain a fourth edge image, or primary edge image. Using mask information derived from a distance transform of a simple contour image to process the complex contours of the second image can remove redundant contours and reduce the computational complexity of image processing.

[0105] For example, extracting features of the second image / matching image includes:

[0106] Step 1: Extract the main edges of the image and record the obtained edge image as C2;

[0107] Step 2: Use the mask information obtained from the above calculation to delete the edges in C2, and then invert the processed edge image to obtain the main edge image E2 of the image, where the grayscale value of the edge pixel position is 0 and the grayscale value of the non-edge pixel position is 255;

[0108] Step 3: Extract the contour in image C2, denoted as V2;

[0109] Step 4: Perform a Euclidean distance transform on the edge image E2 to obtain a distance image DT2; set a distance threshold to generate DT2 that is actually used in the calculation.

[0110] Step5: Calculate the orientation map OM2 of the edges in the edge image E2. First, calculate the orientation of the pixel points on the contour V2, denoted as O2; map O2 to the edge image E2 to calculate the orientation of the contour, and save it as the orientation map OM2.

[0111] After completing the feature extraction operations on the first image and the second image, the target translation parameter and the target scaling factor of the image object can be calculated.

[0112] Optionally, the step of calculating the target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset includes: using the first judgment condition, extracting the contour pixel points in the first distance image and the second distance image whose pixel distances are less than the first distance threshold to obtain the first set of contour pixel points participating in image matching; using the second judgment condition, extracting the contour pixel points in the first boundary orientation map and the second boundary orientation map whose pixel distances are less than the second distance threshold to obtain the second set of contour pixel points participating in image matching; based on the first set of contour pixel points and the second set of contour pixel points, determining the chamfer distance score, the orientation map distance, and the image adjustment factor between the first image and the second image, where the image adjustment factor is used to adjust the proportion of the chamfer distance score and the orientation map distance; sliding the second image on the first image, and inputting the chamfer distance score, the orientation map distance, and the image adjustment factor into the first preset formula to calculate the image sliding score; determining the target sliding position corresponding to the minimum value among all the image sliding scores; based on the target sliding position, determining the target translation parameter.

[0113] For example, when calculating the target translation parameter, it includes:

[0114] Step1: Extract the positions of the contour pixel points participating in image matching, and the judgment conditions are as follows:

[0115] ||DT1(i, j) - DT2(i, j)|| < th1 and ||OM1(i, j) - OM2(i, j)|| < th2;

[0116] where th1 and th2 are two preset distance thresholds respectively, ||DT1(i, j) - DT2(i, j)|| is the pixel distance of a certain pixel point in the first distance image and the second distance image, and ||OM1(i, j) - OM2(i, j)|| is the pixel distance of a certain pixel point in the first boundary orientation map and the second boundary orientation map.

[0117] For pixel positions that do not meet the conditions, they will not participate in the calculation of the chamfer distance score, reducing the calculation amount.

[0118] Step2: Calculate the chamfer distance score. By sliding the first image on the second image, an image sliding score can be obtained for each sliding position, and the first preset formula for its calculation is as follows:

[0119] score = EDT + s * ODT; where score is the image sliding score at a certain sliding position, EDT is the chamfer distance between the first distance image and the second distance image; ODT is the distance between the first boundary direction map and the second boundary direction map, the direction map distance; s is an image adjustment factor that adjusts the ratio of EDT and ODT.

[0120] Step 3: Select the x and y translation values. Determine the target sliding position corresponding to the minimum sliding score among all images. The translation amount corresponding to this position is the target translation parameter required for alignment, i.e., the x and y translation values: dx and dy.

[0121] In an embodiment of the present invention, when correcting the basic image pair, the order of executing scaling and translation is not limited. The translation correction amount and the scaling correction amount can be calculated first, and then alignment can be achieved based on the correction amounts. Alternatively, the translation correction amount can be calculated first and the translation correction can be completed, and then the scaling correction can be achieved on this basis.

[0122] Several implementation methods for determining the target translation parameter and the target scaling factor between the first image and the second image are described below.

[0123] The first method is to determine the target translation parameters and target scaling coefficients between the first image and the second image, including: calculating the target translation parameters of the first image relative to the second image, and translating the first image based on the target translation parameters to obtain a third image; selecting multiple scaling coefficients, and scaling the third image with each scaling coefficient, and calculating the image matching score between the third image and the second image; and using the scaling coefficient corresponding to the smallest score among the multiple image matching scores as the target scaling coefficient.

[0124] By first determining the target translation parameters, the first image is translated using the target translation parameters to obtain a third image; then the third image is scaled, the scaling factor is adjusted, and the image matching score between the third image and the second image under the scaling factor is calculated. By selecting the scaling factor corresponding to the minimum image matching score as the target scaling factor, the third image is scaled using the target scaling factor to achieve correction processing between the two images.

[0125] The above-mentioned target scaling factor includes but is not limited to: scaling width factor, scaling length factor, scaling ratio factor, etc.

[0126] The second method is to determine the target translation parameters and target scaling coefficients between the first image and the second image, including: calculating the target translation parameters of the first image relative to the second image, and translating the first image based on the target translation parameters to obtain a fourth image; selecting multiple scaling coefficients, and scaling the fourth image with each scaling coefficient, and calculating the image matching score between the fourth image and the second image; based on the image matching score, adjusting the scaling coefficient until the change in the image matching score is less than a first threshold, and the scaling coefficient corresponding to the image matching score is used as the target scaling coefficient.

[0127] The method first determines a translation parameter and translates the first image using the target translation parameter to obtain a fourth image. The fourth image is then scaled, and the scaling factor is adjusted until the change in the image matching score between the fourth image and the second image at the scaling factor is less than a first threshold. The scaling factor corresponding to the image matching score is used as the target scaling factor. The first image is scaled using the target scaling factor to achieve correction between the two images.

[0128] The third method is to determine the target translation parameters and target scaling coefficients between the first image and the second image, including: selecting multiple scaling coefficients and scaling the first image with each scaling coefficient respectively; sliding the first image scaled based on each scaling coefficient on the second image, and calculating the image matching score between the first image and the second image; and using the scaling coefficient and translation amount corresponding to the smallest score among the multiple image matching scores as the target scaling coefficient and target translation parameters.

[0129] When selecting a zoom factor, the first image scaled based on each zoom factor can be slid on the second image to calculate the image matching score between it and the second image. The zoom factor and translation amount corresponding to the minimum score in the image matching score are used as the target zoom factor and target translation parameters to achieve correction processing between the two images.

[0130] Step S106 , performing alignment correction on the above-mentioned basic image pair based on each correction parameter to obtain a target image pair.

[0131] Specifically, according to the multiple correction parameters obtained in the above steps, including the target translation parameter and the target scaling factor, the above basic image pair is aligned and corrected to obtain a target image pair that meets the alignment requirements.

[0132] In an embodiment of the present invention, when a preset correction mode is used to perform correction processing on the basic image pair to obtain multiple correction parameters, the basic image pair can be scaled to a preset resolution and subjected to pyramid correction processing to obtain multiple correction parameters.

[0133] Although chamfer distance matching is based on image edge feature extraction, it still requires traversing edge pixel positions, resulting in a significant computational overhead. If the images to be aligned are high-resolution and have complex edge information, this can further impact the real-time nature of image alignment. Therefore, when performing alignment correction on high-resolution base images, a multi-resolution dynamic correction method is required, performing alignment from coarse to fine resolution.

[0134] In this embodiment of the present invention, during image correction, the image is first downsampled to a low resolution, and then upsampled layer by layer for dynamic correction. This approach, known as a pyramiding algorithm, reduces computation time, minimizing runtime at the lowest resolution. A rough preliminary result is obtained, and fine-tuning is performed based on the result at the lowest resolution, eliminating the need for a complete recalculation. If the accuracy requirement is low, correction processing at a low resolution can meet the alignment accuracy requirement. If the accuracy requirement is high and correction processing at a low resolution cannot meet the accuracy requirement, upsampling to a high resolution is performed for correction until the alignment accuracy requirement is met.

[0135] As an optional embodiment of the present invention, after the above-mentioned basic image pair is aligned and corrected based on each correction parameter to obtain the target image pair, the image correction method also includes: determining the image alignment requirement accuracy of the terminal application at the target image resolution; step S1, judging whether the current alignment accuracy image corresponding to the target image pair meets the image alignment requirement accuracy at the first image resolution; step S2, if it is determined that the current alignment accuracy image corresponding to the target image pair does not meet the image alignment requirement accuracy, adjusting the image resolution to the second image resolution, wherein the resolution value of the second image resolution is higher than the first image resolution; step S3, executing the step of correcting the basic image pair using a preset correction mode to obtain multiple correction parameters; step S4, executing the step of aligning and correcting the above-mentioned basic image pair based on each correction parameter to obtain the target image pair; repeating steps S1 to S4 until the current alignment accuracy image meets the image alignment requirement accuracy.

[0136] For example, a visible light image and a depth map are obtained from a visible light camera and a depth camera, respectively. The depth map is transformed into the visible light image coordinate system according to preset calibration parameters, and then adjusted to obtain a preliminary aligned depth map with the same resolution as the visible light image. The preliminary aligned image pair P1 is scaled to a low resolution p and dynamically corrected. After obtaining the correction parameters (dx_1, dy_1, scale_1), the input image pair is corrected to obtain a new image pair, denoted as aligned image pair P2. The corrected aligned image pair is dynamically corrected at a resolution p*s2 (s is a magnification factor, generally 2). After obtaining the correction parameters (dx_2, dy_2, scale_2), the input image pair P2 is corrected to obtain a new image pair, denoted as aligned image pair P3. The resolution used in the correction process is continuously increased, and the dynamic correction process is repeated until the alignment accuracy required by the application is met.

[0137] Through the above implementation, the accuracy of the image correction process is improved. In the embodiment of the present invention, the error can be corrected from 30 pixels to within 4 pixels at VGA resolution in various scenarios, and the alignment accuracy is very high.

[0138] Optionally, the image correction method also includes: comparing the image resolution of the visible light image and the image resolution of the depth image to obtain a comparison result with the minimum resolution; calculating a correction number threshold based on the image resolution obtained from the comparison result and the maximum resolution of the correction processing initially set; during the alignment correction process, if the number of image corrections reaches the correction number threshold, stopping the correction processing.

[0139] The resolution of the initial alignment and the number of dynamic corrections required can be determined based on the resolution of the input image. For example, the resolution of the image with the smaller resolution between the two images (depth image and visible light image) is Tw*Th. Usually, the resolution of the visible light image is much larger than the resolution of the depth image. Set the maximum resolution of the initial alignment to Mw*Mh (usually Mw is 320), and then get the calculation formula for the number of dynamic corrections t as:

[0140] Where Sa = Tw / Mw,

[0141] in the formula =tw / s; s represents the single magnification, usually 2. The initial value m0 = Tw / s can be obtained from the dynamic correction times t. t-1 ; The resolution pyramid of the entire alignment process can be expressed by the following formula:

[0142] m n =Tw / s t-1-n , n=0,1,2...t-1;

[0143] n n =Th / s t-1-n , n=0,1,2...t-1。

[0144] If the result at a certain low resolution (upsampled t' times) already meets the accuracy requirement, it is possible to choose not to continue upsampling. t is the maximum number of dynamic corrections. The optimization process can be terminated early according to actual needs. The number of sampling times t'<=t.

[0145] Through the above embodiment, the correction parameters (or alignment parameters) can be calculated according to the visible light image and the depth map (or the visible light image and the infrared image), which is not only applicable to the image that can only provide the visible light image and the infrared image, or can only provide the visible light image and the depth image, or can provide the visible light image, the infrared image and the depth image. Figure 3 The present invention is applicable to devices with depth cameras; it can also be applied to situations where the images captured by two cameras have similar contents but large texture differences and where matching based on feature points is impossible; at the same time, the embodiments of the present invention can also improve the alignment errors of binocular devices that are caused by problems such as OIS, falling, frame asynchrony, and different frame rates, and the correction environment is very simple. Without the need for a specific environment or a specific shooting pattern, the image correction process can be completed quickly to obtain images that satisfy the user.

[0146] The present invention is described below in conjunction with another optional embodiment.

[0147] Example 2

[0148] An embodiment of the present invention provides an image correction device, which includes multiple implementation units corresponding to the implementation steps in the above-mentioned embodiment 1.

[0149] Figure 4 is a schematic diagram of an optional image correction device according to an embodiment of the present invention. Figure 4 As shown, the image correction device may include: an acquisition unit 41, a first correction unit 43, and a second correction unit 45, wherein:

[0150] An acquisition unit 41 is configured to acquire a visible light image and a depth image taken of a target object, and form a basic image pair after transformation, wherein the basic image pair includes a first image and a second image;

[0151] A first correction unit 43 is configured to perform correction processing on the basic image pair using a preset correction mode to obtain a plurality of correction parameters;

[0152] The second correction unit 45 is configured to perform alignment correction on the base image pair based on each correction parameter to obtain a target image pair.

[0153] The image correction device can acquire a visible light image and a depth image of a target object through an acquisition unit 41, transforming them into a base image pair, wherein the base image pair includes a first image and a second image. A first correction unit 45 corrects the base image pair using a preset correction mode to obtain multiple correction parameters. A second correction unit 47 then performs alignment correction on the base image pair based on each correction parameter to obtain a target image pair. In this embodiment, alignment can be performed on images captured by multiple cameras, achieving dynamic correction. The correction environment is simple, and alignment correction can be completed using images captured by the device. This solves the technical issues in related technologies such as the inability to achieve dynamic correction between two different cameras, low adaptability to the environment, poor image alignment, and a potential loss of user interest.

[0154] Optionally, the first correction unit includes: a first correction module, configured to scale the basic image pair to a preset resolution, and perform pyramid correction processing to obtain a plurality of correction parameters.

[0155] Optionally, the acquisition unit includes: a first transformation module, used to transform the depth image into the image coordinate system of the visible light image based on preset calibration parameters, and adjust it to obtain a preliminary aligned depth map with the same resolution as the visible light image, wherein the visible light image and the preliminary aligned depth map are combined to form a basic image pair, the first image is the visible light image, and the second image is the preliminary aligned depth map.

[0156] Optionally, the first correction unit further includes: a first determination module for determining a target translation parameter and a target scaling factor between the first image and the second image; and a second determination module for determining a plurality of correction parameters based on the target translation parameter and the target scaling factor.

[0157] Optionally, the image correction device also includes: a first processing unit, used to pre-process the preliminary aligned depth map in the basic image pair to obtain a first image before correcting the basic image pair using a preset correction mode to obtain multiple correction parameters; and a second processing unit, used to filter the visible light image in the basic image pair to obtain a second image.

[0158] Optionally, the first determination module includes: a first calculation module, used to calculate the target translation parameters of the first image relative to the second image, and translate the first image based on the target translation parameters to obtain a third image; a first scaling module, used to select multiple scaling factors, and scale the third image with each scaling factor, and calculate the image matching score between the third image and the second image; a second determination module, used to use the scaling factor corresponding to the smallest score among the multiple image matching scores as the target scaling factor.

[0159] Optionally, the first determination module also includes: a second calculation module, used to calculate the target translation parameters of the first image relative to the second image, and translate the first image based on the target translation parameters to obtain a fourth image; a second scaling module, used to select multiple scaling coefficients, and scale the fourth image with each scaling coefficient, and calculate the image matching score between the fourth image and the second image; a third determination module, used to adjust the above scaling coefficient until the score change in the above image matching score is less than the first threshold, and the scaling coefficient corresponding to the above image matching score is used as the target scaling coefficient.

[0160] Optionally, the first determination module also includes: a third scaling module, used to select multiple scaling factors and scale the first image with each scaling factor respectively; a third calculation module, used to slide the first image scaled based on each scaling factor on the second image, and calculate the image matching score between it and the second image; a fourth determination module, used to use the scaling factor and translation amount corresponding to the minimum score among the multiple image matching scores as the target scaling factor and target translation parameters.

[0161] Optionally, the first processing unit includes: a first mapping module, used to map the depth value of each pixel point in the preliminary aligned depth map in the base image pair to a preset pixel range; and / or, a first adjustment module, used to adjust the image contrast of the preliminary aligned depth map to obtain a first image.

[0162] Optionally, the first determination module also includes: a first extraction module, used to extract image features of the first image to obtain a first feature subset, wherein the above-mentioned first feature subset includes a first distance image, a first boundary direction map and first mask information; a second extraction module, used to extract image features of the second image to obtain a second feature subset, wherein the above-mentioned second feature subset includes a second distance image and a second boundary direction map; a fourth calculation module, used to calculate the target translation parameters of the first image relative to the second image based on the above-mentioned first feature subset and the above-mentioned second feature subset.

[0163] Optionally, the first extraction module includes: a first extraction submodule, used to extract all boundary pixel points of each target object in the first image to obtain a first edge image; a first inversion submodule, used to perform inversion processing on the first edge image to obtain a second edge image; a second extraction submodule, used to extract the contour of the first edge image to obtain a first contour array, and calculate the pixel direction corresponding to each pixel point based on the first contour array to obtain a first contour direction array; a first transformation submodule, used to perform a preset distance transformation processing on the second edge image based on a first preset distance threshold to obtain a first distance image; a first calculation submodule, used to calculate the first boundary direction map corresponding to the boundaries of each target object in the second edge image based on the first contour direction array; and a first determination submodule, used to determine the first feature subset based on the first distance image and the first boundary direction map.

[0164] Optionally, the first transformation submodule includes: a second determination submodule, used to determine first mask information based on a first preset distance threshold, wherein the first mask information is used to shield part of the edge information in the second image; and an addition submodule, used to add the first mask information to the first feature subset.

[0165] Optionally, the second extraction module includes: a second extraction submodule, used to extract all boundary pixel points of each target object in the second image to obtain a third edge image; a deletion submodule, used to use the first mask information to delete the contours in the third edge image; a second inversion submodule, used to invert the third edge image after deletion to obtain a fourth edge image; a second calculation submodule, used to extract the contours of the fourth edge image to obtain a second contour array, and calculate the pixel direction corresponding to each pixel point based on the second contour array to obtain a second contour direction array; a second transformation submodule, used to perform a preset distance transformation on the fourth edge image based on a second preset distance threshold to obtain a second distance image; a third calculation submodule, used to calculate the second boundary direction map corresponding to the boundaries of each target object in the fourth edge image based on the second contour direction array; and a third determination submodule, used to obtain a second feature subset based on the second distance image and the second boundary direction map.

[0166] Optionally, the fourth calculation module includes: a third extraction submodule, used to use the first judgment condition to extract contour pixel points whose pixel distance between the first distance image and the second distance image is less than the first distance threshold, to obtain a first contour pixel point set participating in image matching; a fourth extraction submodule, used to use the second judgment condition to extract contour pixel points whose pixel distance between the first boundary direction map and the second boundary direction map is less than the second distance threshold, to obtain a second contour pixel point set participating in image matching; a fifth determination submodule, used to determine the chamfer distance score, direction map distance and image adjustment factor between the first image and the second image based on the first contour pixel point set and the second contour pixel point set, wherein the image adjustment factor is used to adjust the chamfer distance score and the direction map distance ratio; a fourth calculation submodule, used to slide the second image on the first image, and input the chamfer distance score, direction map distance and image adjustment factor into the first preset formula to calculate the image sliding score; a sixth determination submodule, used to determine the target sliding position corresponding to the minimum score among all image sliding scores; and a seventh determination submodule, used to determine the target translation parameter based on the target sliding position.

[0167] Optionally, the first correction module includes: a first acquisition submodule, used to obtain the alignment accuracy value of the terminal application, and determine multiple correction resolutions based on the alignment accuracy value and the resolution of the basic image pair, wherein the multiple correction resolutions include at least: a preset resolution, which is the minimum resolution among the multiple correction resolutions; a first correction submodule, used to scale the basic image pair to the preset resolution, and perform pyramid correction processing until the alignment accuracy value is satisfied, thereby obtaining multiple correction parameters.

[0168] Optionally, the image correction device also includes: a determination unit for determining the image alignment requirement accuracy of the terminal application at the target image resolution; a first judgment unit for executing step S1 to determine whether the current alignment accuracy image corresponding to the target image pair meets the image alignment requirement accuracy at the first image resolution; a first adjustment unit for executing step S2, if it is determined that the current alignment accuracy image corresponding to the target image pair does not meet the image alignment requirement accuracy, adjusting the image resolution to a second image resolution, wherein the resolution value of the second image resolution is higher than the first image resolution; a first execution unit for executing step S3, executing the step of correcting the basic image pair using a preset correction mode to obtain multiple correction parameters; a second execution unit for executing step S4, executing the step of aligning and correcting the above-mentioned basic image pair based on each correction parameter to obtain the target image pair; repeating steps S1 to S4 until the current alignment accuracy image meets the image alignment requirement accuracy.

[0169] Optionally, the image correction device also includes: a comparison unit, used to compare the image resolution of the visible light image and the image resolution of the depth image to obtain a comparison result with the minimum resolution; a calculation unit, used to calculate a correction number threshold based on the image resolution obtained from the comparison result and the initial set maximum resolution of the correction processing; and a stop unit, used to stop the correction processing if the number of image corrections reaches the correction number threshold during the alignment correction process.

[0170] The above-mentioned image correction device may further include a processor and a memory. The above-mentioned acquisition unit 41, first correction unit 43, second correction unit 45, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0171] The processor includes a kernel that retrieves corresponding program units from a memory. One or more kernels may be provided, and kernel parameters are adjusted to align and correct the base image pair based on each correction parameter to obtain a target image pair.

[0172] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0173] According to another aspect of an embodiment of the present invention, an image correction system is also provided, including: a first image capture device for capturing a visible light image of a target object; a second image capture device for capturing a depth image of the target object; a correction device for transforming the visible light image and the depth image captured of the target object to form a basic image pair, wherein the basic image pair includes a first image and a second image, a preset correction mode is used to correct the basic image pair to obtain multiple correction parameters, and the basic image pair is aligned and corrected based on each correction parameter to obtain a target image pair; and a result output device is used to output the aligned target image pair to a preset terminal display interface.

[0174] According to another aspect of an embodiment of the present invention, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned image correction methods by executing the executable instructions.

[0175] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned image correction methods.

[0176] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program initialized with the following method steps: obtaining a visible light image and a depth image taken of a target object, and forming a basic image pair after transformation, wherein the basic image pair includes a first image and a second image; correcting the basic image pair using a preset correction mode to obtain multiple correction parameters; and aligning and correcting the basic image pair based on each correction parameter to obtain a target image pair.

[0177] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0178] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another coefficient, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0180] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0181] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0182] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0183] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An image correction method, characterized in that: include: Obtaining a visible light image and a depth image taken of a target object, and transforming them to form a basic image pair, wherein the basic image pair includes a first image and a second image; Correcting the base image pair using a preset correction mode to obtain a plurality of correction parameters includes: determining a target translation parameter and a target scaling factor between the first image and the second image; and determining a plurality of correction parameters based on the target translation parameter and the target scaling factor; The step of determining a target translation parameter and a target scaling factor between the first image and the second image includes: extracting image features of the first image to obtain a first feature subset, wherein the first feature subset includes a first range image, a first boundary direction map, and first mask information; extracting image features of the second image to obtain a second feature subset, wherein the second feature subset includes a second range image and a second boundary direction map; and calculating a target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset; The step of extracting image features of the first image to obtain a first feature subset includes: extracting all boundary pixel points of each target object in the first image to obtain a first edge image; performing color inversion processing on the first edge image to obtain a second edge image; extracting contours from the first edge image to obtain a first contour array, and calculating the pixel direction corresponding to each pixel based on the first contour array to obtain a first contour direction array; performing a preset distance transformation processing on the second edge image based on a first preset distance threshold to obtain the first distance image; calculating the first boundary direction map corresponding to the boundaries of each target object in the second edge image based on the first contour direction array; and determining the first feature subset based on the first distance image and the first boundary direction map; wherein the first mask information is used to shield part of the edge information in the second image; The base image pair is aligned and corrected based on each correction parameter to obtain a target image pair.

2. The image correction method according to claim 1, wherein: The step of correcting the basic image pair using a preset correction mode to obtain a plurality of correction parameters includes: The basic image pair is scaled to a preset resolution, and pyramid correction processing is performed to obtain the multiple correction parameters.

3. The image correction method according to claim 1, wherein: The steps of obtaining a visible light image and a depth image taken of a target object and transforming them into a basic image pair include: Based on preset calibration parameters, the depth image is transformed into the image coordinate system of the visible light image, and is adjusted to obtain a preliminary aligned depth map with the same resolution as the visible light image, wherein the visible light image and the preliminary aligned depth map are combined to form the basic image pair, the first image is the visible light image, and the second image is the preliminary aligned depth map.

4. The image correction method according to claim 1, wherein: Before correcting the basic image pair using a preset correction mode to obtain a plurality of correction parameters, the image correction method further includes: Preprocessing the preliminary aligned depth map in the base image pair to obtain the first image; Perform filtering processing on the visible light image in the basic image pair to obtain the second image.

5. The image correction method according to claim 1, wherein: The step of determining a target translation parameter and a target scaling factor between the first image and the second image comprises: calculating the target translation parameter of the first image relative to the second image, and translating the first image based on the target translation parameter to obtain a third image; selecting a plurality of scaling factors, scaling the third image by each scaling factor, and calculating an image matching score between the third image and the second image; The scaling factor corresponding to the smallest score among the plurality of image matching scores is used as the target scaling factor.

6. The image correction method according to claim 1, wherein: The step of determining a target translation parameter and a target scaling factor between the first image and the second image comprises: calculating the target translation parameter of the first image relative to the second image, and translating the first image based on the target translation parameter to obtain a fourth image; selecting a plurality of scaling factors, scaling the fourth image by each scaling factor, and calculating an image matching score between the fourth image and the second image; The scaling factor is adjusted until a score change in the image matching score is less than a first threshold, and the scaling factor corresponding to the image matching score is used as a target scaling factor.

7. The image correction method according to claim 1, wherein: The step of determining a target translation parameter and a target scaling factor between the first image and the second image comprises: Selecting a plurality of scaling factors, and scaling the first image by each scaling factor; Sliding the first image scaled based on each of the scaling factors on the second image, and calculating an image matching score between the first image and the second image; The zoom coefficient and translation amount corresponding to the minimum score among the plurality of image matching scores are used as the target zoom coefficient and the target translation parameter.

8. The image correction method according to claim 4, characterized in that: The step of preprocessing the preliminary aligned depth map in the base image pair to obtain the first image comprises: Mapping the depth value of each pixel in the preliminary aligned depth map of the base image pair to a preset pixel range; and / or, The image contrast of the preliminary aligned depth map is adjusted to obtain the first image.

9. The image correction method according to claim 1, wherein: The step of performing a preset distance transformation process on the second edge image based on a first preset distance threshold to obtain the first distance image includes: determining the first mask information based on the first preset distance threshold; The first mask information is added to the first feature subset.

10. The image correction method according to claim 9, characterized in that: The step of extracting image features of the second image to obtain a second feature subset includes: Extracting all boundary pixel points of each target object in the second image to obtain a third edge image; Using the first mask information to perform a deleting process on the contour in the third edge image; performing color inversion processing on the third edge image after the deletion processing to obtain a fourth edge image; Extracting contours from the fourth edge image to obtain a second contour array, and calculating a pixel direction corresponding to each pixel based on the second contour array to obtain a second contour direction array; Based on a second preset distance threshold, performing a preset distance transformation process on the fourth edge image to obtain the second distance image; Calculating the second boundary direction map corresponding to each target object boundary in the fourth edge image based on the second contour direction array; The second feature subset is obtained based on the second range image and the second boundary direction map.

11. The image correction method according to claim 1, wherein: The step of calculating a target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset comprises: Using a first judgment condition, extracting contour pixel points whose pixel distance between the first range image and the second range image is less than a first distance threshold, to obtain a first contour pixel point set participating in image matching; Using a second judgment condition, extracting contour pixel points whose pixel distance between the first boundary direction map and the second boundary direction map is less than a second distance threshold, to obtain a second contour pixel point set participating in image matching; Determining a chamfer distance score, a directional pattern distance, and an image adjustment factor between the first image and the second image based on the first contour pixel point set and the second contour pixel point set, wherein the image adjustment factor is used to adjust the chamfer distance score and the directional pattern distance ratio; Sliding the second image on the first image, and inputting the chamfer distance score, the direction pattern distance, and the image adjustment factor into a first preset formula to calculate an image sliding score; Determine the target sliding position corresponding to the minimum sliding score among all images; Based on the target sliding position, a target translation parameter is determined.

12. The image correction method according to claim 2, wherein: The step of scaling the basic image pair to a preset resolution and performing pyramid correction processing to obtain the multiple correction parameters includes: Obtaining an alignment accuracy value of a terminal application, and determining a plurality of correction resolutions based on the alignment accuracy value and a resolution of the base image pair, wherein the plurality of correction resolutions at least include: a preset resolution, the preset resolution being a minimum resolution among the plurality of correction resolutions; The basic image pair is scaled to the preset resolution, and pyramid correction processing is performed until the alignment accuracy value is satisfied, thereby obtaining the plurality of correction parameters.

13. The image correction method according to claim 12, wherein: The image correction method further includes: Determine the image alignment accuracy required by the terminal application at the target image resolution; Step S1, determining whether the current alignment accuracy image corresponding to the target image pair meets the image alignment requirement accuracy at the first image resolution; Step S2: if it is determined that the current alignment accuracy image corresponding to the target image pair does not meet the image alignment requirement accuracy, adjusting the image resolution to a second image resolution, wherein the resolution value of the second image resolution is higher than the first image resolution; Step S3, performing correction processing on the basic image pair using a preset correction mode to obtain a plurality of correction parameters; Step S4, performing alignment correction on the base image pair based on each correction parameter to obtain a target image pair; Repeat steps S1 to S4 until the current alignment accuracy image reaches the image alignment requirement accuracy.

14. The image correction method according to claim 1, wherein: The image correction method further includes: comparing the image resolution of the visible light image and the image resolution of the depth image to obtain a comparison result with the minimum resolution; Calculating a correction number threshold based on the image resolution obtained from the comparison result and the initially set maximum resolution for correction processing; During the alignment correction process, if the number of image corrections reaches the correction number threshold, the correction process is stopped.

15. An image correction device, characterized in that: include: an acquisition unit, configured to acquire a visible light image and a depth image taken of a target object, and form a basic image pair after transformation, wherein the basic image pair includes a first image and a second image; A first correction unit, configured to perform correction processing on the basic image pair using a preset correction mode to obtain a plurality of correction parameters; The first correction unit further includes: a first determination module for determining a target translation parameter and a target scaling factor between the first image and the second image; a second determination module for determining a plurality of correction parameters based on the target translation parameter and the target scaling factor; The first determination module further includes: a first extraction module for extracting image features of the first image to obtain a first feature subset, wherein the first feature subset includes a first range image, a first boundary direction map, and first mask information; a second extraction module for extracting image features of the second image to obtain a second feature subset, wherein the second feature subset includes a second range image and a second boundary direction map; and a fourth calculation module for calculating a target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset; The first extraction module includes: a first extraction submodule for extracting all boundary pixels of each target object in the first image to obtain a first edge image; a first inversion submodule for performing inversion processing on the first edge image to obtain a second edge image; a second extraction submodule for extracting contours from the first edge image to obtain a first contour array, and calculating the pixel direction corresponding to each pixel based on the first contour array to obtain a first contour direction array; a first transformation submodule for performing a preset distance transformation processing on the second edge image based on a first preset distance threshold to obtain the first distance image; a first calculation submodule for calculating the first boundary direction map corresponding to the boundaries of each target object in the second edge image based on the first contour direction array; a first determination submodule for determining the first feature subset based on the first distance image and the first boundary direction map; wherein the first mask information is used to shield part of the edge information in the second image; The second correction unit is configured to perform alignment correction on the base image pair based on each correction parameter to obtain a target image pair.

16. An image correction system, characterized in that: include: a first image capturing device for capturing a visible light image of a target object; a second image capturing device for capturing a depth image of the target object; A correction device is configured to acquire a visible light image and a depth image taken of a target object, and form a base image pair after modification, wherein the base image pair includes a first image and a second image; perform correction processing on the base image pair using a preset correction mode to obtain a plurality of correction parameters; and perform alignment correction on the base image pair based on each of the correction parameters to obtain a target image pair; Correcting the base image pair using a preset correction mode to obtain a plurality of correction parameters includes: determining a target translation parameter and a target scaling factor between the first image and the second image; and determining a plurality of correction parameters based on the target translation parameter and the target scaling factor; The step of determining a target translation parameter and a target scaling factor between the first image and the second image includes: extracting image features of the first image to obtain a first feature subset, wherein the first feature subset includes a first range image, a first boundary direction map, and first mask information; extracting image features of the second image to obtain a second feature subset, wherein the second feature subset includes a second range image and a second boundary direction map; and calculating a target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset; The step of extracting image features of the first image to obtain a first feature subset includes: extracting all boundary pixel points of each target object in the first image to obtain a first edge image; performing color inversion processing on the first edge image to obtain a second edge image; extracting contours from the first edge image to obtain a first contour array, and calculating the pixel direction corresponding to each pixel based on the first contour array to obtain a first contour direction array; performing a preset distance transformation processing on the second edge image based on a first preset distance threshold to obtain the first distance image; calculating the first boundary direction map corresponding to the boundaries of each target object in the second edge image based on the first contour direction array; and determining the first feature subset based on the first distance image and the first boundary direction map; wherein the first mask information is used to shield part of the edge information in the second image; The result output device is used to output the aligned target image pair to a preset terminal display interface.

17. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the image correction method according to any one of claims 1 to 14 by executing the executable instructions.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the image correction method according to any one of claims 1 to 14.

Images