Image stitching method, device, computer readable storage medium and terminal device

Abstract

The present application belongs to the field of image processing technology, and in particular, relates to an image stitching method, device, computer-readable storage medium and terminal device. The method comprises: obtaining a first image to be stitched, a second image to be stitched and a distance to be stitched; wherein the first image to be stitched is an image captured by a preset first eyepiece lens, and the second image to be stitched is an image captured by a preset second eyepiece lens; mapping the second image to be stitched according to a preset first homography matrix to obtain a mapped image; wherein the first homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset first reference stitching distance; determining the overlap area between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and the distance to be stitched; fusing the first image to be stitched, the mapped image and the overlap area to obtain a stitched image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image stitching method, device, computer-readable storage medium and terminal device. Background Art

[0002] In the field of surveillance, a wider field of view and better picture clarity are often contradictory. This is because with a wider field of view, the total number of local pixels is often smaller, while when there are more local pixels, the field of view of the picture is usually smaller. In practice, in order to obtain a wider field of view and better picture clarity at the same time, image stitching technology is required.

[0003] Image stitching technology usually uses multiple cameras to collect images, and then stitches the images collected by each camera through an image stitching algorithm to finally obtain an image with a large field of view and high definition. In existing image stitching methods, many hardware parameters such as focal length, image width, camera spacing and sensor width are often required, and these hardware parameters may have certain errors with the actual ones, thus affecting the accuracy of the image stitching results. Summary of the invention

[0004] In view of this, embodiments of the present application provide an image stitching method, apparatus, computer-readable storage medium, and terminal device to solve the problem of poor accuracy of existing image stitching methods due to hardware parameter errors.

[0005] A first aspect of an embodiment of the present application provides an image stitching method, which may include:

[0006] Acquire a first image to be stitched, a second image to be stitched, and a distance to be stitched; wherein the first image to be stitched is an image captured by a preset first eyepiece lens, and the second image to be stitched is an image captured by a preset second eyepiece lens;

[0007] Mapping the second image to be stitched according to a preset first homography matrix to obtain a mapped image; wherein the first homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset first reference stitching distance;

[0008] Determining an overlapping area between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and the distance to be stitched;

[0009] The first image to be stitched, the mapped image and the overlapped area are fused to obtain a stitched image.

[0010] In a specific implementation of the first aspect, before determining an overlapping area between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and the distance to be stitched, the method may further include:

[0011] Calculating a first stitching pixel offset according to the first homography matrix and a preset second homography matrix; wherein the second homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset second reference stitching distance; the first reference stitching distance is not equal to the second reference stitching distance;

[0012] The stitching pixel offset relationship is determined according to the first stitching pixel offset and a preset binocular disparity formula.

[0013] In a specific implementation manner of the first aspect, calculating the first stitching pixel offset according to the first homography matrix and a preset second homography matrix may include:

[0014] Mapping a preset target pixel position according to the first homography matrix to obtain a first mapped pixel position;

[0015] Mapping the target pixel position according to the second homography matrix to obtain a second mapped pixel position;

[0016] The first stitching pixel offset is calculated according to the first mapped pixel position and the second mapped pixel position.

[0017] In a specific implementation of the first aspect, determining the overlapping area between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and the distance to be stitched may include:

[0018] Calculate the offset of the pixel to be stitched according to the stitching pixel offset relationship and the distance to be stitched;

[0019] Determining a pixel correspondence relationship between the first image to be stitched and the mapped image according to the pixel offset to be stitched;

[0020] An overlapping area between the first image to be stitched and the mapped image is determined according to the pixel correspondence relationship.

[0021] In a specific implementation of the first aspect, before fusing the first image to be stitched, the mapped image, and the overlapped area, the method may further include:

[0022] Calculate a first average pixel value and a second average pixel value; wherein the first average pixel value is an average pixel value of the first image to be stitched in the overlapping area, and the second average pixel value is an average pixel value of the mapped image in the overlapping area;

[0023] Calculating an absolute value of a difference between the first average pixel value and the second average pixel value;

[0024] If the absolute value is greater than a preset threshold, color gain is performed on the first image to be stitched or the mapped image.

[0025] In a specific implementation manner of the first aspect, performing color gain on the first image to be stitched or the mapped image may include:

[0026] Calculating a first absolute value and a second absolute value; wherein the first absolute value is the absolute value of the difference between the first average pixel value and a preset middle pixel value; and the second absolute value is the absolute value of the difference between the second average pixel value and the middle pixel value;

[0027] If the first absolute value is less than the second absolute value, taking the ratio of the first average pixel value to the second average pixel value as a first gain coefficient, and performing color gain on the mapped image according to the first gain coefficient;

[0028] If the first absolute value is greater than or equal to the second absolute value, a ratio of the second average pixel value to the first average pixel value is used as a second gain coefficient, and color gain is performed on the first image to be stitched according to the second gain coefficient.

[0029] In a specific implementation of the first aspect, the fusing the first image to be stitched, the mapping image, and the overlapped area to obtain the stitched image may include:

[0030] Determine an image in the first images to be stitched that does not belong to the overlapped area as a first boundary image, and determine an image in the first images to be stitched that belongs to the overlapped area as a first overlapped image;

[0031] Determine an image in the mapped image that does not belong to the overlapped area as a second boundary image, and determine an image in the mapped image that belongs to the overlapped area as a second overlapped image;

[0032] Performing weighted processing on the first overlapped image and the second overlapped image to obtain an overlapped region fused image;

[0033] The first boundary image, the overlapped area fused image and the second boundary image are spliced ​​to obtain the spliced ​​image.

[0034] A second aspect of the embodiments of the present application provides an image stitching device, which may include:

[0035] An image acquisition module, used to acquire a first image to be stitched, a second image to be stitched, and a distance to be stitched; wherein the first image to be stitched is an image captured by a preset first eyepiece lens, and the second image to be stitched is an image captured by a preset second eyepiece lens;

[0036] An image mapping module, used for mapping the second image to be stitched according to a preset first homography matrix to obtain a mapping image; wherein the first homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset first reference stitching distance;

[0037] An overlap region determining module, configured to determine an overlap region between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and the distance to be stitched;

[0038] The fusion and stitching module is used to fuse the first image to be stitched, the mapping image and the overlapped area to obtain a stitched image.

[0039] In a specific implementation of the second aspect, the image stitching device may further include:

[0040] A first stitching pixel offset calculation module, used to calculate a first stitching pixel offset according to the first homography matrix and a preset second homography matrix; wherein the second homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset second reference stitching distance; the first reference stitching distance is not equal to the second reference stitching distance;

[0041] The splicing pixel offset relationship determining module is used to determine the splicing pixel offset relationship according to the first splicing pixel offset and a preset binocular parallax formula.

[0042] In a specific implementation of the second aspect, the first splicing pixel offset calculation module may include:

[0043] A first mapping unit, configured to map a preset target pixel position according to the first homography matrix to obtain a first mapped pixel position;

[0044] A second mapping unit, configured to map the target pixel position according to the second homography matrix to obtain a second mapped pixel position;

[0045] The first splicing pixel offset calculation unit is used to calculate the first splicing pixel offset according to the first mapping pixel position and the second mapping pixel position.

[0046] In a specific implementation of the second aspect, the overlap area determination module may include:

[0047] A pixel offset calculation unit to be spliced, used for calculating the pixel offset to be spliced ​​according to the splicing pixel offset relationship and the distance to be spliced;

[0048] a pixel correspondence determination unit, configured to determine a pixel correspondence between the first image to be spliced ​​and the mapped image according to the pixel offset to be spliced;

[0049] An overlap region determining unit is used to determine an overlap region between the first image to be stitched and the mapped image according to the pixel correspondence relationship.

[0050] In a specific implementation of the second aspect, the image stitching device may further include:

[0051] A first calculation module, configured to calculate a first average pixel value and a second average pixel value; wherein the first average pixel value is an average pixel value of the first image to be stitched in the overlapped area, and the second average pixel value is an average pixel value of the mapped image in the overlapped area;

[0052] A second calculation module, configured to calculate an absolute value of a difference between the first average pixel value and the second average pixel value;

[0053] A color gain module is used to perform color gain on the first image to be stitched or the mapped image if the absolute value is greater than a preset threshold.

[0054] In a specific implementation of the second aspect, the color gain module may include:

[0055] an absolute value calculation unit, configured to calculate a first absolute value and a second absolute value; wherein the first absolute value is an absolute value of a difference between the first average pixel value and a preset intermediate pixel value; and the second absolute value is an absolute value of a difference between the second average pixel value and the intermediate pixel value;

[0056] a first color gain unit, configured to use a ratio of the first average pixel value to the second average pixel value as a first gain coefficient, and perform color gain on the mapped image according to the first gain coefficient if the first absolute value is smaller than the second absolute value;

[0057] The second color gain unit is configured to use a ratio of the second average pixel value to the first average pixel value as a second gain coefficient if the first absolute value is greater than or equal to the second absolute value, and perform color gain on the first image to be stitched according to the second gain coefficient.

[0058] In a specific implementation of the second aspect, the fusion and splicing module may include:

[0059] A first determining unit, configured to determine an image in the first images to be stitched that does not belong to the overlapped area as a first boundary image, and to determine an image in the first images to be stitched that belongs to the overlapped area as a first overlapped image;

[0060] A second determining unit, configured to determine an image in the mapped image that does not belong to the overlapped area as a second boundary image, and to determine an image in the mapped image that belongs to the overlapped area as a second overlapped image;

[0061] A weighted processing unit, used for performing weighted processing on the first coincident image and the second coincident image to obtain a fused image of the coincident region;

[0062] The stitching processing unit is used to stitch the first boundary image, the overlapped area fused image and the second boundary image to obtain the stitched image.

[0063] A third aspect of the embodiments of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned image stitching methods are implemented.

[0064] A fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned image stitching methods when executing the computer program.

[0065] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of any one of the above-mentioned image stitching methods.

[0066] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a first image to be stitched, a second image to be stitched, and a distance to be stitched; wherein, the first image to be stitched is an image captured by a preset first eyepiece lens, and the second image to be stitched is an image captured by a preset second eyepiece lens; the second image to be stitched is mapped according to a preset first homography matrix to obtain a mapping image; wherein, the first homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset first reference stitching distance; the overlapping area between the first image to be stitched and the mapping image is determined according to a preset stitching pixel offset relationship and the distance to be stitched; the first image to be stitched, the mapping image, and the overlapping area are fused to obtain a stitched image. Through the embodiments of the present application, there is no need to use a large number of hardware parameters such as focal length, image width, camera spacing, and sensor width in the image stitching process, which avoids the introduction of errors in these hardware parameters and effectively improves the accuracy of the image stitching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 This is a flow chart of an embodiment of an image stitching method in an embodiment of the present application;

[0069] Figure 2 is a schematic flow chart of a process for determining a splicing pixel offset relationship;

[0070] Figure 3 A schematic flow chart for determining an overlapping area between a first image to be stitched and a mapped image according to a preset stitching pixel offset relationship and a distance to be stitched;

[0071] Figure 4 A schematic flow chart of color gain for an image with large color differences;

[0072] Figure 5 A schematic flow chart of performing color gain on a first image to be stitched or a mapped image;

[0073] Figure 6 A schematic flow chart of fusing the first image to be stitched, the mapped image and the overlapped area;

[0074] Figure 7 Schematic diagram of the image fusion and stitching process;

[0075] Figure 8 This is a structural diagram of an embodiment of an image stitching device in an embodiment of the present application;

[0076] Fig. 9 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0079] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0080] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0081] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0082] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0083] The embodiment of the present application proposes an image stitching method that does not require the use of a large number of hardware parameters such as focal length, image width, camera spacing, and sensor width, thereby avoiding the introduction of errors in these hardware parameters during the image stitching process, and effectively improving the accuracy of the image stitching result. The execution subject of the embodiment of the present application can be any terminal device, including but not limited to mobile phones, tablet computers, desktop computers, notebooks, PDAs, etc., and the embodiment of the present application does not specifically limit the type of terminal device.

[0084] See also Figure 1 , an embodiment of an image stitching method in the embodiment of the present application may include:

[0085] Step S101, obtaining a first image to be stitched, a second image to be stitched, and a distance to be stitched.

[0086] Among them, the first image to be stitched (denoted as A) is an image captured by a preset first eyepiece lens, and the second image to be stitched (denoted as B) is an image captured by a preset second eyepiece lens. The first eyepiece lens and the second eyepiece lens can be two adjacent lenses in the same multi-eye camera. The multi-eye camera can be a binocular camera, a three-eye camera, a four-eye camera, etc. The type of the multi-eye camera can be a spherical eagle eye camera, a bowl-shaped eagle eye camera, a surround camera, etc. The distance to be stitched (denoted as L0) is the distance between the object in the overlapping area of ​​the first image to be stitched and the second image to be stitched and the first eyepiece lens. The overlapping area can be an image area corresponding to the same actual scene area in the images captured by different eyepiece lenses. The specific method for obtaining the distance to be stitched can be set according to the actual situation. For example, the distance to be stitched preset by the system can be obtained from the specified storage location of the terminal device, or the distance to be stitched set by the user can be obtained through the human-computer interaction interface of the terminal device. The embodiment of the present application does not specifically limit this.

[0087] Specifically, a multi-eye camera may include multiple lenses, and two adjacent lenses among the multiple lenses may be used as the first eyepiece lens and the second eyepiece lens, respectively. For example, the lens on the left of the two adjacent lenses may be used as the first eyepiece lens and the lens on the right may be used as the second eyepiece lens, or the lens on the right of the two adjacent lenses may be used as the first eyepiece lens and the lens on the left may be used as the second eyepiece lens.

[0088] In practical applications, when using a multi-eye camera for image acquisition, if the acquired image does not have image distortion, the image acquired by the first eye lens can be directly used as the first image to be stitched, and the image acquired by the second eye lens can be used as the second image to be stitched. If the acquired image has image distortion, the image acquired by the first eye lens can be firstly subjected to distortion correction, and the corrected image can be used as the first image to be stitched, and the image acquired by the second eye lens can be subjected to distortion correction, and the corrected image can be used as the second image to be stitched. Since the first eye lens and the second eye lens are adjacent to each other, and the orientations of the lenses in the multi-eye camera are usually consistent, there is usually an overlapping area in the first image to be stitched and the second image to be stitched.

[0089] Step S102: Map the second image to be stitched according to a preset first homography matrix to obtain a mapped image.

[0090] The first homography matrix (denoted as H1) is the homography matrix of the first eyepiece lens and the second eyepiece lens at a preset first reference stitching distance (denoted as L1). The value of the first reference stitching distance can be set according to actual conditions, and the embodiment of the present application does not specifically limit it.

[0091] In an embodiment of the present application, a first homography matrix may be predetermined before step S102. Specifically, the distance between the object in the overlapped area and the first eyepiece lens may be pre-adjusted to a first reference stitching distance, and image acquisition may be performed at this distance. If the acquired image does not have image distortion, the image acquired by the first eyepiece lens may be directly used as the first reference stitching image, and the image acquired by the second eyepiece lens may be used as the second reference stitching image. If the acquired image has image distortion, the image acquired by the first eyepiece lens may be firstly subjected to distortion correction, and the corrected image may be used as the first reference stitching image, and the image acquired by the second eyepiece lens may be subjected to distortion correction, and the corrected image may be used as the second reference stitching image.

[0092] After obtaining the first reference stitched image and the second reference stitched image, image feature points can be extracted from the two images respectively. The specific type of image feature points to be extracted can be set according to actual conditions, for example, it can be at least one of the following types: SIFT, SURF, ORB, HOG and LBP.

[0093] Based on the extracted image feature points, the Random Sample Consensus (RANSAC) method can be used to match the corresponding image feature points in the two images to obtain matching point pairs in the two images, and the first homography matrix is ​​determined based on the matching point pairs, as shown in the following formula:

[0094]

[0095] Among them, (u′, v′) is the pixel position coordinate value of the matching point in the first reference stitching image, and the corresponding homogeneous expression is (u′, v′, 1), (u, v) is the pixel position coordinate value of the matching point in the second reference stitching image, and the corresponding homogeneous expression is (u, v, 1), and the matching point in the first reference stitching image and the matching point in the second reference stitching image form a matching point pair.

[0096] Similarly, a second homography matrix (denoted as H2) can also be determined according to the above process, wherein the second homography matrix is ​​the homography matrix of the first eyepiece lens and the second eyepiece lens at a preset second reference stitching distance (denoted as L2), the first reference stitching distance is not equal to the second reference stitching distance, and the value of the second reference stitching distance can be set according to actual conditions, and the embodiment of the present application does not specifically limit it. The determination process of the second homography matrix is ​​similar to the determination process of the first homography matrix, and only the first reference stitching distance needs to be replaced by the second reference stitching distance. The specific process can refer to the determination process of the first homography matrix, and the embodiment of the present application will not be repeated here.

[0097] After determining the first homography matrix, the second image to be stitched can be mapped according to the first homography matrix to obtain a mapped image (denoted as B'), as shown in the following formula:

[0098] p B′ =H1·p B

[0099] Among them, p B is the homogeneous expression of the pixel position of the second image to be stitched, p B′ is the homogeneous expression of the pixel position of the mapped image. By mapping each pixel in the second image to be spliced ​​one by one through this expression, the mapped image can be obtained.

[0100] Step S103: determining an overlapping area between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and a distance to be stitched.

[0101] In the embodiment of the present application, before step S103, the following steps may be performed in advance: Figure 2 The process shown determines the stitching pixel offset relationship:

[0102] Step S201: Calculate a first stitching pixel offset according to a first homography matrix and a second homography matrix.

[0103] At various stitching distances, the binocular lens composed of the first eyepiece lens and the second eyepiece lens conforms to the binocular parallax formula shown below:

[0104]

[0105] Wherein, L is the stitching distance, that is, the distance between the object in the overlapping area of ​​the image collected by the first eyepiece lens and the image collected by the second eyepiece lens and the first eyepiece lens, F is the focal length of the first eyepiece lens, and b x ,B y is the distance between the binocular lens optical center and the image in the x and y directions, d x ,d y is the actual physical size of a single pixel on the first eyepiece sensor in the x and y directions of the image, p x ,p y is the disparity value in the x,y direction of the image, and When the camera is fixed, K x and K y Fixed but unknown value.

[0106] At the first reference stitching distance, there are:

[0107]

[0108] Among them, the disparity value p at the first reference stitching distance x1 ,p y1 Due to K x and K y Unknown, therefore unknown.

[0109] At the second reference stitching distance, there are:

[0110]

[0111] Among them, Δp x1 ,Δp y1 is the pixel offset of the second reference stitching distance relative to the first reference stitching distance, which is recorded as the first stitching pixel offset.

[0112] For any pixel position p(u,v,1), record it as the target pixel position and map it using the first homography matrix and the second homography matrix respectively:

[0113] p1′=H1·p

[0114] p2′=H2·p

[0115] Among them, p1′ is the first mapped pixel position obtained by mapping the target pixel position through the first homography matrix, and the corresponding homogeneous expression is (u′1, v′1, 1); p2′ is the second mapped pixel position obtained by mapping the target pixel position through the second homography matrix, and the corresponding homogeneous expression is (u′2, v′2, 1).

[0116] The first splicing pixel offset can be calculated according to the first mapping pixel position and the second mapping pixel position:

[0117]

[0118] Step S202: determining a splicing pixel offset relationship according to the first splicing pixel offset and a binocular parallax formula.

[0119] The pixel offset Δp in the first stitching x1 ,Δp y1 , and L1 and L2 are both known quantities, substitute these known quantities into the binocular disparity formula under the first reference stitching distance and the second reference stitching distance to calculate K x , K y , p x1 ,p y1 The specific value of .

[0120] At this time, the offset relationship of the stitching pixels at any distance can be expressed as:

[0121]

[0122] Among them, Δp x ,Δp y is the pixel offset of any stitching distance L relative to the first reference stitching distance.

[0123] After determining the splicing pixel offset relationship, the following Figure 3 The process shown determines the overlap area between the first image to be stitched and the mapped image:

[0124] Step S1031, calculating the offset of the pixel to be stitched according to the stitching pixel offset relationship and the distance to be stitched.

[0125] Substituting the distance to be stitched L0 into the above stitching pixel offset relationship, we can get:

[0126]

[0127] Among them, Δp x0 ,Δp y0 That is the pixel offset to be stitched.

[0128] Step S1032: determining a pixel correspondence between the first image to be spliced ​​and the mapped image according to the offset of the pixel to be spliced.

[0129] The pixel correspondence between the first image to be stitched and the mapped image can be expressed as follows:

[0130]

[0131] Among them, (uA ,v A ) is the pixel position coordinate in the first image to be stitched, (u′ B ,v′ B ) are the pixel position coordinates in the mapped image.

[0132] Step S1033: determining an overlapping area between the first image to be stitched and the mapped image according to the pixel correspondence relationship.

[0133] The pixels having a corresponding relationship between the first image to be stitched and the mapped image can be obtained by using the pixel correspondence relationship, and the area where these pixels are located is the overlap area (denoted as R) between the two images.

[0134] Step S104: performing fusion processing on the first image to be stitched, the mapped image and the overlapped area to obtain a stitched image.

[0135] In a specific implementation of the embodiment of the present application, the first image to be stitched, the mapped image and the overlapped area may be directly fused to obtain a stitched image.

[0136] In another specific implementation method in the embodiment of the present application, considering that there may be a large color difference between the first image to be stitched and the mapped image, color gain can be performed on the image first to reduce the color difference, and then the first image to be stitched, the mapped image and the overlapping area are fused to obtain a stitched image.

[0137] like Figure 4 As shown, performing color gain on an image with a large color difference may include the following process:

[0138] Step S401: Calculate a first average pixel value and a second average pixel value.

[0139] The first average pixel value is the average pixel value of the first image to be stitched in the overlapped area, which can be calculated according to the following formula:

[0140]

[0141] Where K is the number of pixels of the first image to be stitched in the overlapped area, k is a single pixel of the first image to be stitched in the overlapped area, and r Ak , g ak and b Ak are the red component, green component and blue component of pixel k of the first image to be stitched, respectively, A is the first average pixel value.

[0142] The second average pixel value is the average pixel value of the mapped image in the overlapped area, which can be calculated according to the following formula:

[0143]

[0144] Where K is the number of pixels of the second image to be stitched in the overlapped area, k is a single pixel of the second image to be stitched in the overlapped area, and r B′k , g B′k and b B′k are the red component, green component and blue component of pixel k of the second image to be stitched, respectively, B′ is the second average pixel value.

[0145] Step S402: Calculate the absolute value of the difference between the first average pixel value and the second average pixel value.

[0146] If the absolute value is less than or equal to the preset threshold, it means that the color difference between the first image to be stitched and the mapped image is small, so there is no need to perform color gain, and step S403 is no longer executed; if the absolute value is greater than the threshold, it means that there is a large color difference between the first image to be stitched and the mapped image, and step S403 is continued. The specific value of the threshold can be set according to actual conditions, for example, it can be set to 15 or other values, and the embodiment of the present application does not specifically limit this.

[0147] Step S403: Perform color gain on the first image to be stitched or the mapped image.

[0148] like Figure 5 As shown, step S403 may specifically include the following process:

[0149] Step S4031, calculate the first absolute value and the second absolute value.

[0150] The first absolute value is the absolute value of the difference between the first average pixel value and the preset middle pixel value, which can be calculated according to the following formula:

[0151] S A =abs(I A -Mid)

[0152] Among them, Mid is the middle pixel value. Taking the commonly used 256 color levels as an example, the corresponding middle pixel value is 128. abs is the absolute value function. S A is the first absolute value.

[0153] The second absolute value is the absolute value of the difference between the second average pixel value and the middle pixel value, which can be calculated according to the following formula:

[0154] S B′ =abs(I B′ -Mid)

[0155] Among them, S B′is the second absolute value.

[0156] If the first absolute value is smaller than the second absolute value, execute step S4032; if the first absolute value is greater than or equal to the second absolute value, execute step S4033.

[0157] Step S4032: taking the ratio of the first average pixel value to the second average pixel value as a first gain coefficient, and performing color gain on the mapped image according to the first gain coefficient.

[0158] Specifically, the RGB value of each pixel in the mapped image may be multiplied by the first gain coefficient to obtain a new RGB value. Since the upper limit of the RGB value is 256, if the multiplication result is greater than 256, the new RGB value is set to 256.

[0159] Step S4033: taking the ratio of the second average pixel value to the first average pixel value as a second gain coefficient, and performing color gain on the first image to be stitched according to the second gain coefficient.

[0160] Specifically, the RGB value of each pixel in the first image to be stitched may be multiplied by the second gain coefficient to obtain a new RGB value. Since the upper limit of the RGB value is 256, if the multiplication result is greater than 256, the new RGB value is set to 256.

[0161] In the embodiment of the present application, any fusion method such as feathering fusion and multi-band fusion can be used to fuse the first image to be spliced, the mapped image and the overlapped area according to the actual situation. As an example, the fusion process may specifically include the following: Figure 6 The process shown:

[0162] Step S1041: determine the image that does not belong to the overlapped area in the first images to be stitched as a first boundary image, and determine the image that belongs to the overlapped area in the first images to be stitched as a first overlapped image.

[0163] Step S1042: determine the image in the mapped image that does not belong to the overlapped area as the second boundary image, and determine the image in the mapped image that belongs to the overlapped area as the second overlapped image.

[0164] Step S1043: perform weighted processing on the first overlapped image and the second overlapped image to obtain an overlapped region fused image.

[0165] Each pixel in the first overlapping image and the second overlapping image corresponds one to one. Taking any pair of corresponding pixels as an example, the pixel in the first overlapping image can be recorded as the first pixel, the pixel in the first overlapping image that is in the same row as the first pixel and closest to the first boundary image can be recorded as the first boundary pixel, and the distance between the first pixel and the first boundary pixel is the first fusion distance. Similarly, the pixel in the second overlapping image can be recorded as the second pixel, the pixel in the second overlapping image that is in the same row as the second pixel and closest to the second boundary image can be recorded as the second boundary pixel, and the distance between the second pixel and the second boundary pixel is the second fusion distance.

[0166] Then, the first weight corresponding to the first pixel and the second weight corresponding to the second pixel can be determined according to the first fusion distance and the second fusion distance. The first weight is negatively correlated with the first fusion distance and positively correlated with the second fusion distance, that is, the smaller the first fusion distance and the larger the second fusion distance, the larger the first weight; the second weight is positively correlated with the first fusion distance and negatively correlated with the second fusion distance, that is, the larger the first fusion distance and the smaller the second fusion distance, the larger the second weight.

[0167] After determining the first weight and the second weight, the first product of the pixel value of the first pixel and the first weight, and the second product of the pixel value of the second pixel and the second weight are calculated respectively, and then the sum of the first product and the second product is determined as the pixel value of the fused pixel, which is the pixel after the first pixel and the second pixel are fused.

[0168] By traversing each pair of corresponding pixels in the first coincident image and the second coincident image and calculating the corresponding fused pixels respectively, a fused image of the coincident area can be obtained.

[0169] Step S1044: stitching the first boundary image, the overlapped region fused image and the second boundary image to obtain a stitched image.

[0170] Figure 7 The figure is a schematic diagram of the complete image fusion and stitching process, in which the first boundary image, the overlapped area fusion image and the second boundary image are sequentially connected to form a final stitched image.

[0171] Through the embodiments of the present application, there is no need to use numerous hardware parameters such as focal length, image width, camera spacing and sensor width during the image stitching process, thereby avoiding the introduction of errors in these hardware parameters and effectively improving the accuracy of the image stitching results.

[0172] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0173] Corresponding to the image stitching method described in the above embodiment, Figure 8 A structural diagram of an embodiment of an image stitching device provided in an embodiment of the present application is shown.

[0174] In this embodiment, an image stitching device may include:

[0175] The image acquisition module 801 is used to acquire a first image to be stitched, a second image to be stitched, and a distance to be stitched; wherein the first image to be stitched is an image captured by a preset first eyepiece lens, and the second image to be stitched is an image captured by a preset second eyepiece lens;

[0176] An image mapping module 802 is used to map the second image to be stitched according to a preset first homography matrix to obtain a mapped image; wherein the first homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset first reference stitching distance;

[0177] An overlap region determining module 803, configured to determine an overlap region between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and the distance to be stitched;

[0178] The fusion and stitching module 804 is used to fuse the first image to be stitched, the mapping image and the overlapped area to obtain a stitched image.

[0179] In a specific implementation of the embodiment of the present application, the image stitching device may further include:

[0180] A first stitching pixel offset calculation module, used to calculate a first stitching pixel offset according to the first homography matrix and a preset second homography matrix; wherein the second homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset second reference stitching distance; the first reference stitching distance is not equal to the second reference stitching distance;

[0181] The splicing pixel offset relationship determining module is used to determine the splicing pixel offset relationship according to the first splicing pixel offset and a preset binocular parallax formula.

[0182] In a specific implementation of the embodiment of the present application, the first splicing pixel offset calculation module may include:

[0183] A first mapping unit, configured to map a preset target pixel position according to the first homography matrix to obtain a first mapped pixel position;

[0184] A second mapping unit, configured to map the target pixel position according to the second homography matrix to obtain a second mapped pixel position;

[0185] The first splicing pixel offset calculation unit is used to calculate the first splicing pixel offset according to the first mapping pixel position and the second mapping pixel position.

[0186] In a specific implementation of the embodiment of the present application, the overlap area determination module may include:

[0187] A pixel offset calculation unit to be spliced, used for calculating the pixel offset to be spliced ​​according to the splicing pixel offset relationship and the distance to be spliced;

[0188] a pixel correspondence determination unit, configured to determine a pixel correspondence between the first image to be spliced ​​and the mapped image according to the pixel offset to be spliced;

[0189] An overlap region determining unit is used to determine an overlap region between the first image to be stitched and the mapped image according to the pixel correspondence relationship.

[0190] In a specific implementation of the embodiment of the present application, the image stitching device may further include:

[0191] A first calculation module, configured to calculate a first average pixel value and a second average pixel value; wherein the first average pixel value is an average pixel value of the first image to be stitched in the overlapped area, and the second average pixel value is an average pixel value of the mapped image in the overlapped area;

[0192] A second calculation module, configured to calculate an absolute value of a difference between the first average pixel value and the second average pixel value;

[0193] A color gain module is used to perform color gain on the first image to be stitched or the mapped image if the absolute value is greater than a preset threshold.

[0194] In a specific implementation of the embodiment of the present application, the color gain module may include:

[0195] an absolute value calculation unit, configured to calculate a first absolute value and a second absolute value; wherein the first absolute value is an absolute value of a difference between the first average pixel value and a preset intermediate pixel value; and the second absolute value is an absolute value of a difference between the second average pixel value and the intermediate pixel value;

[0196] a first color gain unit, configured to use a ratio of the first average pixel value to the second average pixel value as a first gain coefficient, and perform color gain on the mapped image according to the first gain coefficient if the first absolute value is smaller than the second absolute value;

[0197] The second color gain unit is configured to use a ratio of the second average pixel value to the first average pixel value as a second gain coefficient if the first absolute value is greater than or equal to the second absolute value, and perform color gain on the first image to be stitched according to the second gain coefficient.

[0198] In a specific implementation of the embodiment of the present application, the fusion and splicing module may include:

[0199] A first determining unit, configured to determine an image in the first images to be stitched that does not belong to the overlapped area as a first boundary image, and to determine an image in the first images to be stitched that belongs to the overlapped area as a first overlapped image;

[0200] A second determining unit, configured to determine an image in the mapped image that does not belong to the overlapped area as a second boundary image, and to determine an image in the mapped image that belongs to the overlapped area as a second overlapped image;

[0201] A weighted processing unit, used for performing weighted processing on the first coincident image and the second coincident image to obtain a fused image of the coincident region;

[0202] The stitching processing unit is used to stitch the first boundary image, the overlapped area fused image and the second boundary image to obtain the stitched image.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0204] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0205] Fig. 9 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0206] like Fig. 9As shown, the terminal device 9 of this embodiment includes: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90. When the processor 90 executes the computer program 92, the steps in the above-mentioned various image stitching method embodiments are implemented, for example Figure 1 Alternatively, when the processor 90 executes the computer program 92, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 8 The functions of modules 801 to 804 are shown.

[0207] Exemplarily, the computer program 92 may be divided into one or more modules / units, which are stored in the memory 91 and executed by the processor 90 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 92 in the terminal device 9.

[0208] The terminal device 9 may be a computing device such as a mobile phone, a tablet computer, a desktop computer, a notebook, a PDA, etc. Those skilled in the art will appreciate that Fig. 9 It is only an example of the terminal device 9 and does not constitute a limitation on the terminal device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 9 may also include input and output devices, network access devices, buses, etc.

[0209] The processor 90 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0210] The memory 91 may be an internal storage unit of the terminal device 9, such as a hard disk or memory of the terminal device 9. The memory 91 may also be an external storage device of the terminal device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 9. Further, the memory 91 may also include both an internal storage unit of the terminal device 9 and an external storage device. The memory 91 is used to store the computer program and other programs and data required by the terminal device 9. The memory 91 may also be used to temporarily store data that has been output or is to be output.

[0211] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0212] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0213] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0214] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0215] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0216] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0217] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0218] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image stitching method, characterized in that: include: Acquire a first image to be stitched, a second image to be stitched, and a distance to be stitched; wherein the first image to be stitched is an image captured by a preset first eyepiece lens, and the second image to be stitched is an image captured by a preset second eyepiece lens; Mapping the second image to be stitched according to a preset first homography matrix to obtain a mapped image; wherein the first homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset first reference stitching distance; Determine an overlap area between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and the distance to be stitched; wherein the stitching pixel offset relationship is used to characterize a pixel offset of any stitching distance relative to the first reference stitching distance; The first image to be stitched, the mapped image and the overlapped area are fused to obtain a stitched image.

2. The image stitching method according to claim 1, characterized in that: Before determining the overlap area between the first image to be stitched and the mapped image according to the preset stitching pixel offset relationship and the distance to be stitched, the method further includes: Calculating a first stitching pixel offset according to the first homography matrix and a preset second homography matrix; wherein the second homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset second reference stitching distance; the first reference stitching distance is not equal to the second reference stitching distance; The stitching pixel offset relationship is determined according to the first stitching pixel offset and a preset binocular disparity formula.

3. The image stitching method according to claim 2, characterized in that: The calculating the first splicing pixel offset according to the first homography matrix and a preset second homography matrix includes: Mapping a preset target pixel position according to the first homography matrix to obtain a first mapped pixel position; Mapping the target pixel position according to the second homography matrix to obtain a second mapped pixel position; The first stitching pixel offset is calculated according to the first mapped pixel position and the second mapped pixel position.

4. The image stitching method according to claim 1, characterized in that: The determining the overlapping area between the first image to be stitched and the mapped image according to the preset stitching pixel offset relationship and the distance to be stitched includes: Calculate the offset of the pixel to be stitched according to the stitching pixel offset relationship and the distance to be stitched; Determining a pixel correspondence relationship between the first image to be stitched and the mapped image according to the pixel offset to be stitched; An overlapping area between the first image to be stitched and the mapped image is determined according to the pixel correspondence relationship.

5. The image stitching method according to claim 1, characterized in that: Before fusing the first image to be stitched, the mapping image and the overlapped area, the method further includes: Calculate a first average pixel value and a second average pixel value; wherein the first average pixel value is an average pixel value of the first image to be stitched in the overlapping area, and the second average pixel value is an average pixel value of the mapped image in the overlapping area; Calculating an absolute value of a difference between the first average pixel value and the second average pixel value; If the absolute value is greater than a preset threshold, color gain is performed on the first image to be stitched or the mapped image.

6. The image stitching method according to claim 5, characterized in that: The performing color gain on the first image to be stitched or the mapped image includes: Calculating a first absolute value and a second absolute value; wherein the first absolute value is the absolute value of the difference between the first average pixel value and a preset middle pixel value; and the second absolute value is the absolute value of the difference between the second average pixel value and the middle pixel value; If the first absolute value is less than the second absolute value, taking the ratio of the first average pixel value to the second average pixel value as a first gain coefficient, and performing color gain on the mapped image according to the first gain coefficient; If the first absolute value is greater than or equal to the second absolute value, a ratio of the second average pixel value to the first average pixel value is used as a second gain coefficient, and color gain is performed on the first image to be stitched according to the second gain coefficient.

7. The image stitching method according to any one of claims 1 to 6, characterized in that: The step of fusing the first image to be stitched, the mapping image and the overlapped area to obtain a stitched image includes: Determine an image in the first images to be stitched that does not belong to the overlapped area as a first boundary image, and determine an image in the first images to be stitched that belongs to the overlapped area as a first overlapped image; Determine an image in the mapped image that does not belong to the overlapped area as a second boundary image, and determine an image in the mapped image that belongs to the overlapped area as a second overlapped image; Performing weighted processing on the first overlapped image and the second overlapped image to obtain an overlapped region fused image; The first boundary image, the overlapped area fused image and the second boundary image are spliced ​​to obtain the spliced ​​image.

8. An image stitching device, characterized in that: include: An image acquisition module, used to acquire a first image to be stitched, a second image to be stitched, and a distance to be stitched; wherein the first image to be stitched is an image captured by a preset first eyepiece lens, and the second image to be stitched is an image captured by a preset second eyepiece lens; An image mapping module, used for mapping the second image to be stitched according to a preset first homography matrix to obtain a mapping image; wherein the first homography matrix is ​​a homography matrix of the first eyepiece lens and the second eyepiece lens at a preset first reference stitching distance; an overlap region determination module, configured to determine an overlap region between the first image to be stitched and the mapped image according to a preset stitching pixel offset relationship and the distance to be stitched; wherein the stitching pixel offset relationship is used to characterize a pixel offset of any stitching distance relative to the first reference stitching distance; The fusion and stitching module is used to fuse the first image to be stitched, the mapping image and the overlapped area to obtain a stitched image.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image stitching method according to any one of claims 1 to 7 are implemented.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image stitching method according to any one of claims 1 to 7 are implemented.

Images