Image stitching method and apparatus, and panoramic camera

Abstract

The present application is applicable to the technical field of image processing, and provides an image stitching method, applied to a panoramic camera. The panoramic camera is provided with a first fisheye lens and a second fisheye lens. The image stitching method comprises: acquiring a first fisheye image by means of the first fisheye lens, and acquiring a second fisheye image by means of the second fisheye lens (S101); mapping the first fisheye image onto a unit spherical surface to obtain a first spherical image, and mapping the second fisheye image onto the unit spherical surface to obtain a second spherical image (S102); rotating the first spherical image by a specified angle to obtain a rotated first spherical image, and rotating the second spherical image by the specified angle to obtain a rotated second spherical image (S103); unwrapping the rotated first spherical image into a first ERP image, and unwrapping the rotated second spherical image into a second ERP image (S104); and stitching the first ERP image and the second ERP image to obtain a target image (S105). Thus, the image stitching accuracy is effectively improved.

Description

Image stitching method and device and panoramic camera TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image stitching method and device and a panoramic camera. BACKGROUND

[0002] In recent years, with the continuous development of science and technology, various devices such as panoramic cameras have sprung up in people's lives. A panoramic camera generally uses at least two lenses to take pictures at the same time, and then stitches the obtained images. However, the stitching accuracy of the images is not high at present. TECHNICAL PROBLEM

[0003] Embodiments of the present application provide an image stitching method, device and panoramic camera, which can solve the problem of low stitching accuracy of images at present. TECHNICAL SOLUTION

[0004] In a first aspect, the embodiments of the present application provide an image stitching method, which is applied to a panoramic camera, the panoramic camera is provided with a first fisheye lens and a second fisheye lens, and the image stitching method comprises the following steps: obtaining a first fisheye image based on the first fisheye lens and obtaining a second fisheye image based on the second fisheye lens; mapping the first fisheye image to a unit sphere to obtain a first spherical image, the first spherical image being a spherical image corresponding to the first fisheye image, and mapping the second fisheye image to the unit sphere to obtain a second spherical image, the second spherical image being a spherical image corresponding to the second fisheye image; rotating the first spherical image by a specified angle to obtain a rotated first spherical image, and rotating the second spherical image by the specified angle to obtain a rotated second spherical image; unfolding the rotated first spherical image into a first ERP image and unfolding the rotated second spherical image into a second ERP image, wherein the first ERP image is an image obtained by unfolding the rotated first spherical image using an ERP method, and the second ERP image is an image obtained by unfolding the rotated second spherical image using the ERP method; and stitching the first ERP image and the second ERP image to obtain a target image.

[0005] In a possible implementation manner of the first aspect, the first fisheye lens and the second fisheye lens are symmetrically arranged based on a first plane, the first fisheye lens and the second fisheye lens are respectively located on two sides of the first plane, and before the rotating the first spherical image by a specified angle to obtain a rotated first spherical image and the rotating the second spherical image by the specified angle to obtain a rotated second spherical image, the method comprises: establishing a spherical coordinate system, the spherical coordinate system being a three-dimensional rectangular coordinate system, an origin of the spherical coordinate system being an intersection of an optical center connecting line and the first plane, the optical center connecting line being a connecting line between an optical center of the first fisheye lens and an optical center of the second fisheye lens, a first axis of the spherical coordinate system being perpendicular to the first plane, a second axis of the spherical coordinate system being perpendicular to the first axis and on the first plane, a direction in which the second axis is located being vertical upward, and a third axis of the spherical coordinate system being perpendicular to a second plane, the second plane being a plane in which the first axis and the second axis are located; and correspondingly, the rotating the first spherical image by a specified angle to obtain a rotated first spherical image and the rotating the second spherical image by the specified angle to obtain a rotated second spherical image comprise: rotating the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and rotating the second spherical image by 90 degrees around the third axis in the specified direction to obtain a rotated second spherical image.

[0006] In a possible implementation manner of the first aspect, the rotating the first spherical image by a specified angle to obtain a rotated first spherical image and the rotating the second spherical image by the specified angle to obtain a rotated second spherical image comprise: rotating the first spherical image by 90 degrees around the third axis in a counterclockwise direction to obtain a rotated first spherical image, and rotating the second spherical image by 90 degrees around the third axis in the counterclockwise direction to obtain a rotated second spherical image.

[0007] In a possible implementation manner of the first aspect, the splicing the first ERP image and the second ERP image to obtain a target image comprises: determining a target region in the first ERP image and a target region in the second ERP image, the target region in the first ERP image being a region in the first ERP image having a plurality of same feature points as the second ERP image, and the target region in the second ERP image being a region in the second ERP image having a plurality of same feature points as the first ERP image; determining a relative displacement between the same feature points based on the target region in the first ERP image and the target region in the second ERP image; and splicing the first ERP image and the second ERP image based on the relative displacement to obtain the target image.

[0008] In a possible implementation manner of the first aspect, the splicing the first ERP image and the second ERP image to obtain a target image based on the relative displacement comprises: performing orthogonal decomposition on the relative displacement in a two-dimensional longitude and latitude coordinate system to obtain a component of the relative displacement in a longitude extension direction; performing registration on the target region in the first ERP image and the target region in the second ERP image based on the component of the relative displacement in the longitude extension direction to obtain a registration result; and splicing the first ERP image and the second ERP image according to the registration result to obtain the target image.

[0009] In a possible implementation manner of the first aspect, the target image is a two-dimensional planar image, and after the splicing the first ERP image and the second ERP image to obtain a target image, the method further comprises: converting the target image into a third spherical image; rotating the third spherical image by 90 degrees in a reverse direction around the third axis to obtain a rotated third spherical image, wherein the reverse direction is a direction opposite to the specified direction; and unfolding the rotated third spherical image into a third ERP image, wherein the third ERP image is an image obtained by unfolding the rotated third spherical image using the ERP manner.

[0010] In a second aspect, an image stitching apparatus is provided. The image stitching apparatus is applied to a panoramic camera. The panoramic camera is provided with a first fisheye lens and a second fisheye lens. The image stitching apparatus comprises: a first obtaining unit, configured to obtain a first fisheye image based on the first fisheye lens, and obtain a second fisheye image based on the second fisheye lens; a second obtaining unit, configured to map the first fisheye image onto a unit sphere to obtain a first spherical image, the first spherical image being a spherical image corresponding to the first fisheye image, and map the second fisheye image onto the unit sphere to obtain a second spherical image, the second spherical image being a spherical image corresponding to the second fisheye image; a third obtaining unit, configured to rotate the first spherical image by a specified angle to obtain a rotated first spherical image, and rotate the second spherical image by the specified angle to obtain a rotated second spherical image; an unfolding unit, configured to unfold the rotated first spherical image into a first ERP image, and unfold the rotated second spherical image into a second ERP image, wherein the first ERP image is an image obtained by unfolding the rotated first spherical image by using an ERP method, and the second ERP image is an image obtained by unfolding the rotated second spherical image by using the ERP method; and a stitching unit, configured to stitch the first ERP image and the second ERP image to obtain a target image.

[0011] In a third aspect, a panoramic camera is provided. The panoramic camera comprises a first fisheye lens, a second fisheye lens, a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor implements the method of any one of the above aspects when executing the computer program.

[0012] In a fourth aspect, a computer readable storage medium is provided. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the method of any one of the above aspects.

[0013] It can be understood that the beneficial effects of the second aspect described above can be referred to the related description in the first aspect described above, and will not be repeated here. Beneficial effects

[0014] The beneficial effects of the embodiment of the present application compared with the prior art are: the image splicing method in the embodiment of the present application is applied to a panoramic camera, and the first spherical image can be rotated by a specified angle to obtain a rotated first spherical image, and the second spherical image can be rotated by the specified angle to obtain a rotated second spherical image; the rotated first spherical image is unfolded into a first ERP image, the rotated second spherical image is unfolded into a second ERP image, and then the first ERP image and the second ERP image are spliced to obtain a target image, so that higher splicing precision can be achieved. The specific reasons are as follows: compared with the prior art (the prior art is: without rotating the spherical image, directly unfolding the spherical image by using the ERP method, and then completing splicing), the embodiment of the present application can obtain a larger number of image pixels for completing splicing, so that higher splicing precision can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative effort.

[0016] FIG. 1 is a flow diagram of an image splicing method according to an embodiment of the present application; FIG. 2 is a schematic diagram of a panoramic camera according to an embodiment of the present application; FIG. 3 is a schematic diagram of an image obtained based on the prior art; FIG. 4 is a schematic diagram of an image according to an embodiment of the present application, including a schematic diagram of a first ERP image and a second ERP image; and FIG. 5 is a schematic diagram of an image splicing device according to an embodiment of the present application. Embodiments of the present application

[0017] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary details.

[0018] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0019] It should also be understood that the term “and / or” as used herein refers to any one of the associated listed items, optionally, and all possible combinations of the items, and encompasses the items recited by the expression.

[0020] As used in the description of the application and the appended claims, the term “if’ can be interpreted to mean “when” or “upon” or “in response to determining” or “in response to detecting” depending on the context. Similarly, the phrase “if it is determined” or “if [a described condition or event] is detected” can be interpreted to mean “upon determining” or “in response to determining” or “upon [the described condition or event] being detected” or “in response to [the described condition or event] being detected,” depending on the context.

[0021] In addition, in the description of the application and the appended claims, the terms “first”, “second”, “third”, etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0022] Reference in the specification to “one embodiment” or “some embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase “in one embodiment” or “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms “including”, “containing”, “having” and variations thereof are meant to encompass the terms “including but not limited to”, unless otherwise expressly specified or limited by context.

[0023] Embodiment one

[0024] FIG. 1 shows a flow diagram of an image stitching method provided by an embodiment of the application, which can be applied to a panoramic camera. The image stitching method is applied to a panoramic camera, which is provided with a first fisheye lens and a second fisheye lens. For example, the panoramic camera 20 can be as shown in FIG. 2, in which the first fisheye lens 201 and the second fisheye lens 202 are symmetrically arranged on the two sides of the panoramic camera. The image stitching method includes steps S101, S102, S103, S104 and S105. Details are as follows:

[0025] Step S101, acquiring a first fisheye image based on the first fisheye lens and a second fisheye image based on the second fisheye lens.

[0026] The first fisheye image and the second fisheye image are images obtained by simultaneous shooting, the first fisheye image and the second fisheye image are both two-dimensional images, the field of view angle corresponding to the first fisheye image and the field of view angle corresponding to the second fisheye image are both greater than 180 degrees, part of the region in the first fisheye image and part of the region in the second fisheye image are the same, that is, part of the region in the first fisheye image and part of the region in the second fisheye image have the same feature points, part of the region in the first fisheye image is located at the edge position in the first fisheye image, part of the region in the second fisheye image is located at the edge position in the second fisheye image, and the pixel distortion of part of the region in the first fisheye image and part of the region in the second fisheye image is extremely serious.

[0027] In step S102, the first fisheye image is mapped to a unit sphere to obtain a first spherical image corresponding to the first fisheye image, and the second fisheye image is mapped to the unit sphere to obtain a second spherical image corresponding to the second fisheye image.

[0028] The unit sphere is a spherical surface of a unit sphere.

[0029] By way of example and not limitation, the step S102 includes: using the intrinsic and extrinsic parameters obtained by calibrating the first fisheye lens, mapping the coordinates of each pixel in the first fisheye image in the image coordinate system (two-dimensional coordinate system) to the unit sphere to obtain the spherical coordinates corresponding to the first fisheye image, that is, the first spherical image can be obtained, the spherical coordinates are coordinates in the spherical coordinate system; using the intrinsic and extrinsic parameters obtained by calibrating the second fisheye lens, mapping the coordinates of each pixel in the second fisheye image in the image coordinate system to the unit sphere to obtain the spherical coordinates corresponding to the second fisheye image, that is, the second spherical image can be obtained.

[0030] In step S103, the first spherical image is rotated by a specified angle to obtain a rotated first spherical image, and the second spherical image is rotated by the specified angle to obtain a rotated second spherical image.

[0031] Optionally, the first fisheye lens and the second fisheye lens are symmetrically arranged based on a first plane, the first fisheye lens and the second fisheye lens are respectively located on two sides of the first plane, and before the step S103, the method further comprises: establishing a spherical coordinate system, the spherical coordinate system being a three-dimensional rectangular coordinate system, an origin of the spherical coordinate system being an intersection of an optical center connecting line and the first plane, the optical center connecting line being a connecting line between an optical center of the first fisheye lens and an optical center of the second fisheye lens, a first axis of the spherical coordinate system being perpendicular to the first plane, a second axis of the spherical coordinate system being perpendicular to the first axis and on the first plane, a direction in which the second axis is located being vertical upward, and a third axis of the spherical coordinate system being perpendicular to a second plane, the second plane being a plane in which the first axis and the second axis are located; correspondingly, the step S103 comprises: rotating the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and rotating the second spherical image by 90 degrees around the third axis in the specified direction to obtain a rotated second spherical image.

[0032] Specifically, the first plane is a symmetry plane of the panoramic camera, and neither the first fisheye lens nor the second fisheye lens passes through the first plane.

[0033] By way of example, and not limitation, the rotating the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and rotating the second spherical image by 90 degrees around the third axis in the specified direction to obtain a rotated second spherical image comprises: rotating the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image based on a specified relationship and a rotation matrix, wherein the specified relationship is , wherein, is used to represent a rotated spherical image, is used to represent an unrotated spherical image. is used to represent a rotation matrix, rotating the second spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated second spherical image based on the specified relationship and the rotation matrix. Wherein, may be 90 degrees or -90 degrees; , and , and The same point of the value 90 degrees and the value -90 degrees is that both represent a rotation of 90 degrees. The difference between the value 90 degrees and the value -90 degrees is that they represent different rotation directions.

[0034] In the embodiment, the first spherical image is rotated by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and the second spherical image is rotated by 90 degrees around the third axis in the specified direction to obtain a rotated second spherical image, and then the rotated first spherical image and the rotated second spherical image are unfolded by using the equirectangular projection mode, so that the edges of the unfolded images are relatively flat, and the number of pixels abandoned for splicing can be effectively reduced in the subsequent splicing process, that is, more image pixels are obtained for completing splicing, so that higher splicing precision is achieved.

[0035] In some embodiments, the rotating the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and the rotating the second spherical image by 90 degrees around the third axis in the specified direction to obtain a rotated second spherical image, comprises: rotating the first spherical image by 90 degrees around the third axis in a clockwise direction to obtain a rotated first spherical image, and rotating the second spherical image by 90 degrees around the third axis in the clockwise direction to obtain a rotated second spherical image.

[0036] In some embodiments, the rotating the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and the rotating the second spherical image by 90 degrees around the third axis in the specified direction to obtain a rotated second spherical image, comprises: rotating the first spherical image by 90 degrees around the third axis in a counterclockwise direction to obtain a rotated first spherical image, and rotating the second spherical image by 90 degrees around the third axis in the counterclockwise direction to obtain a rotated second spherical image.

[0037] In step S104, the rotated first spherical image is unfolded into a first equirectangular projection (ERP) image, and the rotated second spherical image is unfolded into a second ERP image, wherein the first ERP image is an image obtained by unfolding the rotated first spherical image by using the ERP mode, and the second ERP image is an image obtained by unfolding the rotated second spherical image by using the ERP mode.

[0038] As an example but not limitation, the first ERP image and the second ERP image can be displayed in a two-dimensional latitude-longitude coordinate system, the top-left corner vertex of the first ERP image / the top-left corner vertex of the second ERP image being the origin of the two-dimensional latitude-longitude coordinate system, the two-dimensional latitude-longitude coordinate system can be represented as a uv coordinate system, the horizontal coordinate u can be linearly mapped to the longitude, the longitude taking a value range of [-180°, 180°]; the vertical coordinate v can be linearly mapped to the latitude, the latitude taking a value range of [-90°, 90°].

[0039] In the prior art, if the spherical image is not rotated, the spherical image is directly unfolded using the ERP method, and the unfolded result can be as shown in FIG. 3, and the unfolded images 301 and 302 are obtained, wherein the area enclosed by the dashed line is used to represent the invalid area caused by the visual field blind area. Theoretically, the pixels that can be used for stitching are distributed on the left and right sides of the image 301 and the left and right sides of the image 302, and actually, the prior art uses one side of the image 301 and one side of the image 302 for stitching, and the other side of the image 301 and the other side of the image 302 are not used for stitching, that is, the number of pixels used for stitching is actually small, and the stitching efficiency and stitching accuracy are low.

[0040] In the present application, the first spherical image can be rotated by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and the second spherical image can be rotated by 90 degrees around the third axis in a specified direction to obtain a rotated second spherical image, and then the first ERP image and the second ERP image are obtained, and the first ERP image 401 and the second ERP image 402 can be as shown in FIG. 4, the area that can be used for stitching in the first ERP image 401 is the black area of the first ERP image 401, which is located at the lower end of the first ERP image 401; the area that can be used for stitching in the second ERP image 402 is the black area in the second ERP image 402, which is located at the upper end of the second ERP image 402, since the pixels that can be used for stitching in the first ERP image 401 are all concentrated on one side of the first ERP image 401, and the pixels that can be used for stitching in the second ERP image 402 are also all concentrated on one side of the second ERP image 402, in this way, the number of pixels used for image stitching can be greatly improved, and then the stitching accuracy and the stitching efficiency can be improved.

[0041] Step S105, stitching the first ERP image and the second ERP image to obtain a target image.

[0042] Specifically, the first ERP image and the second ERP image are both rectangular regions, the rectangular region has a length and a width, and the length of the first ERP image and the length of the second ERP image are used for stitching to obtain a target image.

[0043] Optionally, due to the manufacturing process limitation of two fisheye lenses, there is a parallax between the two fisheye lenses,

[0044] In order to reduce the negative influence of the parallax on image stitching, therefore, the step S105 comprises: a step a1, a step a2 and a step a3, wherein:

[0045] The step a1 is to determine a target region in the first ERP image and a target region in the second ERP image, the target region in the first ERP image being a region in the first ERP image having a plurality of same feature points with the second ERP image, and the target region in the second ERP image being a region in the second ERP image having a plurality of same feature points with the first ERP image.

[0046] The step a2 is to determine a relative displacement between the same feature points based on the target region in the first ERP image and the target region in the second ERP image.

[0047] The step a3 is to stitch the first ERP image and the second ERP image based on the relative displacement to obtain a target image.

[0048] In a two-dimensional latitude and longitude coordinate system, the target region in the first ERP image spans a longitude range in the latitude direction equal to the longitude range spanned by the first ERP image in the latitude direction, and the target region in the second ERP image spans a longitude range in the latitude direction equal to the longitude range spanned by the second ERP image in the latitude direction.

[0049] For example, the target region in the first ERP image has a feature point b1 and a feature point b2, and the target region in the second ERP image has a feature point c1 and a feature point c2, wherein the feature point b1 and the feature point c1 are the same feature points, and the feature point b2 and the feature point c2 are the same feature points. The relative displacement between the feature point b1 and the feature point c1 is determined, and the relative displacement between the feature point b2 and the feature point c2 is determined. Correspondingly, the step a3 comprises: stitching the first ERP image and the second ERP image based on the relative displacement between the feature point b1 and the feature point c1 and the relative displacement between the feature point b2 and the feature point c2 to obtain a target image.

[0050] In some embodiments, the step a3 comprises: orthogonally decomposing the relative displacement in a two-dimensional longitude-latitude coordinate system to obtain a component of the relative displacement in a longitude-extending direction; registering the target region in the first ERP image and the target region in the second ERP image based on the component of the relative displacement in the longitude-extending direction to obtain a registration result; and splicing the first ERP image and the second ERP image according to the registration result to obtain a target image.

[0051] For example, the two-dimensional longitude-latitude coordinate system is represented as a uv coordinate system, and the longitude-extending direction is a direction in which a v-axis is located. Correspondingly, the relative displacement is orthogonally decomposed in the two-dimensional longitude-latitude coordinate system to obtain a component of the relative displacement in the direction in which the v-axis is located; the target region in the first ERP image and the target region in the second ERP image are registered based on the component of the relative displacement in the direction in which the v-axis is located to obtain a registration result; and the first ERP image and the second ERP image are spliced according to the registration result to obtain a target image.

[0052] Optionally, the target image is a two-dimensional planar image, and after the step S105, the method further comprises: converting the target image into a third spherical image; rotating the third spherical image by 90 degrees in a reverse direction around the third axis to obtain a rotated third spherical image, wherein the reverse direction is opposite to the specified direction; and unfolding the rotated third spherical image into a third ERP image, wherein the third ERP image is an image obtained by unfolding the rotated third spherical image using the ERP manner.

[0053] In some embodiments, after the step of unfolding the rotated third spherical image into a third ERP image, the method further comprises: performing target detection based on the third ERP image to obtain a target detection result. In this way, a more accurate target detection result can be obtained.

[0054] In some embodiments, after the step of unfolding the rotated third spherical image into a third ERP image, the method further comprises: performing target detection based on the third ERP image to obtain a target detection result. In this way, a more accurate target detection result can be obtained.

[0055] The image splicing method in the embodiments of the present application is applied to a panoramic camera. The first spherical image can be rotated by a specified angle to obtain a rotated first spherical image, and the second spherical image can be rotated by the specified angle to obtain a rotated second spherical image. The rotated first spherical image is unfolded into a first ERP image, and the rotated second spherical image is unfolded into a second ERP image. Then, the first ERP image and the second ERP image are spliced to obtain a target image. Therefore, higher splicing accuracy can be achieved. The specific reasons are as follows: Compared with the prior art (the prior art is that a spherical image is not rotated, but is directly unfolded into an ERP image, and then splicing is completed), the embodiments of the present application can obtain a larger number of image pixels for completing splicing, thereby achieving higher splicing accuracy.

[0056] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution. The execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0057] Embodiment Two

[0058] Corresponding to the image splicing method described in the above embodiments, FIG. 5 shows a schematic diagram of an image splicing device provided by the embodiments of the present application. For ease of illustration, only the parts related to the embodiments of the present application are shown.

[0059] The image splicing device is applied to a panoramic camera, and the panoramic camera is provided with a first fisheye lens and a second fisheye lens. The image splicing device comprises a first acquisition unit 501, a second acquisition unit 502, a third acquisition unit 503, an unfolding unit 504, and a splicing unit 505, wherein:

[0060] The first acquisition unit 501 is configured to acquire a first fisheye image based on the first fisheye lens, and acquire a second fisheye image based on the second fisheye lens.

[0061] The second acquisition unit 502 is configured to map the first fisheye image onto a unit sphere to obtain a first spherical image, the first spherical image being a spherical image corresponding to the first fisheye image, and map the second fisheye image onto the unit sphere to obtain a second spherical image, the second spherical image being a spherical image corresponding to the second fisheye image.

[0062] The third acquisition unit 503 is configured to rotate the first spherical image by a specified angle to obtain a rotated first spherical image, and rotate the second spherical image by the specified angle to obtain a rotated second spherical image.

[0063] The unfolding unit 504 is configured to unfold the rotated first spherical image into a first ERP image and to unfold the rotated second spherical image into a second ERP image, wherein the first ERP image is an image obtained by unfolding the rotated first spherical image using an ERP method, and the second ERP image is an image obtained by unfolding the rotated second spherical image using the ERP method.

[0064] The splicing unit 505 is configured to splice the first ERP image and the second ERP image to obtain a target image.

[0065] Optionally, the first fisheye lens and the second fisheye lens are symmetrically arranged based on a first plane, the first fisheye lens and the second fisheye lens are respectively located on two sides of the first plane, and the image splicing apparatus comprises a coordinate system establishing unit.

[0066] The coordinate system establishing unit is configured to establish a spherical coordinate system before the third obtaining unit performs the rotating the first spherical image by a specified angle to obtain a rotated first spherical image and the rotating the second spherical image by the specified angle to obtain a rotated second spherical image, the spherical coordinate system is a three-dimensional rectangular coordinate system, an origin of the spherical coordinate system is an intersection of an optical center connecting line and the first plane, the optical center connecting line is a connecting line between an optical center of the first fisheye lens and an optical center of the second fisheye lens, a first axis of the spherical coordinate system is perpendicular to the first plane, a second axis of the spherical coordinate system is perpendicular to the first axis and is on the first plane, a direction in which the second axis is located is vertically upward, a third axis of the spherical coordinate system is perpendicular to a second plane, and the second plane is a plane in which the first axis and the second axis are located; correspondingly, when performing the rotating the first spherical image by a specified angle to obtain a rotated first spherical image and the rotating the second spherical image by the specified angle to obtain a rotated second spherical image, the third obtaining unit is configured to rotate the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and to rotate the second spherical image by 90 degrees around the third axis in the specified direction to obtain a rotated second spherical image.

[0067] Optionally, the target image is a two-dimensional plane image, and the image splicing apparatus further comprises an image processing unit.

[0068] The image processing unit is configured to: after the splicing unit 505 performs the splicing of the first ERP image and the second ERP image to obtain a target image, convert the target image into a third spherical image; rotate the third spherical image by 90 degrees in a reverse direction around the third axis to obtain a rotated third spherical image, wherein the reverse direction is opposite to the specified direction; and develop the rotated third spherical image into a third ERP image, wherein the third ERP image is an image obtained by developing the rotated third spherical image into the third ERP image using the ERP method.

[0069] In some embodiments, the image splicing apparatus further comprises a target detection unit.

[0070] The target detection unit is configured to, after the image processing unit develops the rotated third spherical image into a third ERP image, perform target detection based on the third ERP image to obtain a target detection result. In this way, a more accurate target detection result can be obtained.

[0071] The present application also provides another preferred embodiment of an image splicing apparatus. In this embodiment, the image splicing apparatus comprises a processor, wherein the processor is configured to execute the following program modules with a memory: a first acquisition unit 501 configured to acquire a first fisheye image based on a first fisheye lens and acquire a second fisheye image based on a second fisheye lens; a second acquisition unit 502 configured to map the first fisheye image onto a unit sphere to obtain a first spherical image, which is a spherical image corresponding to the first fisheye image, and map the second fisheye image onto the unit sphere to obtain a second spherical image, which is a spherical image corresponding to the second fisheye image; a third acquisition unit 503 configured to rotate the first spherical image by a specified angle to obtain a rotated first spherical image and rotate the second spherical image by the specified angle to obtain a rotated second spherical image; a developing unit 504 configured to develop the rotated first spherical image into a first ERP image and develop the rotated second spherical image into a second ERP image, wherein the first ERP image is an image obtained by developing the rotated first spherical image into the first ERP image using an ERP method, and the second ERP image is an image obtained by developing the rotated second spherical image into the second ERP image using the ERP method; and a splicing unit 505 configured to splice the first ERP image and the second ERP image to obtain a target image.

[0072] In some embodiments, the processor is further configured to execute the following program modules with a memory: a coordinate system establishing unit.

[0073] In some embodiments, the target image is a two-dimensional planar image, and the processor further executes the following program modules stored in the memory: an image processing unit.

[0074] In some embodiments, the processor further executes the following program modules stored in the memory: a target detection unit.

[0075] It should be noted that technical details not described in detail in this embodiment can refer to the image stitching method provided by each embodiment in Embodiment One.

[0076] Embodiment Three

[0077] The panoramic camera in this embodiment includes a first fisheye lens, a second fisheye lens, at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor implements the steps in any of the above image stitching method embodiments when executing the computer program.

[0078] The processor implements the steps in any of the above image stitching method embodiments when executing the computer program, such as steps S101 to S105 shown in FIG. 1. Alternatively, the processor implements the functions of the units in the above device embodiments when executing the computer program, such as the functions of units 501 to 505 shown in FIG. 5.

[0079] Those skilled in the art can understand that this embodiment is only an example of a panoramic camera and does not limit the panoramic camera, which can include more or fewer components than the illustration, or combine certain components or different components, such as also including an input / output device, a network access device, etc.

[0080] The processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0081] The memory can be an internal storage unit of the panoramic camera in some embodiments, such as a hard disk or a memory of the panoramic camera. The memory can also be an external storage device of the panoramic camera in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, and the like equipped on the panoramic camera. Further, the memory can include both the internal storage unit and the external storage device of the panoramic camera. The memory is used to store an operating system, an application program, a BootLoader, data, and other programs, such as program codes of the computer program. The memory can also be used to temporarily store data that has been output or is to be output.

[0082] It should be noted that the information interaction and execution process between the above devices / units are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought by the above devices / units can be referred to the method embodiments part, which will not be described here.

[0083] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0084] The computer readable storage medium of the present application embodiment stores a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.

[0085] The computer program product of the present application embodiment, when running on the panoramic camera, enables the panoramic camera to execute the steps in each of the above method embodiments.

[0086] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above-mentioned embodiment methods through a computer program to instruct relevant hardware to complete, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the panoramic camera, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0087] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0088] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0089] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutually can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0090] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0091] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Industrial applicability

[0092] The present application is applied to a panoramic camera, and the present application can effectively improve the precision of image stitching.

Claims

1. An image stitching method, characterized by, The image stitching method is applied to a panoramic camera provided with a first fisheye lens and a second fisheye lens, and the image stitching method comprises: acquiring a first fisheye image based on the first fisheye lens and a second fisheye image based on the second fisheye lens; mapping the first fisheye image onto a unit sphere to obtain a first spherical image corresponding to the first fisheye image, and mapping the second fisheye image onto the unit sphere to obtain a second spherical image corresponding to the second fisheye image; rotating the first spherical image by a specified angle to obtain a rotated first spherical image, and rotating the second spherical image by the specified angle to obtain a rotated second spherical image; unfolding the rotated first spherical image into a first ERP image and unfolding the rotated second spherical image into a second ERP image, wherein the first ERP image is obtained by unfolding the rotated first spherical image using an ERP method, and the second ERP image is obtained by unfolding the rotated second spherical image using the ERP method; stitching the first ERP image and the second ERP image to obtain a target image.

2. The image stitching method of claim 1, wherein, The first fisheye lens and the second fisheye lens are symmetrically arranged based on a first plane, and the first fisheye lens and the second fisheye lens are located on two sides of the first plane, and before the rotating the first spherical image by a specified angle to obtain a rotated first spherical image, and the rotating the second spherical image by the specified angle to obtain a rotated second spherical image, the method comprises: establishing a spherical coordinate system, wherein the spherical coordinate system is a three-dimensional rectangular coordinate system, the origin of the spherical coordinate system is the intersection of the optical center connecting line and the first plane, the optical center connecting line is the connecting line between the optical center of the first fisheye lens and the optical center of the second fisheye lens, the first axis of the spherical coordinate system is perpendicular to the first plane, the second axis of the spherical coordinate system is perpendicular to the first axis on the first plane, the direction of the second axis is vertically upward, and the third axis of the spherical coordinate system is perpendicular to the second plane, wherein the second plane is the plane on which the first axis and the second axis are located; correspondingly, the rotating the first spherical image by a specified angle to obtain a rotated first spherical image, and the rotating the second spherical image by the specified angle to obtain a rotated second spherical image, comprises: rotating the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and rotating the second spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated second spherical image.

3. The image stitching method of claim 2, wherein, The rotating the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and the rotating the second spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated second spherical image, comprises: rotating the first spherical image by 90 degrees in a counterclockwise direction around the third axis to obtain a rotated first spherical image, and rotating the second spherical image by 90 degrees in a counterclockwise direction around the third axis to obtain a rotated second spherical image.

4. The image stitching method of claim 1, wherein, The splicing the first ERP image and the second ERP image to obtain a target image comprises: determining a target region in the first ERP image and a target region in the second ERP image, the target region in the first ERP image being a region in the first ERP image having a plurality of same feature points as the second ERP image, and the target region in the second ERP image being a region in the second ERP image having a plurality of same feature points as the first ERP image; determining a relative displacement between the same feature points based on the target region in the first ERP image and the target region in the second ERP image; splicing the first ERP image and the second ERP image based on the relative displacement to obtain a target image.

5. The image stitching method of claim 4, wherein, The splicing the first ERP image and the second ERP image based on the relative displacement to obtain a target image comprises: orthogonally decomposing the relative displacement in a two-dimensional latitude-longitude coordinate system to obtain a component of the relative displacement in a meridian direction; registering the target region in the first ERP image and the target region in the second ERP image based on the component of the relative displacement in the meridian direction to obtain a registration result; splicing the first ERP image and the second ERP image according to the registration result to obtain a target image.

6. The image stitching method of claim 2, wherein, The target image is a two-dimensional planar image, and after the splicing the first ERP image and the second ERP image to obtain a target image, the method comprises: converting the target image into a third spherical image; rotating the third spherical image by 90 degrees in a reverse direction around the third axis to obtain a rotated third spherical image, wherein the reverse direction is a direction opposite to the specified direction; unfolding the rotated third spherical image into a third ERP image, wherein the third ERP image is an image obtained by unfolding the rotated third spherical image using the ERP method.

7. An image stitching apparatus characterized by comprising: The image splicing device is applied to a panoramic camera, and the panoramic camera is provided with a first fisheye lens and a second fisheye lens. The first acquisition unit is configured to acquire a first fisheye image based on the first fisheye lens and a second fisheye image based on the second fisheye lens. The second acquisition unit is configured to map the first fisheye image onto a unit sphere to obtain a first spherical image, the first spherical image being a spherical image corresponding to the first fisheye image, and map the second fisheye image onto the unit sphere to obtain a second spherical image, the second spherical image being a spherical image corresponding to the second fisheye image. The third acquisition unit is configured to rotate the first spherical image by a specified angle to obtain a rotated first spherical image, and rotate the second spherical image by the specified angle to obtain a rotated second spherical image. The unwinding unit is configured to unwind the rotated first spherical image into a first ERP image, and unwind the rotated second spherical image into a second ERP image, wherein the first ERP image is obtained by unwinding the rotated first spherical image using an ERP method, and the second ERP image is obtained by unwinding the rotated second spherical image using the ERP method. The splicing unit is configured to splice the first ERP image and the second ERP image to obtain a target image.

8. The image stitching apparatus of claim 7, wherein, The first fisheye lens and the second fisheye lens are symmetrically arranged based on a first plane, the first fisheye lens and the second fisheye lens are respectively located on two sides of the first plane, and the image splicing device comprises a coordinate system establishing unit. Before the third acquisition unit performs the rotation of the first spherical image by a specified angle to obtain a rotated first spherical image and the rotation of the second spherical image by the specified angle to obtain a rotated second spherical image, the coordinate system establishing unit establishes a spherical coordinate system. The spherical coordinate system is a three-dimensional rectangular coordinate system, the origin of the spherical coordinate system is the intersection of an optical center connecting line and the first plane, the optical center connecting line is a connecting line between the optical center of the first fisheye lens and the optical center of the second fisheye lens, the first axis of the spherical coordinate system is perpendicular to the first plane, the second axis of the spherical coordinate system is perpendicular to the first axis on the first plane, the direction of the second axis is vertically upward, and the third axis of the spherical coordinate system is perpendicular to the second plane, the second plane being the plane on which the first axis and the second axis are located. Correspondingly, the third acquisition unit is configured to, when performing the rotation of the first spherical image by a specified angle to obtain a rotated first spherical image and the rotation of the second spherical image by the specified angle to obtain a rotated second spherical image: rotate the first spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated first spherical image, and rotate the second spherical image by 90 degrees around the third axis in a specified direction to obtain a rotated second spherical image.

9. A panoramic camera comprising a first fisheye lens, a second fisheye lens, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images