Multi-view-angle-based stage fusion panoramic look-around image generation method and device

Abstract

The embodiment of the invention provides a multi-view-angle-based stage fusion panoramic look-around image generation method and device, equipment and a computer readable storage medium. The method comprises the following steps: acquiring a target image through a target fisheye camera; constructing a multi-view spherical projection model according to the target image; and according to the spherical projection model, generating a target visual angle panoramic spliced image. In this way, the problems of geometric distortion, obvious edge seams, inconsistent brightness, texture information loss and the like existing in a traditional double-fisheye image splicing method are solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular to a method, apparatus, device, and computer-readable storage medium for generating a panoramic surround view image based on multi-perspective stage fusion. Background Art

[0002] Currently, panoramic image generation and stitching technology has become a core support tool for intelligent construction monitoring, virtual inspection, and mobile sensing. This is especially true in enclosed or semi-enclosed spaces (such as tunnels, underground projects, and mines). Conventional flat cameras offer limited viewing angles, making them difficult to meet the construction site's demand for wide-area, real-time perception. Consequently, the industry is increasingly turning to dual-fisheye camera panoramic stitching solutions. These solutions utilize two back-to-back 240° fisheye cameras to generate 360° panoramic images or videos.

[0003] Although the conventional dual-fisheye image stitching technology currently used in the market can achieve 360° image stitching, it still has the following problems: 1. Large geometric errors and obvious feature distortion; 2. The splicing edges are obvious and the lighting is inconsistent; 3. Texture details are lost and high-frequency information cannot be retained; 4. The algorithm is not suitable for embedded deployment, and the contradiction between power consumption and performance is prominent. Summary of the Invention

[0004] According to an embodiment of the present application, a multi-perspective, stage-fused panoramic surround image generation scheme is provided. By establishing a spherical projection model based on Lie algebraic transformation, it can efficiently handle the data fusion problem of non-concentric multiple cameras and support the collaborative operation of more cameras in a unified coordinate system. The image stitching method disclosed in this disclosure effectively eliminates the problem of sudden brightness changes at the image stitching edges, greatly improving the ability to preserve edge texture details. The resulting panoramic image has a high-contrast, highly natural visual effect. In other words, it solves the problems of geometric distortion, obvious edge seams, inconsistent brightness, and missing texture information that exist in traditional dual-fisheye image stitching methods.

[0005] In a first aspect of the present application, a method for generating a panoramic surround view image based on multi-perspective stage fusion is provided. The method comprises: Obtain target images through the target fisheye camera; Constructing a multi-view spherical projection model according to the target image; A target-viewing angle panoramic stitching image is generated according to the spherical projection model.

[0006] Furthermore, constructing a multi-perspective spherical projection model based on the target image includes: Project the pixel coordinates in the target image onto the target sphere through the following beam projection model and the preset beam collinearity constraint, and construct a multi-view spherical projection model; Among them, the beam projection model includes: ; Among them, is the target spherical coordinate; is the scaling factor; is the rotation matrix of the camera; is the internal parameter transformation of the camera; is the displacement of the camera.

[0007] Furthermore, the beam collinearity constraint includes: ; Among them, is the scale factor; is the Lie algebra transformation; is the global position of the map point.

[0008] Furthermore, generating the target-view panoramic stitching image according to the spherical projection model includes: Extract feature points from the spherical projection model to obtain key point descriptors adapted to the spherical geometric structure; Based on the key point descriptors, perform geometric consistency screening through a preset algorithm to obtain a feature matching homography matrix; Perform image fusion based on the feature matching homography matrix to generate a target-view panoramic stitching image.

[0009] Furthermore, the extracting feature points from the spherical projection model to obtain key point descriptors adapted to the spherical geometric structure includes: Within the spherical neighborhood window of the spherical projection model, perform weighted statistics on the gray gradient direction to obtain the main direction of the feature points; According to the main direction, perform point pair sampling and brightness comparison on pixel points in the following manner to obtain key point descriptors adapted to the spherical geometric structure: ; where p is the pixel point being processed; and are two-dimensional offset vectors; is the image gray function.

[0010] Furthermore, the step of performing geometric consistency screening based on the key point descriptor through a preset algorithm to obtain a feature matching homography matrix includes: Processing the key point descriptors by means of an XOR operation to obtain a target set; The target set is screened for geometric consistency in the following manner to construct a feature matching homography matrix: ; Wherein, argHmin is the independent variable for calculating the minimum value of the function; pk and pk' are the coordinates of the corresponding feature points in the image respectively.

[0011] Furthermore, performing image fusion based on the feature matching homography matrix to generate a target perspective panoramic stitched image includes: The following formula is used to perform image fusion and generate a panoramic stitching image of the target perspective: ; Among them, Ω is the splicing area; is the area element in the two-dimensional integral; v is the synthesized target image; ; Where f is the gradient field of the target image; H is the feature matching homography matrix.

[0012] In a second aspect of the present application, a device for generating a panoramic surround view image based on multi-perspective stage fusion is provided. The device comprises: An acquisition module is used to acquire a target image through a target fisheye camera; A construction module, configured to construct a multi-view spherical projection model based on the target image; A generation module is used to generate a target perspective panoramic stitching image based on the spherical projection model.

[0013] In a third aspect of the present application, an electronic device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the program.

[0014] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present application is implemented.

[0015] The embodiment of the present application provides a method for generating panoramic surround view images based on multi-perspective stage fusion. A target image is acquired through a target fisheye camera. A multi-perspective spherical projection model is constructed based on the target image. A target-perspective panoramic stitched image is generated based on the spherical projection model. This method solves the problems of geometric distortion, obvious edge seams, inconsistent brightness, and missing texture information existing in traditional dual-fisheye image stitching methods.

[0016] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of the present application, nor are they intended to limit the scope of the present application. Other features of the present application will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein: Figure 1 Flowchart of a method for generating a panoramic surround view image based on multi-view stage fusion according to an embodiment of the present application; Figure 2 Schematic diagram of a multi-view spherical projection model according to an embodiment of the present application; Figure 3 Schematic diagram of a feature extraction structure according to an embodiment of the present application; Figure 4 is a schematic diagram of the fusion principle according to an embodiment of the present application; Figure 5 4 is a block diagram of a device for generating a panoramic surround view image based on multi-view stage fusion according to an embodiment of the present application; Figure 6 A schematic diagram of the structure of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0018] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0019] In addition, the term "and / or" in this text merely describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this text generally represents an "or" relationship between the front and rear associated objects.

[0020] Figure 1 The flowchart of a multi-view stage fusion panoramic surround image generation method according to an embodiment of the present disclosure is shown. The method includes: S110, obtaining a target image through a target fisheye camera.

[0021] In some embodiments, the target image, that is, the image to be processed, can be obtained through two preset fisheye cameras.

[0022] S120, constructing a multi-view spherical projection model according to the target image.

[0023] In some embodiments, to solve the distortion problem caused by the non-concentric design of the fisheye camera ignored by the traditional stitching method. In the present disclosure, a method of constructing a spherical projection model is adopted to solve the problem that the imaging centers S of different cameras are inconsistent.

[0024] The multi-lens combined panoramic camera consists of a series of independent and fixed fisheye lenses. Each lens has an independent projection center, but due to manufacturing tolerances, it does not exactly coincide with the theoretical center S of the sphere. Therefore, when performing panoramic stitching, the pixel coordinates need to be projected onto a sphere with a specified radius to obtain unified panoramic coordinates. To avoid the positioning error caused by non-concentricity, global coordinate transformation can be adopted. By calibrating the rotation matrix and position vector of a single camera in the panoramic camera coordinate system, the spatial alignment of multi-view projection can be achieved.

[0025] As Figure 2 shown, to unify the projection relationship, the following beam projection model can be adopted for coordinate transformation: ; where is the target spherical coordinate; is the scaling factor; is the rotation matrix of the camera; is the internal parameter transformation of the camera; is the displacement of the camera.

[0026] Furthermore, in practical applications, the pixel coordinates need to be projected onto a sphere with a radius r: ; Therefore, the following optimization strategy can be used to calculate the beam collinearity constraint: ; in, is the scale factor; is the Lie algebraic transformation; The global position of the map point (the coordinates of the three-dimensional point in the environment in the world coordinate system).

[0027] Furthermore, spherical coordinates can be expressed in polar coordinates: ; ; ; in, and are the polar angles in the horizontal and vertical directions respectively: ; Where W and H are the width and height of the panoramic image respectively.

[0028] In summary, the first step in generating panoramic surround images in the present disclosure is to establish corresponding spatial models for fisheye cameras with multiple perspectives. This not only optimizes the spatial coordinate conversion of dual fisheye cameras, but also, when the system is further expanded to more fisheyes, the above model can still achieve high-quality multi-perspective panoramic surround image generation based on the spatial information of different fisheyes.

[0029] S130: Generate a target-viewing-angle panoramic stitching image according to the spherical projection model.

[0030] In some embodiments, after establishing a multi-view spherical projection model, feature point extraction is performed on the spherical coordinates to extract key point descriptors that adapt to the spherical geometric structure.

[0031] Specifically, if Figure 3 As shown in Figure 2, the grayscale gradient direction is weighted statistically analyzed within the spherical neighborhood window to obtain the main direction θ of the feature point: ; in, is the weighted gradient direction angle; is the Gaussian weight; and Pixel Grayscale gradient component in spherical coordinates; Furthermore, under the guidance of the main direction, point - to - point sampling is performed on the pixel point p and brightness comparison is carried out to extract the key - point descriptor adapted to the spherical geometric structure: ; where p is the pixel point to be processed; and are two - dimensional offset vectors; is the image grayscale function, which returns the pixel intensity value (range [0, 255]) at the coordinate point; Based on the descriptor sets {SPHORB(p1)} and {SPHORB(p2)} obtained from two fisheye cameras respectively, the Hamming distance is calculated through XOR operation, and the point pair with the smallest distance is selected as the initial match, and the matching result is output: ; where pk and pk' are the corresponding feature - point coordinates in the two images respectively.

[0032] Furthermore, the homography matrix H is solved using the matching point pairs M, and geometric constraints are established for geometric consistency screening. Four pairs of matching points are randomly selected from M (H has 8 degrees of freedom), a linear equation system is solved using the selected point pairs, a temporary H is calculated, and the H with the largest number of inliers is selected as the final H, and the matrix parameters are optimized using all inliers through the least - squares method. The calculation method of the homography matrix H is as follows: ; where argHmin is the independent variable that calculates the minimum value of the function; pk and pk' are the corresponding feature - point coordinates in the images respectively; The homography matrix H constructed after fitting between the images is as follows: ; ; ; where h11, h12, h21, h22 are affine - transformation parameters used to control rotation, scaling, and shear; h13, h23 are translation parameters representing the displacement amounts in the x and y - axis directions respectively; h31, h32 are perspective - transformation parameters used to describe the deformation in the depth direction.

[0033] This matrix H represents the projection of the fisheye image from spherical coordinates to a unified stitching - plane coordinate system, which is used to complete the geometric registration between fisheye images, eliminate false matches, ensure spatial consistency, and provide an important basis for subsequent image fusion.

[0034] In some embodiments, during the process of stitching images, in order to make the synthesized image more natural, the synthetic boundary needs to maintain a seamless connection. However, if the original image and the target image have significantly different texture features, there will be obvious boundaries in the directly synthesized image.

[0035] In response to the above problems, the following idea is proposed in the present disclosure, as Figure 4 shown. According to the gradient information of the source image and the boundary information of the target image, the image pixels in the synthesis region are reconstructed by using the interpolation method to achieve continuity in the gradient domain, thereby achieving seamless fusion at the boundary. That is, to ensure that the gradient field in the stitching region is as consistent as possible in space. The goal is to minimize the difference between the newly synthesized image v and the original image f in the gradient space within the stitching region Ω, thereby eliminating the seam problem. The optimization objective function is as follows: Inverse transform the gradient of the source image and map it to the target image coordinate system: ; ; where f is the gradient field of the target image; H is the feature matching homography matrix; Minimize the gradient difference within the region Ω aligned by H: ; where Ω is the stitching region; is the area element in the two-dimensional integral; v is the synthesized target image; By the above method, the optimal pixel values are solved within the stitching region, making the gradient at the boundary smooth. It has a good effect in eliminating color inconsistency problems and edge seams. For the possible image texture loss problem (mainly for low-frequency information processing) in the above fusion method, the following method can be used to optimize the above synthesized image.

[0036] Specifically, the Laplacian pyramid can be used to perform hierarchical processing on the image V, and weighted fusion is performed separately in the spaces of different frequency bands, thereby reducing the influence caused by illumination and color differences: ; where and are respectively the Laplacian pyramid decomposition layers of the input image v; and are respectively the weight matrices generated by the Gaussian pyramid, used to control the fusion intensity of different regions; ⊙ represents the element-wise multiplication operation; This approach allows for independent processing of information at different spatial scales. The low-frequency portion is used to match overall brightness and color, eliminating the effects of global illumination variations; the high-frequency portion is used to maintain edge clarity, ensuring no blur or texture distortion after stitching. This results in a more natural transition between stitched areas, significantly reducing the texture loss that can occur with traditional methods.

[0037] According to the embodiments of the present disclosure, the following technical effects are achieved: Compared with the traditional method of directly stitching images, in this application, the projection center of each camera is independently modeled. By constructing a unified spherical coordinate system, images from different perspectives can be aligned on the same panoramic sphere, improving stitching accuracy and spatial consistency. It is not only suitable for dual cameras, but can also be expanded to multi-camera systems, with good scalability.

[0038] The feature point extraction and matching method disclosed in this paper solves the problem that traditional stitching methods, which use general algorithms such as planar SIFT or SURF, are difficult to adapt to the severe distortion of fisheye images. It effectively reduces mismatched points and improves the stability and matching accuracy of image registration.

[0039] This solves the problem that traditional linear weighted fusion methods cannot simultaneously handle inconsistent lighting, preserve high-frequency textures, and produce noticeable stitching seams. It improves image quality by focusing on low-frequency brightness continuity and high-frequency texture detail preservation, significantly alleviating the image blurring caused by traditional weighted averaging methods.

[0040] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0041] The above is an introduction to the method embodiment. The following is a device embodiment to further illustrate the solution described in this application.

[0042] Figure 5 FIG. 5 shows a block diagram 500 of a device for generating a panoramic surround view image based on multi-view stage fusion according to an embodiment of the present application, as shown in FIG. Figure 5 Shown include: An acquisition module 510 is configured to acquire a target image through a target fisheye camera; A construction module 520 is configured to construct a multi-view spherical projection model based on the target image; A generation module 530, configured to generate a panoramic stitching image of a target view according to the spherical projection model.

[0043] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the described modules can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0044] Figure 6 The figure shows a schematic structural diagram of a terminal device or a server suitable for implementing the embodiments of the present application.

[0045] As Figure 6 shown, the terminal device or the server includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage section 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the terminal device or the server are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0046] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as required. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as required, so that a computer program read from it can be installed into the storage section 608 as required.

[0047] Specifically, according to the embodiments of the present application, the above method flow steps can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the system of the present application are executed.

[0048] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. And in this application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0049] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram can represent a module, a program segment, or a part of code, and the aforementioned module, program segment, or part of code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0050] The units or modules involved in the embodiments described in this application can be implemented in software or in hardware. The described units or modules can also be provided in a processor. Among them, the names of these units or modules do not, in some cases, constitute a limitation on the units or modules themselves.

[0051] As another aspect, this application also provides a computer-readable storage medium. This computer-readable storage medium can be included in the electronic device described in the above embodiments; it can also exist alone without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the foregoing programs are executed by one or more processors, the methods described in this application are implemented.

[0052] The above description is only a preferred embodiment of this application and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the application involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the foregoing inventive concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions described in this application.

Claims

1. A method for generating a stage-fused panoramic surround-view image based on multiple perspectives, characterized in that including: obtaining a target image through a target fisheye camera; constructing a multi-view spherical projection model according to the target image; generating a panoramic stitching image of a target view according to the spherical projection model.

2. The method according to claim 1, wherein The constructing a multi-view spherical projection model according to the target image includes: projecting the pixel coordinates in the target image onto a target sphere through the following beam projection model and a preset beam collinearity constraint to construct a multi-view spherical projection model; wherein, the beam projection model includes: ； Among them, is the target spherical coordinate; is the scaling factor; is the rotation matrix of the camera; is the internal parameter transformation of the camera; is the displacement of the camera.

3. The method according to claim 2, wherein the beam collinearity constraint includes: ； Among them, is the scale factor; is a Lie algebra transformation; is the global position of the map point.

4. The method according to claim 3, characterized in that, The generating a panoramic stitching image of a target view according to the spherical projection model includes: extracting feature points from the spherical projection model to obtain key point descriptors adapted to the spherical geometric structure; performing geometric consistency screening on the basis of the key point descriptors through a preset algorithm to obtain a feature matching homography matrix; performing image fusion based on the feature matching homography matrix to generate a panoramic stitching image of a target view.

5. The method according to claim 4, wherein The extracting feature points from the spherical projection model to obtain key point descriptors adapted to the spherical geometric structure includes: performing weighted statistics on the gray gradient direction within a spherical neighborhood window of the spherical projection model to obtain the main direction of the feature points; according to the main direction, performing point pair sampling and brightness comparison on pixel points in the following manner to obtain key point descriptors adapted to the spherical geometric structure: ； where p is the pixel point being processed; and is a two-dimensional offset vector; is the image grayscale function.

6. The method according to claim 4, characterized in that The performing geometric consistency screening on the basis of the key point descriptors through a preset algorithm to obtain a feature matching homography matrix includes: processing the key point descriptors in a way of exclusive OR operation to obtain a target set; performing geometric consistency screening on the target set in the following manner to construct a feature matching homography matrix: ； where argHmin is the independent variable that calculates the minimum value of the function; pk and pk' are the corresponding feature point coordinates in the image respectively.

7. The method according to claim 4, characterized in that, The performing image fusion based on the feature matching homography matrix to generate a panoramic stitching image of a target view includes: performing image fusion through the following formula to generate a panoramic stitching image of a target view: ； where Ω is the stitching area; v is the synthesized target image; is the area element in the two-dimensional integral; ； where f is the gradient field of the target image; H is the feature matching homography matrix.

8. A multi-view-based stage fusion panoramic surround image generation device, characterized in that including: an obtaining module, configured to obtain a target image through a target fisheye camera; a constructing module, configured to construct a multi-view spherical projection model according to the target image; a generating module, configured to generate a panoramic stitching image of a target view according to the spherical projection model.

9. An electronic device, comprising a memory and a processor, wherein a computer program is stored on the memory, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images