Image splicing method and device, terminal and computer readable storage medium

Through the panoramic camera array and three-dimensional model projection technology, the image overlap area and optimal suture lines were determined, and the texture deformation and ghosting problems in 360° panoramic video stitching were solved, achieving high-quality image stitching effect.

CN118941753APending Publication Date: 2024-11-12ZHEJIANG DAHUA TECH CO LTD

Patent Information

Application Number
CN202411381204.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, the image stitching effect is poor, especially in the 360° panoramic video stitching, and the target center video stitching is difficult to achieve effective constraints, and feature extraction and matching are difficult.

Method used

Multiple video frames are obtained by using a panoramic camera array to determine the image overlap area and optimal suture through three-dimensional model projection, perform stitching with the smallest pixel difference, and combine dynamic programming and beam adjustment to perform pose optimization and exposure correction to achieve image weighted fusion and feature matching.

Benefits of technology

Improve image stitching effect, reduce texture deformation and ghosting, and enhance visual consistency, especially in the case of sparse acquisition equipment, ensuring the quality of stitching images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941753A_ABST
    Figure CN118941753A_ABST
Patent Text Reader

Abstract

The invention provides an image splicing method and device, a terminal and a computer readable storage medium. The image splicing method comprises the following steps: acquiring a plurality of current video frames containing different parts of a target object and an observation viewpoint; selecting a current video frame collected by a collection device adjacent to the observation viewpoint from the plurality of current video frames as a first video frame and a second video frame; projecting the first video frame to a cylindrical surface of the three-dimensional model, and determining an image overlapping region of the first video frame and the second video frame on the three-dimensional model and an optimal suture line between the first video frame and the second video frame; and splicing the first video frame and the second video frame according to the optimal suture line to obtain a spliced image of the target object under the observation viewpoint. According to the method, the image overlapping region is directly determined on the three-dimensional model, and then the optimal suture line is searched in the image overlapping region, so that the image splicing effect of the current video frame acquired by the adjacent acquisition equipment containing the target object can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image stitching method, device, terminal and computer-readable storage medium. Background Art

[0002] Video stitching refers to stitching multiple synchronized video streams into a video with a wide angle and a large field of view. It is widely used in wide-area video acquisition, immersive virtual reality or mixed reality, etc. On this basis, 3D video needs to ensure that the viewing angle can be switched.

[0003] Currently, 360° panoramic videos with fixed viewpoints are developing rapidly. Through processes such as feature extraction and matching, camera parameter estimation, registration, and fusion, videos can be spliced ​​and mapped to a cylinder or sphere to meet viewing requirements from different viewpoints. However, due to the imaging principle of the camera, cylindrical or spherical mappings at different viewpoints often have texture deformation problems, and overlapping areas of cameras that are not aligned will have problems such as ghosting. Even if the mapping relationship in the splicing is accurate enough, visual defects will occur when moving objects pass through the seams or enter the overlapping areas of the field of view. On the other hand, due to the lack of effective constraints on the camera, video stitching at the center of the target is often more difficult in the aspects of feature extraction, matching, and registration. Summary of the invention

[0004] The main technical problem solved by the present invention is to provide an image stitching method, device, terminal and computer-readable storage medium to solve the problem of poor image stitching effect in the prior art.

[0005] In order to solve the above technical problems, the first technical solution adopted by the present invention is: to provide an image stitching method, the image stitching method comprising: Acquire multiple current video frames containing different parts of the target object and observation viewpoints, wherein the multiple current video frames are simultaneously acquired by a surround panoramic camera array composed of multiple acquisition devices surrounding the target object; Selecting a current video frame captured by a capture device adjacent to an observation viewpoint from a plurality of current video frames as a first video frame and a second video frame; the positions of the observation viewpoint and each capture device are different; Projecting the first video frame onto a cylindrical surface of a three-dimensional model, determining an image overlap region of the first video frame and the second video frame on the three-dimensional model and an optimal stitching line between the first video frame and the second video frame; the pixel difference on both sides of the optimal stitching line is minimal, and the optimal stitching line is in the image overlap region; The first video frame and the second video frame are spliced ​​together according to the optimal stitching line to obtain a spliced ​​image of the target object under the observation viewpoint.

[0006] The specific steps of determining the image overlapping area include: Projecting the first video frame onto a cylindrical surface of the three-dimensional model to obtain a first cylindrical projection image; Shooting the first cylindrical projection image from the position of the acquisition device for acquiring the second video frame to obtain a second cylindrical projection image; The second cylindrical projection image is matched with the second video frame to determine an image overlap area between the first video frame and the second video frame on the three-dimensional model.

[0007] The specific steps of determining the optimal stitching line between the first video frame and the second video frame include: Downsampling the local images in the first video frame and the second video frame corresponding to the image overlap area to obtain preprocessed images corresponding to the first video frame and the second video frame respectively; the resolution of the preprocessed images is lower than the resolution of the first video frame and the second video frame; Using dynamic programming or graph cut method to search in the preprocessed images corresponding to the first video frame and the second video frame respectively, to determine the optimal stitching line between the preprocessed image of the first video frame and the preprocessed image of the second video frame; An optimal stitching line between a preprocessed image of the first video frame and a preprocessed image of the second video frame is upsampled to determine an optimal stitching line between the first video frame and the second video frame.

[0008] The image stitching method further includes: A transition area is selected in the image overlapping area; the transition area contains the optimal stitching line, and the transition area does not exceed the image overlapping area; The pixel value of each pixel point in the transition area corresponding to the first video frame is weightedly fused with the pixel value of the pixel point in the second video frame to obtain a preprocessed spliced ​​image.

[0009] The image stitching method further includes: Extract features of the local area of ​​the image overlap region corresponding to the first video frame and the local area of ​​the second video frame to obtain regional feature maps corresponding to the first video frame and the second video frame respectively; Performing feature matching on the region feature maps corresponding to the first video frame and the second video frame respectively corresponding to the image overlapping region, to obtain a matching relationship between the region feature map of the first video frame and the region feature map of the second video frame; The bundle adjustment method is used to optimize the position relationship between the acquisition devices in the surround view panoramic camera array and the mapping relationship between the acquisition devices and the three-dimensional model based on the matching relationship between the regional feature map of the first video frame and the regional feature map of the second video frame.

[0010] The image stitching method further includes: Collect images in advance through each acquisition device in the panoramic camera array to obtain sample images; Determining an exposure compensation value of the surround view panoramic camera array based on a gain coefficient of each acquisition device and the illumination intensity of the sample images respectively acquired; The exposure parameters of each acquisition device in the surround view panoramic camera array are corrected based on the exposure compensation value.

[0011] Wherein, based on the gain coefficient of each acquisition device and the illumination intensity of the sample images respectively acquired, the exposure compensation value of the surround view panoramic camera array is determined, including: Determine brightness data of the acquisition device based on the product of the gain coefficient of the acquisition device and the light intensity of the acquired sample image; Determine a brightness error value between adjacent acquisition devices based on a difference in brightness data of adjacent acquisition devices in the surround view panoramic camera array; The minimum brightness error value is selected as the exposure compensation value of the surround view panoramic camera array.

[0012] The image stitching method further includes: Compare the current video frame with the historical video frame to determine whether the current video frame contains a moving target; the moving target is located between the acquisition device and the target object; the historical video frame is an image containing the target object that was acquired by the acquisition device before the current video frame was acquired; In response to detecting a moving target in the current video frame, extracting the moving target and projecting the moving target onto a cylindrical surface of the three-dimensional model; In response to the existence of a blank area on the cylindrical surface of the three-dimensional model, the blank area is completed based on a historical video frame that does not contain a moving target.

[0013] The image stitching method further includes: In response to the moving target projected onto the three-dimensional model passing through the optimal stitching line, the optimal stitching line is updated based on a current video frame containing the moving target captured by an adjacent acquisition device.

[0014] The image stitching method further includes: Temporal smoothing is performed on the spliced ​​images corresponding to the current video frames of a preset number of consecutive frames containing the moving target.

[0015] In order to solve the above technical problems, the second technical solution adopted by the present invention is: to provide an image stitching device, the image stitching device comprising: An acquisition module is used to acquire multiple current video frames containing different parts of the target object and observation viewpoints, where the multiple current video frames are simultaneously acquired by a surround panoramic camera array composed of multiple acquisition devices surrounding the target object; A selection module is used to select a current video frame captured by a capture device adjacent to an observation viewpoint from multiple current video frames as a first video frame and a second video frame; the position between the observation viewpoint and each capture device is different; A processing module, used for projecting the first video frame onto a cylindrical surface of the three-dimensional model, determining an image overlap region of the first video frame and the second video frame on the three-dimensional model and an optimal stitching line between the first video frame and the second video frame; the pixel difference on both sides of the optimal stitching line is the smallest, and the optimal stitching line is in the image overlap region; The stitching module is used to stitch the first video frame and the second video frame according to the optimal stitching line to obtain a stitched image of the target object under the observation viewpoint.

[0016] To solve the above technical problems, the third technical solution adopted by the present invention is: to provide a terminal, the terminal includes a memory, a processor and a computer program stored in the memory and running on the processor, the processor is used to execute program data to implement the steps in the above-mentioned image stitching method.

[0017] In order to solve the above technical problems, the fourth technical solution adopted by the present invention is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned image stitching method are implemented.

[0018] The beneficial effects of the present invention are as follows: different from the prior art, an image stitching method, device, terminal and computer-readable storage medium are provided, wherein the image stitching method comprises acquiring multiple current video frames and observation viewpoints containing different parts of a target object, wherein the multiple current video frames are simultaneously acquired by a surround panoramic camera array composed of multiple acquisition devices surrounding the target object; selecting current video frames acquired by an acquisition device adjacent to the observation viewpoint from the multiple current video frames as first video frames and second video frames; the positions of the observation viewpoint and each acquisition device are different; projecting the first video frame onto a cylinder of a three-dimensional model, determining an image overlap area of ​​the first video frame and the second video frame on the three-dimensional model and an optimal stitching line between the first video frame and the second video frame; the pixel difference on both sides of the optimal stitching line is the smallest, and the optimal stitching line is in the image overlap area; stitching the first video frame and the second video frame according to the optimal stitching line to obtain a stitched image of the target object under the observation viewpoint. The present application helps to improve the image stitching effect of the current video frames captured by adjacent acquisition devices containing the target object by projecting the first video frame captured by the acquisition device adjacent to the observation viewpoint onto the cylinder of the three-dimensional model, directly determining the image overlapping area on the three-dimensional model, and then searching for the optimal stitching line in the image overlapping area. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is a flowchart of the image stitching method provided by the present invention; Figure 2 It is a flowchart of a specific embodiment of the image stitching method provided by the present invention; Figure 3 It is a schematic diagram of the framework of an embodiment of an image stitching device provided by the present invention; Figure 4 It is a schematic diagram of the framework of a terminal embodiment provided by the present invention; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0021] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0023] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0024] In order to enable those skilled in the art to better understand the technical solution of the present invention, an image stitching method provided by the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0026] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0027] The image stitching method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. In some embodiments, the terminal or server can implement the image stitching method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports virtual scenes, such as a game APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0028] The image stitching method provided in the embodiment of the present application is explained below using a server implementation as an example.

[0029] See also Figure 1 , Figure 1 It is a flowchart of the image stitching method provided by the present invention.

[0030] This embodiment provides an image stitching method, which includes the following steps.

[0031] S1: Acquire multiple current video frames containing different parts of the target object and observation viewpoints, where the multiple current video frames are simultaneously acquired by a surround panoramic camera array composed of multiple acquisition devices surrounding the target object.

[0032] S2: Selecting a current video frame captured by a capture device adjacent to the observation viewpoint from multiple current video frames as the first video frame and the second video frame; the positions of the observation viewpoint and each capture device are different.

[0033] S3: Projecting the first video frame onto a cylindrical surface of the three-dimensional model, determining an image overlap area between the first video frame and the second video frame on the three-dimensional model and an optimal stitching line between the first video frame and the second video frame; the pixel difference on both sides of the optimal stitching line is the smallest, and the optimal stitching line is in the image overlap area.

[0034] S4: stitching the first video frame and the second video frame according to the optimal stitching line to obtain a stitched image of the target object under the observation viewpoint.

[0035] The technical solution provided in this embodiment is applicable to relatively sparse acquisition devices, that is, the surround-view panoramic camera array includes fewer acquisition devices, and the acquisition devices arranged around the target object are sparse.

[0036] In one embodiment, in order to reduce the amount of calculation and improve the image stitching effect, the parameters of the acquisition device may be calibrated in advance.

[0037] See also Figure 2 , Figure 2 It is a flowchart of a specific embodiment of the image stitching method provided by the present invention.

[0038] S10: Preliminarily collect images through each collection device in the panoramic camera array to obtain sample images.

[0039] S20: Determine an exposure compensation value of the surround view panoramic camera array based on the gain coefficient of each acquisition device and the illumination intensity of the sample images respectively acquired.

[0040] In a specific embodiment, the brightness data of the acquisition device is determined based on the product of the gain coefficient of the acquisition device and the light intensity of the sample image acquired; the brightness error value between adjacent acquisition devices in the surround-view panoramic camera array is determined based on the difference in brightness data of adjacent acquisition devices; and the minimum brightness error value is selected as the exposure compensation value of the surround-view panoramic camera array. That is, the exposure compensation value of the surround-view panoramic camera array is . Among them, ij is the number of adjacent acquisition devices, is the gain coefficient corresponding to acquisition device i, is the light intensity corresponding to the acquisition device i.

[0041] S30: Correcting exposure parameters of each acquisition device in the surround view panoramic camera array based on the exposure compensation value.

[0042] In one embodiment, feature extraction is performed on the image overlapping area to obtain a regional feature map of the first video frame in the image overlapping area and a regional feature map of the second video frame in the image overlapping area. Among them, SuperPoint, SIFT and other methods can be used to extract features from the image to obtain a feature map. Feature matching is performed on the regional feature map of the first target image in the image overlapping area and the regional feature map of the second target image in the image overlapping area. The bundle adjustment method is used to optimize the position relationship between the acquisition devices in the surround panoramic camera array and the mapping relationship between the acquisition devices and the three-dimensional model based on the matching relationship between the regional feature map of the first target image in the image overlapping area and the regional feature map of the second target image in the image overlapping area.

[0043] Specifically, the specific implementation method of obtaining multiple current video frames containing different parts of the target object and observation viewpoints in step 1 is as follows.

[0044] In one embodiment, with the target object as the center, multiple acquisition devices are deployed at different positions around the target object. Multiple acquisition devices form a surround-view panoramic camera array. In this embodiment, the number of acquisition devices can be three or four, which can be set according to actual conditions. The acquisition devices in the surround-view panoramic camera array are evenly arranged around the target object to ensure that any point on the target object is visible in at least one acquisition device. Each acquisition device is fixedly set, and the position of each acquisition device is fixed relative to the target object. The target object in this embodiment can be a building, a vehicle, an animal, a plant, etc. The acquisition device can be a fisheye camera, a SLR camera, etc.

[0045] In one embodiment, the observation viewpoint and the acquisition device are all around the target object, so that the user can observe part of the target object from the observation viewpoint. The current view cone of the observation viewpoint is determined based on the position of the observation viewpoint and information such as the pitch angle, yaw angle, and roll angle, and the visible surface of the three-dimensional model visible within the view cone is calculated, and only the visible part of the three-dimensional model is retained. The viewing angle of the observation viewpoint can be a preset angle, which can be 60°, 360°, 120°, 180°, 90°, etc., and can be set according to actual conditions.

[0046] In one embodiment, structured light scanning and multiple views are used to reconstruct the three-dimensional model. Figure 3 Dimensional reconstruction usually estimates the pose of the acquisition device during reconstruction, while methods such as structured light scanning require additional calibration of internal and external parameters to obtain the transformation matrix from the model coordinate system to the pixel coordinate system.

[0047] Specifically, the specific implementation method of determining the image overlapping area in step S3 is as follows.

[0048] Since there are fewer acquisition devices set up around the target object and the visual span of each acquisition device is larger, the overlapping areas between the current video frames captured by adjacent acquisition devices are smaller. If the two-dimensional current video frames captured by the acquisition devices are directly matched, the accuracy of the obtained image overlapping areas is low.

[0049] In one embodiment, the first video frame is projected onto a cylinder of a three-dimensional model to obtain a first cylindrical projection image; the first cylindrical projection image is photographed from the position of a capture device for obtaining a second video frame to obtain a second cylindrical projection image; the second cylindrical projection image is matched with the second video frame to determine an image overlap area between the first video frame and the second video frame on the three-dimensional model. By photographing a projection image of the first video frame on the three-dimensional model from the position of a capture device for obtaining the second video frame, and then performing image matching between the second video frame and the photographed image, the accuracy of the image overlap area corresponding to the first video frame and the second video frame can be improved, so as to improve the image stitching effect.

[0050] Specifically, a specific implementation of determining the optimal stitching line between the first video frame and the second video frame in step S3 is as follows.

[0051] In one embodiment, in order to reduce the amount of calculation, the local images corresponding to the image overlap area in the first video frame and the second video frame can be downsampled to obtain preprocessed images corresponding to the first video frame and the second video frame respectively; the resolution of the preprocessed images is lower than the resolution of the first video frame and the second video frame. Dynamic programming or graph cut method is used to search in the preprocessed images corresponding to the first video frame and the second video frame respectively to determine the optimal stitching line between the preprocessed image of the first video frame and the preprocessed image of the second video frame. The optimal stitching line between the preprocessed image of the first video frame and the preprocessed image of the second video frame is upsampled to determine the optimal stitching line between the first video frame and the second video frame.

[0052] In one embodiment, in order to reduce the amount of calculation and take into account the stitching effect, a transition area is selected in the image overlapping area; the transition area contains the optimal stitching line, and the transition area does not exceed the image overlapping area; the pixel value corresponding to each pixel point in the transition area in the first video frame and the pixel value corresponding to the second video frame are weighted fused to obtain a preprocessed stitched image. Among them, the weight of the pixel point closer to the optimal stitching line in the transition area is smaller, so as to ensure that the pixel value of the fused pixel point is smoother. Methods such as Laplace pyramid fusion can also be used to obtain a better visual effect for the preprocessed stitched image. The width of the transition area is determined by the distribution of the image overlapping area and the position of the optimal stitching line.

[0053] The boundaries of the transition regions and the optimal seam lines are mapped to the 3D model.

[0054] In one embodiment, the current video frame is compared with the historical video frame to determine whether the current video frame contains a moving target; the moving target is located between the acquisition device and the target object; the historical video frame is an image containing the target object captured by the acquisition device before the current video frame.

[0055] In a specific embodiment, a low-resolution optical flow is used to detect whether there is a moving target in the current video frame. Alternatively, a frame difference method or a background modeling method may be used to detect whether there is a moving target in the current video frame. Specifically, if the current video frame contains a vehicle but the historical video frame does not contain the vehicle, the vehicle is regarded as the moving target in the current video frame.

[0056] In response to no moving target being detected in the current video frame, the current video frame captured by the corresponding acquisition device is stitched through an optimal stitching line on the three-dimensional model to obtain a spliced ​​image.

[0057] In response to detecting a moving target in the current video frame, the moving target is extracted and projected onto a cylindrical surface of the three-dimensional model.

[0058] In response to the existence of a blank area on the cylindrical surface of the three-dimensional model, the blank area is completed based on a historical video frame that does not contain a moving target.

[0059] In one embodiment, the area within the current field of view at the observation viewpoint where the projection information of the acquisition device cannot be obtained due to occlusion is filled based on the projection of the adjacent current video frame to ensure visual consistency.

[0060] In one embodiment, in response to a moving target projected onto the three-dimensional model passing through an optimal stitching line, the optimal stitching line is updated based on a current video frame containing the moving target captured by an adjacent acquisition device.

[0061] In this embodiment, since the optimal stitching line is determined on the three-dimensional model, the optimization function of the optimal stitching line also includes depth consistency to ensure that the stitching avoids areas with significant depth changes. Consistent with initialization, the optimal stitching line is calculated at a low resolution.

[0062] In one embodiment, temporal smoothing is performed on a spliced ​​image corresponding to a current video frame of a preset number of consecutive frames containing a moving target.

[0063] In this embodiment, on the basis of dynamically calculating the optimal stitching seam, time sequence smoothing is performed at the same time to ensure that the visual effect of the moving target before and after passing through the stitching seam is coherent, eliminate the local mutation caused by the dynamic object passing through the seam, and ensure that there is no obvious ghosting or deformation. In addition to improving the smoothness of the dynamic foreground sequence, the dynamic stitching seam can also reduce the artifact problem caused by some time synchronization errors on the dynamic target.

[0064] In this embodiment, by projecting the first video frame captured by the acquisition device adjacent to the observation viewpoint onto the cylinder of the three-dimensional model, by directly determining the image overlapping area on the three-dimensional model, and then searching for the optimal stitching line in the image overlapping area, it helps to improve the image stitching effect of the current video frame captured by the adjacent acquisition device that contains the target object.

[0065] See also Figure 3 , Figure 3 Schematic diagram of the framework of an embodiment of the image stitching device provided by the present invention. This embodiment provides an image stitching device 60, which includes an acquisition module 61, a selection module 62, a processing module 63 and a stitching module 64.

[0066] The acquisition module 61 is used to acquire multiple current video frames containing different parts of the target object and observation viewpoints. The multiple current video frames are simultaneously acquired by a surround panoramic camera array composed of multiple acquisition devices surrounding the target object.

[0067] The selection module 62 is used to select the current video frame captured by the capture device adjacent to the observation viewpoint from multiple current video frames as the first video frame and the second video frame; the position between the observation viewpoint and each capture device is different.

[0068] The processing module 63 is used to project the first video frame onto the cylinder of the three-dimensional model, determine the image overlapping area of ​​the first video frame and the second video frame on the three-dimensional model and the optimal stitching line between the first video frame and the second video frame; the pixel difference on both sides of the optimal stitching line is the smallest, and the optimal stitching line is in the image overlapping area.

[0069] The stitching module 64 is used to stitch the first video frame and the second video frame according to the optimal stitching line to obtain a stitched image of the target object under the observation viewpoint.

[0070] The image stitching device provided in this embodiment helps to improve the image stitching effect of the current video frame captured by the adjacent acquisition device containing the target object by projecting the first video frame captured by the acquisition device adjacent to the observation viewpoint onto the cylinder of the three-dimensional model, directly determining the image overlapping area on the three-dimensional model, and then searching for the optimal stitching line in the image overlapping area.

[0071] See also Figure 4 , Figure 4 80 is a schematic diagram of a terminal embodiment provided by the present invention. The terminal 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-mentioned image stitching method embodiments. In a specific implementation scenario, the terminal 80 may include but is not limited to: a microcomputer, a server, and in addition, the terminal 80 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0072] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps of any of the above-mentioned image stitching method embodiments. The processor 82 can also be called a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0073] See also Figure 5 , Figure 5 The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor, and the program instructions 901 are used to implement the steps of any of the above-mentioned image stitching method embodiments.

[0074] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0075] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0076] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0077] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0079] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image stitching method, characterized in that: The image stitching method comprises: Acquire multiple current video frames and observation viewpoints containing different parts of the target object, wherein the multiple current video frames are simultaneously acquired by a surround panoramic camera array composed of multiple acquisition devices surrounding the target object; Selecting the current video frames captured by the capture device adjacent to the observation viewpoint from the multiple current video frames as the first video frame and the second video frame; the positions of the observation viewpoint and each of the capture devices are different; Projecting the first video frame onto a cylindrical surface of a three-dimensional model, determining an image overlap region of the first video frame and the second video frame on the three-dimensional model and an optimal stitching line between the first video frame and the second video frame; the pixel difference on both sides of the optimal stitching line is minimal, and the optimal stitching line is located in the image overlap region; The first video frame and the second video frame are spliced ​​together according to the optimal stitching line to obtain a spliced ​​image of the target object at the observation viewpoint.

2. The image stitching method according to claim 1, characterized in that: The specific steps of determining the image overlapping area include: Projecting the first video frame onto a cylindrical surface of the three-dimensional model to obtain a first cylindrical projection image; Shooting the first cylindrical projection image from the position of the acquisition device from which the second video frame is acquired to obtain a second cylindrical projection image; The second cylindrical projection image is matched with the second video frame to determine the image overlap area of ​​the first video frame and the second video frame on the three-dimensional model.

3. The image stitching method according to claim 1 or 2, characterized in that: The specific steps of determining the optimal stitching line between the first video frame and the second video frame include: Downsampling the local images in the first video frame and the second video frame corresponding to the image overlap area to obtain preprocessed images corresponding to the first video frame and the second video frame respectively; the resolution of the preprocessed images is lower than the resolution of the first video frame and the second video frame; Using dynamic programming or graph cut method to search in the preprocessed images corresponding to the first video frame and the second video frame respectively, to determine the optimal stitching line between the preprocessed image of the first video frame and the preprocessed image of the second video frame; An upsampling process is performed on the optimal stitching line between the preprocessed image of the first video frame and the preprocessed image of the second video frame to determine an optimal stitching line between the first video frame and the second video frame.

4. The image stitching method according to claim 1, characterized in that: The image stitching method further comprises: Selecting a transition area in the image overlapping area; the transition area includes the optimal stitching line, and the transition area does not exceed the image overlapping area; The pixel value corresponding to each pixel point in the transition area in the first video frame is weightedly fused with the pixel value corresponding to the second video frame to obtain a preprocessed spliced ​​image.

5. The image stitching method according to claim 1, characterized in that: The image stitching method further comprises: Extract features of the local area of ​​the image overlap area corresponding to the first video frame and the local area of ​​the image overlap area corresponding to the second video frame to obtain regional feature maps corresponding to the first video frame and the second video frame respectively; Performing feature matching on the region feature maps corresponding to the first video frame and the second video frame respectively corresponding to the image overlapping region to obtain a matching relationship between the region feature map of the first video frame and the region feature map of the second video frame; The bundle adjustment method is used to optimize the position relationship between the acquisition devices in the surround view panoramic camera array and the mapping relationship between the acquisition devices and the three-dimensional model based on the matching relationship between the regional feature map of the first video frame and the regional feature map of the second video frame.

6. The image stitching method according to claim 1, characterized in that: The image stitching method further comprises: Collecting images in advance by each of the acquisition devices in the surround-view panoramic camera array to obtain sample images; Determining an exposure compensation value of the surround-view panoramic camera array based on a gain coefficient of each of the acquisition devices and the illumination intensity of the sample images respectively acquired; The exposure parameters of each of the acquisition devices in the surround-view panoramic camera array are corrected based on the exposure compensation value.

7. The image stitching method according to claim 6, characterized in that: The step of determining the exposure compensation value of the surround-view panoramic camera array based on the gain coefficient of each of the acquisition devices and the illumination intensity of the sample images respectively acquired includes: Determining brightness data of the acquisition device based on the product of the gain coefficient of the acquisition device and the light intensity of the acquired sample image; Determining a brightness error value between adjacent acquisition devices in the surround view panoramic camera array based on a difference in brightness data of adjacent acquisition devices; The smallest brightness error value is selected as the exposure compensation value of the surround view panoramic camera array.

8. The image stitching method according to claim 1, characterized in that: The image stitching method further comprises: Compare the current video frame with the historical video frame to determine whether the current video frame contains a moving target; the moving target is located between the acquisition device and the target object; the historical video frame is an image containing the target object that was acquired by the acquisition device before the current video frame was acquired; In response to detecting the moving target in the current video frame, extracting the moving target and projecting the moving target onto a cylindrical surface of the three-dimensional model; In response to the presence of a blank area on the cylindrical surface of the three-dimensional model, the blank area is completed based on a historical video frame that does not include the moving target.

9. The image stitching method according to claim 8, characterized in that: The image stitching method further comprises: In response to the moving target projected onto the three-dimensional model passing through the optimal stitching line, the optimal stitching line is updated based on a current video frame captured by the adjacent acquisition device and containing the moving target.

10. The image stitching method according to claim 9, characterized in that: The image stitching method further comprises: Temporal smoothing is performed on the spliced ​​image corresponding to the current video frame containing the moving target in a continuous preset number of frames.

11. An image stitching device, characterized in that: The image stitching device comprises: An acquisition module, used to acquire multiple current video frames and observation viewpoints containing different parts of a target object, wherein the multiple current video frames are simultaneously acquired by a surround panoramic camera array composed of multiple acquisition devices surrounding the target object; A selection module, used for selecting the current video frame captured by the acquisition device adjacent to the observation viewpoint from the multiple current video frames as the first video frame and the second video frame; the positions between the observation viewpoint and each of the acquisition devices are different; A processing module, configured to project the first video frame onto a cylindrical surface of a three-dimensional model, determine an image overlap region of the first video frame and the second video frame on the three-dimensional model and an optimal stitching line between the first video frame and the second video frame; the pixel difference on both sides of the optimal stitching line is minimal, and the optimal stitching line is located in the image overlap region; A stitching module is used to stitch the first video frame and the second video frame according to the optimal stitching line to obtain a stitched image of the target object at the observation viewpoint.

12. A terminal, characterized in that: The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor is used to execute program data to implement the steps in the image stitching method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the image stitching method according to any one of claims 1 to 10 are implemented.

Citation Information

Patent Citations

  • Image processing method, device and system and storage medium

    CN109658365A

  • Video fusion algorithm based on dynamic optimal suture line and improved fade-in and fade-out method

    CN113221665A

  • Image three-dimensional reconstruction method, free viewpoint video generation method and electronic equipment

    CN118587367A

  • Splicing seam search method and apparatus for video image, and video image splicing method and apparatus

    WO2023011013A1

Cited By

  • Target image acquisition method and related device

    CN119155534A

  • Near-zero time delay horizontal look-around stitching system based on multi-camera suite

    CN120091231A