Panoramic stitching method based on depth map and electronic equipment
By acquiring and processing color images and depth data of interactive scenes, registering and splicing of depth images, the splicing gap problem caused by the neglect of three-dimensional structure information in the prior art is solved, and high-quality panoramic image stitching is achieved, ensuring that Depth consistency and natural color transition.
Patent Information
- Application Number
- CN202410993160.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-08
AI Technical Summary
The existing panoramic image stitching technology mainly focuses on two-dimensional image information, ignores the three-dimensional structural information of the scene, resulting in discontinuity of stitching gaps and depths, especially when dealing with complex three-dimensional structural scenes.
By obtaining color images and depth data of interactive scenes, determining the original depth image, registering and stitching, using depth data to optimize the color and transition processing of the stitching area, and using technologies such as weighted averaging and bilateral filtering to ensure depth consistency and color Transition is natural.
The splicing gaps are eliminated, the color consistency of the image is maintained, the color distortion is avoided, the splicing efficiency and visual effect are improved, and the application is wide, and it is suitable for multiple technical fields.
Smart Images

Figure CN118918001A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing, and in particular to a panoramic stitching method based on a depth map and an electronic device. Background Art
[0002] With the advancement of computer vision and image processing technology, panoramic image stitching technology has been widely studied and applied. Panoramic image stitching technology aims to merge multiple images into an image with a wide viewing angle, so that viewers can get a 360-degree visual experience. At present, most panoramic image stitching technologies mainly rely on feature matching between images to achieve stitching, such as scale-invariant feature transform (SIFT), accelerated robust features (SURF) and other feature matching algorithms. In addition, some technologies optimize the stitching effect by using image alignment and image fusion techniques, such as multi-band fusion, Poisson fusion and other methods. However, these traditional stitching technologies mainly focus on two-dimensional image information and ignore the three-dimensional structural information of the scene, which may cause stitching gaps and depth discontinuity problems when processing scenes with complex three-dimensional structures. Summary of the invention
[0003] The present disclosure provides a panoramic stitching method based on a depth map and an electronic device to at least solve the above technical problems existing in the prior art.
[0004] According to a first aspect of the present disclosure, a panorama stitching method based on a depth map is provided, the method comprising:
[0005] Acquire a color image and first depth data of a panoramic view of the interactive scene, and determine an original depth image based on the color image and the first depth data;
[0006] Registering the color image and the original depth image to obtain a first depth image;
[0007] splicing the first depth images based on the second depth data of the first depth images to obtain an initial panoramic image;
[0008] The initial panoramic image is optimized to obtain a panoramic stitching image.
[0009] In one possible implementation, acquiring the color image and the first depth data of the interactive scene panorama includes:
[0010] Using a camera sensor to collect original color images of the interactive scene;
[0011] Using a depth sensor to collect original first depth data of the interactive scene;
[0012] Performing filtering processing on the color image and the first depth data to obtain a filtered color image and first depth data;
[0013] The camera sensor and the depth sensor have synchronization time parameters and spatial consistency setting parameters to ensure that the original color image and the original first depth data are collected at the same time.
[0014] In one possible implementation, determining an original depth image based on the color image and the first depth data includes:
[0015] Acquire three-dimensional point cloud data based on the color image and the first depth data;
[0016] An original depth image is determined according to the three-dimensional point cloud data.
[0017] In one possible implementation, registering the color image and the original depth image to obtain the first depth image includes:
[0018] Extracting features of color images under different visual conditions in the interactive scene to obtain image feature points of the color images;
[0019] Matching image feature points of different color images to obtain matched color images;
[0020] The position and scale of the original depth image are adjusted based on the matched color image to obtain the first depth image.
[0021] In one possible implementation, the stitching the first depth image based on the second depth data in the first depth image includes:
[0022] Analyzing the second depth data in the first depth image under different visual conditions in the interactive scene;
[0023] Determine a stitching priority and a stitching order of the first depth image based on the second depth data;
[0024] The first depth images are stitched according to the stitching priority and the stitching order.
[0025] In one possible implementation manner, determining the stitching priority and stitching order of the first depth image based on the second depth data includes:
[0026] determining continuity and overlapping areas of the second depth data;
[0027] determining a splicing order according to the continuity of the second depth data;
[0028] The first depth image in which the second depth data is continuous and has an overlapping area is determined as having a high stitching priority.
[0029] In one possible implementation, the stitching of the first depth images includes:
[0030] Performing edge fusion processing on the spliced first depth image by using a weighted average method;
[0031] A bilateral filter or a non-local mean filter is used to smooth a transition area of the first depth image based on the second depth data.
[0032] In one possible implementation manner, the optimizing the initial panoramic image includes:
[0033] The color of the stitching area in the initial panoramic image is adjusted in combination with the second depth data.
[0034] In one possible implementation manner, adjusting the color of the stitching area in the initial panoramic image includes:
[0035] An adaptive histogram equalization method is used to correct the brightness and color of the spliced area, and a color smoothing process is performed on the spliced area in combination with the second depth data.
[0036] According to a second aspect of the present disclosure, a panoramic stitching device based on a depth map is provided, the device comprising:
[0037] An acquisition module, used to acquire a color image and first depth data of the interactive scene panorama, and determine an original depth image based on the color image and the first depth data;
[0038] A registration module, configured to register the color image and the original depth image to obtain a first depth image;
[0039] a stitching module, configured to stitch the first depth images based on the second depth data of the first depth images to obtain an initial panoramic image;
[0040] The optimization module is used to optimize the initial panoramic image to obtain a panoramic stitching image.
[0041] According to a third aspect of the present disclosure, there is provided an electronic device, including:
[0042] at least one processor; and
[0043] a memory communicatively connected to the at least one processor; wherein,
[0044] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.
[0045] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.
[0046] The present invention discloses a panoramic stitching method and electronic device based on a depth map. The present application obtains a color image and first depth data of the interactive scene panorama through a camera, and determines an original depth image based on the color image and the first depth data; aligns the color image and the original depth image to obtain a first depth image; stitches the first depth image based on the second depth data of the first depth image to obtain an initial panoramic image; and optimizes the initial panoramic image to obtain a panoramic stitching image. The present application not only eliminates the stitching seams, but also maintains the color consistency of the image and avoids the color distortion problem through the precise alignment of the depth map and the effective fusion of the depth data information, as well as the optimization of the transition processing of the stitching area. In addition, the method provided by the present application has wide applicability and is easy to operate. It provides a high-quality panoramic image foundation for multiple technical fields, enriching the application effects and user experience in various fields.
[0047] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:
[0049] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.
[0050] Figure 1 The schematic diagram shows the implementation process of the panoramic stitching method based on the depth map of the embodiment of the present disclosure. Figure 1 ;
[0051] Figure 2 A schematic diagram of the structure of a panoramic stitching device based on a depth map according to an embodiment of the present disclosure is shown;
[0052] Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0053] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0054] In the related art, there are methods for image stitching using depth data. The stitching effect is improved by integrating depth data obtained by a depth camera (such as Kinect) or through a multi-view stereo vision algorithm. The introduction of depth data helps to better understand the three-dimensional structure of the scene, so that depth continuity is taken into account during the stitching process, reducing stitching errors.
[0055] However, although these methods have improved the stitching results to some extent, they still have some limitations. First, the accuracy and coverage of depth data are limited by the performance of the depth sensor. Second, the correlation between depth data and color information is not fully utilized, resulting in poor results in transition area processing and color consistency maintenance.
[0056] Existing technologies include panoramic stitching technology based on feature matching, panoramic stitching technology based on direct methods, and panoramic stitching technology using depth data. Among them, panoramic stitching technology based on feature matching mainly relies on the matching of feature points between images to achieve panoramic image stitching. The SIFT algorithm can find stable feature points for matching under different perspectives, scales, and rotations. However, this method may encounter difficulties when dealing with scenes with a large number of repeated textures or lack of features. Panoramic stitching technology based on direct methods does not rely on the matching of feature points, but directly uses the pixel information of the image to estimate camera motion or image alignment, providing a comparative study of multi-view stereo vision and demonstrating the application of direct methods in panoramic image construction. Although direct methods perform well in some cases, they are usually more sensitive to image noise and illumination changes. Panoramic stitching technology using depth data With the popularity of depth sensors, using depth data to assist panoramic image stitching has become a research hotspot. Using depth data obtained by depth sensors such as Kinect, real-time three-dimensional environment mapping is achieved. This type of technology can effectively improve the stitching effect with the assistance of depth data, but the accuracy and coverage of depth sensors are still limiting factors. In summary, the existing panoramic stitching technology has the following shortcomings:
[0057] (1) Feature matching-based techniques may fail when processing scenes with low texture or repeated texture, and have low stitching efficiency for large-scale scenes.
[0058] (2) Direct-method-based techniques are sensitive to illumination changes and noise, which may result in unsatisfactory stitching effects.
[0059] (3) Although the technology that uses depth data can improve the stitching quality, it is limited by the accuracy and measurement range of the depth sensor, and there may be inconsistencies in the process of fusing depth data and image data.
[0060] The following describes the depth map-based panoramic stitching method and electronic device provided by the present application in conjunction with the accompanying drawings.
[0061] like Figure 1 As shown, the present application provides a panorama stitching method based on a depth map, the method comprising:
[0062] S101, acquiring a color image and first depth data of a panoramic view of an interactive scene, and determining an original depth image based on the color image and the first depth data;
[0063] This application uses a camera to obtain a color image of the interactive scene, and uses a depth sensor to obtain the first depth data. It should be noted that in this application, the camera and the depth sensor are synchronized through an external clock signal to ensure that the color image and the depth map are collected at the same time point. Through a calibration board or other calibration tools, the Zhang Zhengyou calibration method is used to obtain the camera's intrinsic and extrinsic parameters to ensure spatial consistency between different cameras. This ensures the consistency between image data and depth data collected from different sensors.
[0064] Then, the present application combines the collected color image with the first depth data to generate three-dimensional point cloud data, and then converts the three-dimensional point cloud data into a depth map to provide a basis for subsequent depth map stitching.
[0065] S102, registering the color image and the original depth image to obtain a first depth image;
[0066] It should be noted that the image registration method is used in this application to spatially align the depth maps under different viewing angles. The feature point matching algorithm is used for preliminary alignment, and then the RANSAC algorithm is combined to remove false matches to ensure the accuracy of feature point matching. Based on the aligned color image, the iterative closest point algorithm is used to adjust the position and scale of the depth map to ensure the spatial consistency between the color image and the depth map.
[0067] S103, stitching the first depth images based on the second depth data of the first depth images to obtain an initial panoramic image;
[0068] In the present application, the second depth data is used to stitch the first depth image of the interactive scene, and a depth smoothing transition strategy is adopted in the stitching area to ensure the continuity of the depth and reduce the visual discontinuity phenomenon.
[0069] S104, optimizing the initial panoramic image to obtain a panoramic stitching image.
[0070] In this application, the stitched image is optimized, and the optimization may be to fuse the depth data and process the transition area, so as to maintain color consistency, ensure natural color transition, and enhance visual effects.
[0071] The depth map-based panoramic stitching method provided in this application utilizes depth data information for stitching and optimizes the stitching effect, thereby ensuring the depth consistency of the stitching process, improving the stitching efficiency, ensuring a natural transition of stitching colors, and improving the visual effect.
[0072] In some optional embodiments, acquiring the color image and the first depth data of the panoramic view of the interactive scene includes:
[0073] Using a camera sensor to collect original color images of the interactive scene;
[0074] Using a depth sensor to collect original first depth data of the interactive scene;
[0075] Performing filtering processing on the color image and the first depth data to obtain a filtered color image and first depth data;
[0076] The camera sensor and the depth sensor have synchronization time parameters and spatial consistency setting parameters to ensure that the original color image and the original first depth data are collected at the same time.
[0077] In some optional embodiments, determining the original depth image based on the color image and the first depth data includes:
[0078] Acquire three-dimensional point cloud data based on the color image and the first depth data;
[0079] An original depth image is determined according to the three-dimensional point cloud data.
[0080] It is understandable that multiple sensors are used in this application to obtain images of interactive scenes, such as ordinary camera sensors, such as cameras that collect original color images of interactive scenes, and depth cameras that collect original first depth data of interactive scenes. Then, the depth data collected by the depth sensor and the internal and external parameters of the camera are combined with the color image to generate a three-dimensional point cloud. The generated three-dimensional point cloud data is projected onto a two-dimensional plane and converted into a depth map. The hole areas in the depth map are filled by interpolation algorithms (such as nearest neighbor interpolation and bilinear interpolation) to ensure the integrity and continuity of the depth map.
[0081] It should be noted that a camera system equipped with a depth sensor is used in this application to simultaneously collect color images and depth data information of the scene. In order to ensure the consistency between each frame of the image and the corresponding depth data, high-precision time synchronization and spatial calibration technology are used. The time synchronization mechanism can ensure that different sensors collect data at the same time point, thereby avoiding data mismatch problems caused by time differences. Spatial calibration technology is used to calibrate the spatial position and angle of each camera to ensure that the images and depth data obtained from different cameras can be processed in the same spatial coordinate system.
[0082] It is understandable that in the collected raw data, the depth map may contain noise and inaccurate depth data information, so filtering and noise removal are required to improve the quality of the depth data. Common filtering methods include median filtering, Gaussian filtering, etc. For color images, there may be problems with inconsistent brightness and color, which will affect the subsequent stitching effect. This application uses brightness and color correction methods to reduce the lighting and color differences between different images, ensuring that the stitched panoramic image has a consistent visual effect.
[0083] In some optional embodiments, registering the color image and the original depth image to obtain the first depth image includes:
[0084] Extracting features of color images under different visual conditions in the interactive scene to obtain image feature points of the color images;
[0085] Matching image feature points of different color images to obtain matched color images;
[0086] The position and scale of the original depth image are adjusted based on the matched color image to obtain the first depth image.
[0087] It is understandable that in this application, before image stitching, it is necessary to align the images and depth data at different viewing angles. First, an advanced image registration algorithm is used to accurately align the images based on the feature point matching results of the color image. Based on the aligned color image, the position and scale of the depth map are adjusted to ensure the spatial consistency between the color image and the depth map. In this way, accurate basic data can be provided for subsequent depth map stitching and panoramic image generation.
[0088] In some optional embodiments, the stitching the first depth image based on the second depth data in the first depth image includes:
[0089] Analyzing the second depth data in the first depth image under different visual conditions in the interactive scene;
[0090] Determine a stitching priority and a stitching order of the first depth image based on the second depth data;
[0091] The first depth images are stitched according to the stitching priority and the stitching order.
[0092] Specifically, in the present application, determining the stitching priority and stitching order of the first depth image based on the second depth data includes:
[0093] determining continuity and overlapping areas of the second depth data;
[0094] determining a splicing order according to the continuity of the second depth data;
[0095] The first depth image in which the second depth data is continuous and has an overlapping area is determined as having a high stitching priority.
[0096] In some optional embodiments, the step of stitching the first depth images includes:
[0097] Performing edge fusion processing on the spliced first depth image by using a weighted average method;
[0098] A bilateral filter or a non-local mean filter is used to smooth a transition area of the first depth image based on the second depth data.
[0099] Specifically, the present application determines the priority and order of image stitching by analyzing the depth data information in the depth map. According to the continuity of the depth data information and the size of the overlapping area, a hierarchical clustering algorithm (such as K-means) is used to classify the images, and images with continuous depth data information and large overlapping areas are preferentially selected for stitching. During the stitching process, the depth map is used for edge fusion processing, and multi-band fusion technology is used, including using bilateral filtering or non-local mean filtering (NLM) to smooth the depth value, retain the depth edge information, and reduce noise. The depth values of the overlapping areas are fused by weighted averaging or other fusion algorithms to ensure a smooth transition of the depth values and avoid sharp boundaries. Color consistency adjustment is assisted by depth data information, and color correction algorithms (such as histogram matching and color smoothing) are applied, combined with the gradient information of the depth map for refinement processing to ensure that the color transition of the stitching area is natural. During the stitching process, the stitching strategy is dynamically adjusted according to the depth data information to ensure the smooth transition of the depth data information and the consistency of the visual effect.
[0100] In some optional embodiments, the optimizing the initial panoramic image includes:
[0101] The color of the stitching area in the initial panoramic image is adjusted in combination with the second depth data.
[0102] The adjusting the color of the stitching area in the initial panoramic image includes:
[0103] An adaptive histogram equalization method is used to correct the brightness and color of the spliced area, and a color smoothing process is performed on the spliced area in combination with the second depth data.
[0104] It is understandable that the stitched panoramic image is globally optimized, including operations such as geometric correction, color balance and sharpening. Geometric correction ensures that the overall structure of the panoramic image is correct, color balance improves the color consistency of the image, and sharpening ensures the clarity of the image. Through visual inspection and automated quality assessment tools, possible stitching errors or visual inconsistencies are detected. Based on the evaluation results, necessary manual or automatic corrections are made to ensure that the final panoramic image achieves high-quality visual effects. Specifically, adaptive histogram equalization (CLAHE) technology is used for brightness and color correction to reduce the lighting and color differences between different images. In the stitching area, color smoothing is performed in combination with depth information to ensure natural color transitions and enhance visual effects.
[0105] The panoramic stitching method based on depth map provided in this application collects color images and depth information through multi-sensor data, and performs preprocessing to improve data quality. Then the color image and depth data are aligned, and then the panoramic image is stitched using the depth data information, including steps such as priority determination, edge fusion and color adjustment. Finally, the stitched panoramic image is post-processed and optimized, including global optimization and quality evaluation, to finally complete the panoramic image stitching.
[0106] like Figure 2 As shown, an embodiment of the present application provides a panoramic stitching device based on a depth map, the device comprising:
[0107] An acquisition module 201 is used to acquire a color image and first depth data of the interactive scene panorama, and determine an original depth image based on the color image and the first depth data;
[0108] A registration module 202, configured to register the color image and the original depth image to obtain a first depth image;
[0109] A stitching module 203, configured to stitch the first depth images based on the second depth data of the first depth images to obtain an initial panoramic image;
[0110] The optimization module 204 is used to optimize the initial panoramic image to obtain a panoramic stitching image.
[0111] In some optional embodiments, the acquisition module 201 includes:
[0112] A first acquisition unit, used for acquiring an original color image of the interactive scene using a camera sensor;
[0113] A second acquisition unit, used to acquire original first depth data of the interactive scene using a depth sensor;
[0114] a processing unit, configured to perform filtering processing on the color image and the first depth data to obtain the filtered color image and the first depth data;
[0115] The camera sensor and the depth sensor have synchronization time parameters and spatial consistency setting parameters to ensure that the original color image and the original first depth data are collected at the same time.
[0116] In some optional embodiments, the acquisition module 201 further includes:
[0117] an acquisition unit, configured to acquire three-dimensional point cloud data based on the color image and the first depth data;
[0118] The first determining unit is configured to determine an original depth image according to the three-dimensional point cloud data.
[0119] In some optional embodiments, the registration module 202 includes:
[0120] An extraction unit, used to extract features of color images under different visual conditions in the interactive scene to obtain image feature points of the color images;
[0121] A matching unit, used for matching image feature points of different color images to obtain a matched color image;
[0122] The first adjustment unit is configured to adjust the position and scale of the original depth image based on the matched color image to obtain the first depth image.
[0123] In some optional embodiments, the splicing module 203 includes:
[0124] An analysis unit, configured to analyze second depth data in the first depth image under different visual conditions in the interactive scene;
[0125] A second determining unit, configured to determine a stitching priority and a stitching order of the first depth image based on the second depth data;
[0126] A stitching unit is configured to stitch the first depth images according to the stitching priority and the stitching order.
[0127] In some optional embodiments, the second determining unit includes:
[0128] A first determining subunit, configured to determine continuity and an overlapping area of the second depth data;
[0129] A second determining subunit, configured to determine a splicing order according to the continuity of the second depth data;
[0130] The third determining subunit determines the first depth image in which the second depth data are continuous and have an overlapping area as having a high stitching priority.
[0131] In some optional embodiments, the splicing unit includes:
[0132] A first processing subunit, configured to perform edge fusion processing on the spliced first depth image by using a weighted average method;
[0133] The second processing subunit is configured to perform smoothing processing on a transition area of the first depth image by using bilateral filtering or non-local mean filtering based on the second depth data.
[0134] In some optional embodiments, the optimization module 204 includes:
[0135] The second adjustment unit is configured to adjust the color of the stitching area in the initial panoramic image in combination with the second depth data.
[0136] In some optional embodiments, the second adjustment unit includes:
[0137] The third processing subunit is used to correct the brightness and color of the spliced area by using an adaptive histogram equalization method, and to perform color smoothing processing on the spliced area in combination with the second depth data.
[0138] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.
[0139] Figure 3 A schematic block diagram of an example electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0140] like Figure 3As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0141] A number of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0142] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as a panoramic stitching method based on a depth map. For example, in some embodiments, a panoramic stitching method based on a depth map may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the panoramic stitching method based on the depth map described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the depth map-based panoramic stitching method in any other appropriate manner (for example, by means of firmware).
[0143] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0144] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0145] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0147] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0148] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0149] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0150] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0151] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A panorama stitching method based on depth map, characterized in that: The method comprises: Acquire a color image and first depth data of a panoramic view of the interactive scene, and determine an original depth image based on the color image and the first depth data; Registering the color image and the original depth image to obtain a first depth image; splicing the first depth images based on the second depth data of the first depth images to obtain an initial panoramic image; The initial panoramic image is optimized to obtain a panoramic stitching image.
2. The method according to claim 1, characterized in that The step of acquiring the color image and the first depth data of the interactive scene panorama includes: Using a camera sensor to collect original color images of the interactive scene; Using a depth sensor to collect original first depth data of the interactive scene; Performing filtering processing on the color image and the first depth data to obtain a filtered color image and first depth data; The camera sensor and the depth sensor have synchronization time parameters and spatial consistency setting parameters to ensure that the original color image and the original first depth data are collected at the same time.
3. The method according to claim 1 or 2, characterized in that: Determining an original depth image based on the color image and the first depth data includes: Acquire three-dimensional point cloud data based on the color image and the first depth data; An original depth image is determined according to the three-dimensional point cloud data.
4. The method according to claim 1, characterized in that: The registering the color image and the original depth image to obtain a first depth image includes: Extracting features of color images under different visual conditions in the interactive scene to obtain image feature points of the color images; Matching image feature points of different color images to obtain matched color images; The position and scale of the original depth image are adjusted based on the matched color image to obtain the first depth image.
5. The method according to claim 1, characterized in that The stitching the first depth image based on the second depth data in the first depth image includes: Analyzing the second depth data in the first depth image under different visual conditions in the interactive scene; Determine a stitching priority and a stitching order of the first depth image based on the second depth data; The first depth images are stitched according to the stitching priority and the stitching order.
6. The method according to claim 5, characterized in that The determining the stitching priority and stitching order of the first depth image based on the second depth data includes: determining continuity and overlapping areas of the second depth data; determining a splicing order according to the continuity of the second depth data; The first depth image in which the second depth data is continuous and has an overlapping area is determined as having a high stitching priority.
7. The method according to claim 5, characterized in that The stitching of the first depth images includes: Performing edge fusion processing on the spliced first depth image by using a weighted average method; A bilateral filter or a non-local mean filter is used to smooth a transition area of the first depth image based on the second depth data.
8. The method according to claim 1, characterized in that The optimizing the initial panoramic image comprises: The color of the stitching area in the initial panoramic image is adjusted in combination with the second depth data.
9. The method according to claim 8, characterized in that The adjusting the color of the stitching area in the initial panoramic image includes: An adaptive histogram equalization method is used to correct the brightness and color of the spliced area, and a color smoothing process is performed on the spliced area in combination with the second depth data.
10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.
