Panoramic image splicing method and device

By acquiring and processing the environmental image and laser point cloud data of the high pile dock, using color information and bounding box fitting, the problem of splicing difficulties with large number of piles and high similarity on the high pile dock is solved, and the accurate splicing of panoramic images is achieved and fusion improve identification accuracy and splicing stability.

CN119027310APending Publication Date: 2024-11-26WUHAN UNIV OF TECH +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411121047.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

There are many piles on high pile wharfs and are highly similar. It is difficult for traditional splicing methods to accurately identify and distinguish the differences between piles, resulting in unstable or incorrect splicing results.

Method used

By obtaining multiple environmental images and laser point cloud data of the high pile dock, two-dimensional or three-dimensional point cloud images are obtained, and the pile body is distinguished and identified using color information, and fit and splicing through bounding boxes to achieve the accuracy of the panoramic image. Splicing.

Benefits of technology

The accuracy of pile body recognition and splicing stability are improved, and the accurate splicing and fusion of panoramic images in high pile dock environments is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027310A_ABST
    Figure CN119027310A_ABST
Patent Text Reader

Abstract

The invention relates to a panoramic image splicing method and device, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a plurality of environment images and laser point cloud data of a high-pile wharf, obtaining a two-dimensional point cloud image according to the laser point cloud data, and enabling the two-dimensional point cloud image to comprise color information of different pile body types, the piles of the high-pile wharf can be distinguished and recognized through the color information, and then the accuracy of pile body recognition can be improved; further, fitting the plurality of environment images and the two-dimensional point cloud image to obtain a bounding box with color information in each environment image; according to the embodiment of the invention, a plurality of environment images can be spliced according to the bounding box, then the panoramic image can be obtained, and accurate splicing and fusion of the panoramic image in the high-pile wharf environment are realized by using the color information and the environment image splicing process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a panoramic image stitching method and device. Background Art

[0002] High-pile docks are usually used for cargo loading and unloading, ship mooring and other operations. The number of piles under the dock is huge and densely arranged. Due to seawater erosion and climate change, cracks will appear on the pile wall, posing a safety hazard. At present, the commonly used inspection method is manual inspection at low tide, which has safety hazards and low detection efficiency. The use of remote-controlled unmanned boats equipped with cameras for detection has become a current research hotspot. However, if a single camera perspective is used as a navigation method, the field of view is small and is not enough to support the needs of visual navigation and damage detection under high piles; if multiple cameras with different perspectives are used to provide real-time scenes, on the one hand, due to the high similarity of the piles in each camera, it is difficult to help the driver form a sense of orientation and space under the high pile ring directly presented to the driver. On the other hand, if the same damaged pile appears in two different perspectives, it is easy for the inspection personnel to misjudge the number of cracked piles, affecting the inspection efficiency.

[0003] Therefore, it is urgent to propose a panoramic image stitching method and device to solve the technical problem in the prior art that there are many piles on high-pile docks with high similarity, and traditional stitching methods often cannot accurately identify and distinguish the differences between the piles, resulting in unstable or erroneous stitching results. Summary of the invention

[0004] In view of this, it is necessary to provide a panoramic image stitching method and device to solve the technical problem in the prior art that there are a large number of piles on high-pile docks with high similarity, and traditional stitching methods often cannot accurately identify and distinguish the differences between the piles, resulting in unstable or erroneous stitching results.

[0005] In order to solve the above problems, the present invention provides a panoramic image stitching method, comprising: Acquire multiple environmental images and laser point cloud data of the high-pile dock; A two-dimensional point cloud image is obtained according to the laser point cloud data; the two-dimensional point cloud image includes color information representing different pile body categories; Fitting the multiple environment images and the two-dimensional point cloud image to obtain a bounding box with the color information in each environment image; The multiple environment images are stitched together according to the bounding box to obtain a panoramic image.

[0006] In a possible implementation, obtaining a two-dimensional point cloud image according to the laser point cloud data includes: A three-dimensional point cloud image is obtained according to the laser point cloud data; the three-dimensional point cloud image includes color information representing different pile body categories; The three-dimensional point cloud image is projected at four viewing angles to obtain a two-dimensional point cloud image corresponding to each viewing angle; the two-dimensional point cloud image includes color information corresponding to the three-dimensional point cloud image.

[0007] In a possible implementation, obtaining a three-dimensional point cloud image according to the laser point cloud data includes: According to the laser point cloud data, projection images of different viewing angles are obtained; Segmenting and eliminating the projection image to obtain a pile geometric feature point cloud; The pile geometric feature point cloud is clustered and color-labeled to obtain a three-dimensional point cloud image including color information of different pile body categories.

[0008] In a possible implementation, projecting the three-dimensional point cloud image at four viewing angles to obtain a two-dimensional point cloud image corresponding to each viewing angle includes: Obtaining a projection transfer matrix according to the installation positions of the camera and the laser radar and the laser point cloud data; Projecting the three-dimensional point cloud image at four viewing angles to obtain an initial two-dimensional image; Encoding the initial two-dimensional image according to the color information of the three-dimensional point cloud image to obtain a color vector; The color vector is fused with the two-dimensional point cloud feature of the initial two-dimensional image to obtain a two-dimensional point cloud image.

[0009] In a possible implementation, clustering and color-labeling the pile geometric feature point cloud to obtain a three-dimensional point cloud image including color information of different pile categories includes: Clustering the pile geometric feature point cloud to obtain point cloud data corresponding to each pile category; Determine the distance between the point cloud data of each pile type and the installation position of the laser radar to obtain a determination result; The color of each pile category is marked according to the judgment result to obtain a three-dimensional point cloud image with color information.

[0010] In a possible implementation, fitting the multiple environment images and the two-dimensional point cloud image to obtain a bounding box with the color information in each environment image includes: Processing the multiple environment images to obtain a geometric boundary corresponding to each environment image; Superimposing each environment image with the two-dimensional point cloud image to obtain an initial bounding box with color information; The initial bounding box is scaled according to the distance information of the point cloud data in the two-dimensional point cloud image to obtain a bounding box with color information corresponding to each environment image.

[0011] In a possible implementation, stitching the multiple environment images according to the bounding box to obtain a panoramic image includes: Obtaining feature points corresponding to each environment image according to the bounding box; Matching the feature points of the multiple environment images according to the color information to obtain a corresponding relationship between adjacent images; The multiple environment images are spliced ​​according to the corresponding relationship to obtain a panoramic image.

[0012] In a possible implementation manner, the step of stitching the multiple environment images according to the corresponding relationship to obtain a panoramic image includes: Perform perspective transformation estimation on the feature points according to the corresponding relationship to obtain a perspective transformation matrix between adjacent images; Cutting the adjacent images to obtain a plurality of small blocks; Perform perspective transformation on each small square according to the perspective transformation matrix to obtain a perspective image; All stitched images are stitched together to obtain a panoramic image.

[0013] In a possible implementation, stitching all stitched images to obtain a panoramic image includes: All perspective images are stitched together based on the maximum flow minimum cut method to obtain a stitched image; The stitched images are fused based on an image fusion method to obtain a panoramic image.

[0014] On the other hand, the present invention also provides a panoramic image stitching device, comprising: A data acquisition module, used to acquire multiple environmental images and laser point cloud data of the high-pile dock; An image determination module, used to obtain a two-dimensional point cloud image according to the laser point cloud data; the two-dimensional point cloud image includes color information representing different pile body categories; An image fitting module, used for fitting the multiple environment images and the two-dimensional point cloud image to obtain a bounding box with the color information in each environment image; An image stitching module is used to stitch the multiple environment images according to the boundary box to obtain a panoramic image.

[0015] The beneficial effect of the present invention is that multiple environmental images and laser point cloud data of a high-pile wharf can be obtained, and a two-dimensional point cloud image can be obtained according to the laser point cloud data, wherein the two-dimensional point cloud image includes color information of different pile body categories, and the piles of the high-pile wharf can be distinguished and identified by the color information, thereby improving the accuracy of pile body identification; further, by fitting multiple environmental images and two-dimensional point cloud images, a bounding box with color information in each environmental image is obtained; multiple environmental images can be spliced ​​according to the bounding box, thereby obtaining a panoramic image, and the color information and environmental image splicing process are utilized to achieve accurate splicing and fusion of panoramic images in the high-pile wharf environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a flow chart of an embodiment of a panoramic image stitching method provided by the present invention; Figure 2 A schematic diagram of the structure of an embodiment of a solution combining four high-resolution cameras and one central laser radar provided by the present invention; Figure 3 A schematic diagram of an embodiment of the laser point cloud data processing provided by the present invention; Figure 4 A schematic diagram of a process flow of an embodiment of three-dimensional point cloud image projection provided by the present invention; Figure 5 For the present invention Figure 1 A schematic flow chart of an embodiment of step S103; Figure 6 For the present invention Figure 1 A schematic flow chart of an embodiment of step S104; Figure 7 For the present invention Figure 6 A schematic flow chart of an embodiment of step S603; Figure 8 A schematic diagram of the structure of an embodiment of a panoramic image stitching device provided by the present invention; Fig. 9 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0017] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0018] like Figure 1 As shown, a specific embodiment of the present invention discloses a panoramic image stitching method, comprising: S101, acquiring multiple environmental images and laser point cloud data of the high-pile wharf; S102, obtaining a two-dimensional point cloud image according to the laser point cloud data; the two-dimensional point cloud image includes color information representing different pile body categories; S103, fitting multiple environment images and two-dimensional point cloud images to obtain a bounding box with color information in each environment image; S104 , stitching the multiple environment images according to the bounding box to obtain a panoramic image.

[0019] In a specific embodiment of the present invention, in terms of multi-view data collection, a solution combining four high-resolution cameras and a central laser radar is adopted, such as Figure 2 As shown in the figure, four high-resolution cameras are selected. To ensure complete coverage of the field of view, the camera field of view is selected to be 100 degrees. These cameras are evenly distributed around a disk with a diameter of 50 cm and placed on the top of the ship to ensure that the captured images have good clarity and resolution. The installation angle of the camera is fixed at 90 degrees and installed perpendicular to the horizontal plane to achieve full coverage. Thus, multiple environmental images and laser point cloud data of the high-pile wharf can be obtained, and then the multiple piles of the high-pile wharf can be identified and processed according to the laser point cloud data to obtain a two-dimensional point cloud image, wherein the two-dimensional point cloud image includes color information representing different pile categories, which is used to distinguish multiple piles, increasing the difference between piles, and multiple environmental images and two-dimensional point cloud images can also be fitted to obtain a bounding box with color information in each environmental image; thereby ensuring its consistency and accuracy with the actual scene, and then multiple environmental images can be spliced ​​according to the bounding box to obtain a panoramic image, thereby achieving accurate splicing and fusion of images.

[0020] Compared with the prior art, the present embodiment provides a method for obtaining multiple environmental images and laser point cloud data of a high-pile wharf, and can obtain a two-dimensional point cloud image based on the laser point cloud data, wherein the two-dimensional point cloud image includes color information of different pile categories, and the piles of the high-pile wharf can be distinguished and identified by the color information, thereby improving the accuracy of pile identification; further, by fitting multiple environmental images and two-dimensional point cloud images, a bounding box with color information in each environmental image is obtained; multiple environmental images can be spliced ​​according to the bounding box, and then a panoramic image can be obtained, and the color information and environmental image splicing process are utilized to achieve accurate splicing and fusion of panoramic images in the high-pile wharf environment.

[0021] In some embodiments of the present invention, step S102 includes: A three-dimensional point cloud image is obtained according to the laser point cloud data; the three-dimensional point cloud image includes color information representing different pile types; The three-dimensional point cloud image is projected at four viewing angles to obtain a two-dimensional point cloud image corresponding to each viewing angle; the two-dimensional point cloud image includes the corresponding color information in the three-dimensional point cloud image.

[0022] In a specific embodiment of the present invention, after collecting the laser point cloud data, the laser point cloud data can be processed to obtain a three-dimensional point cloud image with color information, and then dimensionality reduction processing can be performed to project the three-dimensional point cloud image at four perspectives to obtain a two-dimensional point cloud image with color information corresponding to each perspective.

[0023] In some embodiments of the present invention, Figure 3 , according to the laser point cloud data, a three-dimensional point cloud image is obtained, including: S301, obtaining projection images of different viewing angles according to the laser point cloud data; S302, segmenting and eliminating the projection image to obtain a pile geometric feature point cloud; S303, clustering and color-labeling the pile geometric feature point cloud to obtain a three-dimensional point cloud image including color information of different pile body categories.

[0024] In a specific embodiment of the present invention, the collected laser point cloud data of the laser radar is preprocessed. This includes commonly used filtering and denoising algorithms, such as statistical filtering and Gaussian filtering. Statistical filtering removes noise points by calculating the statistical characteristics of the neighborhood points around each point, while Gaussian filtering uses Gaussian functions to smooth the point cloud. These algorithms can effectively remove noise points and outliers and improve the quality and accuracy of the data. Projections of different perspectives are obtained through the preprocessed laser point cloud data, and then the projections are segmented to remove the scenes outside the upper edge of the dock and the piles, retaining the geometric features of the piles, that is, obtaining the pile geometric feature point cloud. The specific steps are: segmenting the laser point cloud data, eliminating the environment outside the high piles, and only retaining the point cloud information related to the high piles. A segmentation algorithm based on regional growth is adopted. The algorithm clusters the laser point cloud data and divides the point cloud into different regions according to the spatial relationship between the points. In this way, the background environment outside the high piles can be effectively distinguished, reducing the complexity of subsequent processing.

[0025] In some embodiments of the present invention, step S303 includes: Cluster the pile geometric feature point cloud to obtain the point cloud data corresponding to each pile category; The distance between the point cloud data of each pile type and the installation position of the laser radar is judged to obtain a judgment result; The color of each pile category is marked according to the judgment result to obtain a three-dimensional point cloud image with color information.

[0026] In a specific embodiment of the present invention, a clustering algorithm based on shape and density features is used to address the characteristics of a large number of piles and high similarity in a high-pile dock environment. The algorithm performs feature extraction and similarity measurement on laser point cloud data, clusters similar point clouds into the same pile category, thereby achieving accurate identification and positioning of high piles, and obtaining point cloud data corresponding to each pile category. After clustering, the point cloud data of each pile category can be further judged for distance. Since the point cloud information is essentially the cylindrical surface point cloud coordinates of the pile body after three-dimensional spatial coordinate extraction and clustering, the normal vector is used to estimate the distance to the closest section of the radar, and then the distance to the closest section is judged, that is, the clustered cylinders are color-annotated in turn according to the distance to the closest section, the same color is used as a whole after the same cluster, and different colors are used between different categories to distinguish different cylinders, and then a three-dimensional point cloud image with color information can be obtained.

[0027] In some embodiments of the present invention, Figure 4 , project the 3D point cloud image at four viewing angles to obtain the 2D point cloud image corresponding to each viewing angle, including: S401, obtaining a projection transfer matrix according to the installation positions of the camera and the laser radar and the laser point cloud data; S402, projecting the three-dimensional point cloud image at four viewing angles to obtain an initial two-dimensional image; S403, encoding the initial two-dimensional image according to the color information of the three-dimensional point cloud image to obtain a color vector; S404: Fusing the color vector with the two-dimensional point cloud features of the initial two-dimensional image to obtain a two-dimensional point cloud image.

[0028] In a specific embodiment of the present invention, the three-dimensional point cloud data is projected onto a two-dimensional plane based on a projection method. Specifically, according to the installation positions of the camera and the laser radar and the collected laser point cloud data, the projection transfer matrix is ​​calculated, and the three-dimensional point cloud image is projected at four camera viewing angles to obtain the corresponding initial two-dimensional image. At the same time, the initial two-dimensional image is encoded in combination with the color information in the previous step to further improve the expressiveness and discrimination of the data. A feature extraction method based on a color histogram is adopted to encode the color information of each pixel into a vector, that is, to obtain a color vector, and to fuse it with the two-dimensional point cloud features of the initial two-dimensional image to obtain a two-dimensional point cloud image. In this way, the color information can be fully utilized to reduce the dimension and characterize the point cloud data, thereby improving the robustness and accuracy of the data.

[0029] In some embodiments of the present invention, Figure 5 As shown, step S103 includes: S501, processing multiple environment images to obtain a geometric boundary corresponding to each environment image; S502, superimposing each environment image with the two-dimensional point cloud image to obtain an initial bounding box with color information; S503 , resizing the initial bounding box according to the distance information of the point cloud data in the two-dimensional point cloud image to obtain a bounding box with color information corresponding to each environment image.

[0030] In a specific embodiment of the present invention, a continuous threshold recognition method and a RANSAC (Random Sample Consensus) algorithm are used to extract the geometric boundaries of the piles in multiple environmental images acquired by the camera. Continuous threshold recognition is a classic image edge detection method, which is often used to extract edge information in an image. The algorithm sets a threshold and regards pixels in the image with grayscale values ​​higher or lower than the threshold as edge points, thereby obtaining edge information, as shown in formula (1): (1) In the formula, Represents the edge pixels in the environment image, Represents the grayscale value of the pixel in the image, and T represents the set threshold.

[0031] The continuous threshold recognition method can effectively detect edge information in the image, while the RANSAC algorithm can robustly estimate the geometric model in the image, thereby extracting the geometric boundaries of the pile. RANSAC (Random Sample Consensus) is a robust estimation algorithm that is often used to fit data models and remove outliers. The embodiment of the present invention uses the RANSAC algorithm to estimate the geometric boundaries of the pile in the image for subsequent feature point extraction. The core idea is to estimate the model parameters by randomly selecting data samples, and to evaluate the quality of the model by calculating the fitting error from the data points to the model, and then filter out a set of internal points that meet the requirements.

[0032] The embodiment of the present invention fuses the image acquired by the camera with the point cloud data acquired by the laser radar to obtain a bounding box with color information and distance information. First, the 2D point cloud data is superimposed on the image boundary to obtain an initial bounding box with color information. Then, the distance information of the point cloud data is used to rescale the initial bounding box to obtain a bounding box with color information to ensure its consistency and accuracy with the actual scene.

[0033] In some embodiments of the present invention, Figure 6 As shown, step S104 includes: S601, obtaining feature points corresponding to each environment image according to the bounding box; S602, matching feature points of multiple environment images according to color information to obtain a correspondence between adjacent images; S603: stitching the multiple environment images according to the corresponding relationship to obtain a panoramic image.

[0034] In a specific embodiment of the present invention, the endpoints of the bounding box are used as the feature points required for stitching, so that the feature points corresponding to each environmental image can be obtained. In the fusion process, a feature point matching method based on color information is used to solve the occlusion problem between the piles in the image stitching process. Feature point pairs with the same color information are given priority to ensure the accuracy and stability of the matching. Through feature point matching, the correspondence between adjacent images can be obtained, so that multiple environmental images can be stitched according to the correspondence to obtain a panoramic image, thereby achieving accurate stitching and fusion of images.

[0035] In some embodiments of the present invention, Figure 7 As shown, step S603 includes: S701, performing perspective transformation estimation on the feature points according to the corresponding relationship to obtain a perspective transformation matrix between adjacent images; S702, cutting adjacent images to obtain a plurality of small blocks; S703, performing perspective transformation on each small square according to the perspective transformation matrix to obtain a perspective image; S704: stitch all stitched images to obtain a panoramic image.

[0036] In a specific embodiment of the present invention, the DLT (Direct Linear Transformation) algorithm is used to estimate the perspective transformation matrix of the feature point pair using the matching results and the corresponding relationship. DLT (Direct Linear Transformation) is a method for calculating the perspective transformation matrix, which is commonly used in the fields of image processing and computer vision. In an embodiment of the present invention, the DLT algorithm is used to estimate the perspective transformation matrix of the feature point pair to achieve the affine transformation of the image. As shown in formula (2): (2) In the formula, and They represent the coordinates of the feature points in the original image and the transformed image respectively, and H represents the perspective transformation matrix.

[0037] Then, the adjacent images are perspective transformed, and the APAP technology is introduced to cut the image into multiple small blocks. Each small block is perspective transformed separately to obtain a perspective image, which further improves the accuracy and stability of stitching. Using this method for perspective transformation can effectively reduce the image distortion caused by perspective transformation.

[0038] In some embodiments of the present invention, step S704 includes: All perspective images are stitched together based on the maximum flow minimum cut method to obtain a stitched image; The stitched images are fused based on the image fusion method to obtain a panoramic image.

[0039] In a specific embodiment of the present invention, the maximum flow minimum cut algorithm is used to find the stitching seam to obtain the stitching image. The maximum flow minimum cut method (MFMC) is a graph theory algorithm used to solve network flow problems and can also be used for problems such as image segmentation and image fusion. In image fusion, the maximum flow minimum cut method is used to find the optimal path in image stitching so as to seamlessly combine the features of different images. The algorithm steps are as follows: Construct a network graph: The spliced ​​images are constructed into a graph, where the nodes represent the pixels of the image and the edges represent the connection relationship between the pixels.

[0040] Set capacity: Assign a capacity value to each edge, which indicates the connection strength between pixels. This capacity value can be determined based on factors such as similarity and distance between pixels.

[0041] Calculate the maximum flow: The maximum flow from the source node to the sink node in the graph is calculated by the maximum flow algorithm. In image fusion, the source node usually corresponds to the background area in the spliced ​​image, and the sink node corresponds to the foreground area.

[0042] Find the minimum cut: Use the minimum cut algorithm to find the minimum cut set from the source node to the sink node, split the image into two parts, maximize the flow from the source node to the sink node, and minimize the capacity of the cut set. This ensures the continuity and consistency of the spliced ​​image.

[0043] Generate panoramic image: According to the minimum cut set, the pixels of different images are fused to generate the final panoramic image.

[0044] After stitching the images, the multi-band blending strategy is used to achieve image fusion. This strategy is a commonly used image fusion method that can effectively eliminate the stitching gaps and maintain the continuity and consistency of the image. The multi-band blending strategy mainly includes the following steps: Image pyramid construction: First, the image to be fused is constructed into a Gaussian pyramid. The Gaussian pyramid is obtained by continuously downsampling and Gaussian filtering the image. Each layer is 1 / 2 the size of the previous layer and has different blur levels.

[0045] Laplacian pyramid construction: After the Gaussian pyramid is constructed, each layer of the Gaussian pyramid is upsampled and Gaussian filtered with the image on the previous layer to obtain the corresponding Laplacian pyramid. The Laplacian pyramid represents the difference between the original image and the corresponding layer image in the Gaussian pyramid, that is, the detail information.

[0046] Image fusion: For each layer of the Laplacian pyramid, the stitched images are weighted. The weights here are calculated based on the pixel values ​​in the stitched image and the positions of the corresponding pixels in the two images to maintain the consistency of the stitched area. Finally, the fused image is resampled to obtain the final panoramic image. The multi-band blending strategy can make full use of the multi-scale information of the image pyramid and effectively handle the color discontinuity and brightness differences between the stitched images, making the stitched panoramic image more natural and coherent, eliminating the stitching gaps, and improving the image quality and visual effects.

[0047] The feature point extraction and fusion part of the embodiment of the present invention combines image and point cloud data, utilizes color information and distance information, as well as feature point matching and image fusion algorithms, to achieve accurate stitching and fusion of panoramic videos in a high-pile dock environment, providing a reliable data basis for subsequent ship inspections and environmental monitoring.

[0048] Multi-view information fusion: The embodiment of the present invention uses multiple cameras and LiDAR equipment to comprehensively utilize information from different viewpoints and realizes panoramic video stitching by combining point clouds with images. Compared with traditional methods, it can obtain richer and more comprehensive scene information and improve the accuracy and completeness of stitching.

[0049] Adaptive color labeling: In view of the characteristics of high piles, the embodiment of the present invention introduces a color labeling method in the point cloud processing stage, and labels the boundary point cloud with different colors according to the distance of the pile. This adaptive color labeling method enables more accurate matching of the boundary information of the same pile at different viewing angles, thereby improving the accuracy of splicing.

[0050] Feature point matching and perspective transformation: Use continuous threshold recognition and RANSAC algorithm to match geometric edges in images of different perspectives and obtain corresponding feature point pairs. Use DLT algorithm to estimate the perspective transformation matrix and realize the perspective transformation of the image, so as to achieve accurate splicing and seamless connection of the image.

[0051] Precision improvement and error correction: In the process of feature point matching and perspective transformation, feature point pairs are screened based on the color information of the pile body in the overlapping areas of different viewing angles to eliminate erroneous points, effectively reducing the probability of mismatching caused by feature point loss due to overlap and improving the accuracy of stitching. At the same time, the APAP technology is introduced to cut the image into small blocks, and each small block is individually transformed in perspective, further improving the precision and stability of stitching.

[0052] In order to better implement the panoramic image stitching method in the embodiment of the present invention, based on the panoramic image stitching method, the embodiment of the present invention also provides a panoramic image stitching device, such as Figure 8 As shown, the panoramic image stitching device 800 includes: The data acquisition module 801 is used to acquire multiple environmental images and laser point cloud data of the high-pile wharf; The image determination module 802 is used to obtain a two-dimensional point cloud image according to the laser point cloud data; the two-dimensional point cloud image includes color information representing different pile body categories; An image fitting module 803 is used to fit multiple environment images and two-dimensional point cloud images to obtain a bounding box with color information in each environment image; The image stitching module 804 is used to stitch multiple environment images according to the boundary box to obtain a panoramic image.

[0053] The panoramic image stitching device 800 provided in the above embodiment can implement the technical solution described in the above panoramic image stitching method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above panoramic image stitching method embodiment, which will not be repeated here.

[0054] like Fig. 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902 and a display 903. Fig. 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0055] In some embodiments, the memory 902 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. In other embodiments, the memory 902 may also be an external storage device of the electronic device 900, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 900.

[0056] Furthermore, the memory 902 may include both an internal storage unit of the electronic device 900 and an external storage device. The memory 902 is used to store application software installed in the electronic device 900 and various data.

[0057] In some embodiments, the processor 901 may be a central processing unit (CPU), a microprocessor or other data processing chip, and is used to run program codes or process data stored in the memory 902, such as the panoramic image stitching method of the present invention.

[0058] In some embodiments, the display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 903 is used to display information of the electronic device 900 and to display a visual user interface. The components 901-903 of the electronic device 900 communicate with each other through a system bus.

[0059] In some embodiments of the present invention, when the processor 901 executes the panoramic image stitching program in the memory 902, the following steps may be implemented: Acquire multiple environmental images and laser point cloud data of the high-pile dock; A two-dimensional point cloud image is obtained according to the laser point cloud data; the two-dimensional point cloud image includes color information representing different pile body categories; Fit multiple environment images and two-dimensional point cloud images to obtain a bounding box with color information in each environment image; Multiple environment images are stitched together according to the bounding box to obtain a panoramic image.

[0060] It should be understood that: when the processor 901 executes the panoramic image stitching program in the memory 902, in addition to the above functions, other functions may also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0061] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 900 mentioned, and the electronic device 900 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0062] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the panoramic image stitching method steps or functions provided in the above-mentioned method embodiments can be implemented.

[0063] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0064] The panoramic image stitching method and device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A panoramic image stitching method, characterized in that: include: Acquire multiple environmental images and laser point cloud data of high-pile docks; A two-dimensional point cloud image is obtained according to the laser point cloud data; the two-dimensional point cloud image includes color information representing different pile body categories; Fitting the multiple environment images and the two-dimensional point cloud image to obtain a bounding box with the color information in each environment image; The multiple environment images are stitched together according to the bounding box to obtain a panoramic image.

2. The panoramic image stitching method according to claim 1, characterized in that: The step of obtaining a two-dimensional point cloud image according to the laser point cloud data comprises: A three-dimensional point cloud image is obtained according to the laser point cloud data; the three-dimensional point cloud image includes color information representing different pile body categories; The three-dimensional point cloud image is projected at four viewing angles to obtain a two-dimensional point cloud image corresponding to each viewing angle; the two-dimensional point cloud image includes color information corresponding to the three-dimensional point cloud image.

3. The panoramic image stitching method according to claim 2, characterized in that: The step of obtaining a three-dimensional point cloud image according to the laser point cloud data comprises: According to the laser point cloud data, projection images of different viewing angles are obtained; Segmenting and eliminating the projection image to obtain a pile geometric feature point cloud; The pile geometric feature point cloud is clustered and color-labeled to obtain a three-dimensional point cloud image including color information of different pile body categories.

4. The panoramic image stitching method according to claim 2, characterized in that: The projecting of the three-dimensional point cloud image at four viewing angles to obtain a two-dimensional point cloud image corresponding to each viewing angle includes: Obtaining a projection transfer matrix according to the installation positions of the camera and the laser radar and the laser point cloud data; Projecting the three-dimensional point cloud image at four viewing angles to obtain an initial two-dimensional image; Encoding the initial two-dimensional image according to the color information of the three-dimensional point cloud image to obtain a color vector; The color vector is fused with the two-dimensional point cloud feature of the initial two-dimensional image to obtain a two-dimensional point cloud image.

5. The panoramic image stitching method according to claim 3, characterized in that: The clustering and color marking of the pile geometric feature point cloud to obtain a three-dimensional point cloud image including color information of different pile categories includes: Clustering the pile geometric feature point cloud to obtain point cloud data corresponding to each pile category; Determine the distance between the point cloud data of each pile type and the installation position of the laser radar to obtain a determination result; The color of each pile category is marked according to the judgment result to obtain a three-dimensional point cloud image with color information.

6. The panoramic image stitching method according to claim 1, characterized in that: The fitting of the multiple environment images and the two-dimensional point cloud image to obtain a bounding box with the color information in each environment image includes: Processing the multiple environment images to obtain a geometric boundary corresponding to each environment image; Superimposing each environment image with the two-dimensional point cloud image to obtain an initial bounding box with color information; The initial bounding box is scaled according to the distance information of the point cloud data in the two-dimensional point cloud image to obtain a bounding box with color information corresponding to each environment image.

7. The panoramic image stitching method according to claim 6, characterized in that: The step of stitching the multiple environment images according to the bounding box to obtain a panoramic image includes: Obtaining feature points corresponding to each environment image according to the bounding box; Matching the feature points of the multiple environment images according to the color information to obtain a corresponding relationship between adjacent images; The multiple environment images are spliced ​​according to the corresponding relationship to obtain a panoramic image.

8. The panoramic image stitching method according to claim 7, characterized in that: The step of stitching the multiple environment images according to the corresponding relationship to obtain a panoramic image includes: Perform perspective transformation estimation on the feature points according to the corresponding relationship to obtain a perspective transformation matrix between adjacent images; Cutting the adjacent images to obtain a plurality of small blocks; Perform perspective transformation on each small square according to the perspective transformation matrix to obtain a perspective image; All stitched images are stitched together to obtain a panoramic image.

9. The panoramic image stitching method according to claim 8, characterized in that: The step of stitching all stitched images to obtain a panoramic image includes: All perspective images are stitched together based on the maximum flow minimum cut method to obtain a stitched image; The stitched images are fused based on an image fusion method to obtain a panoramic image.

10. A panoramic image stitching device, characterized in that: include: A data acquisition module, used to acquire multiple environmental images and laser point cloud data of the high-pile dock; An image determination module, used to obtain a two-dimensional point cloud image according to the laser point cloud data; the two-dimensional point cloud image includes color information representing different pile body categories; An image fitting module, used for fitting the multiple environment images and the two-dimensional point cloud image to obtain a bounding box with the color information in each environment image; An image stitching module is used to stitch the multiple environment images according to the boundary box to obtain a panoramic image.

Citation Information

Cited By

  • Panoramic radar image stitching method and device for fitting optical image, and computer equipment

    CN120876219A

  • Training method of image generation algorithm, storage medium, lidar and vehicle

    TWI929680B