Computer vision-based three-dimensional point cloud image fusion method

Abstract

The application discloses a computer vision-based three-dimensional point cloud image fusion method, relates to the technical field of image data processing, and comprises the following steps: S001, collecting image data and corresponding three-dimensional point cloud data, and obtaining a coordinate relationship corresponding to the image data according to a collecting device; S002, according to the distribution between similar regions in two-dimensional image data, superimposing two-dimensional image data of different scales on the largest two-dimensional image data; based on the largest two-dimensional image data, setting different segmented point cloud data according to the number of superimpositions; S003, extracting point cloud data distribution characteristics in each segment, obtaining a superimposed image and a superimposed region corresponding to the superimposed image; based on the density of the sampling region, obtaining a uniform region with consistent density; the spatial characteristics of the overlapping region can be detected more accurately, and the detection result of the image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and in particular to a three-dimensional point cloud image fusion method based on computer vision. BACKGROUND

[0002] With the development of science and technology, intelligent management has been realized in many enterprise parks, and the safety of the enterprise park is the most important in the intelligent management. With the popularization of unmanned inspection vehicles, unmanned inspection vehicles are often used to replace traditional methods such as manual inspection of the park, which greatly improves the security efficiency and the operation cost of the park. However, in the process of inspection of the unmanned inspection vehicle, it is necessary to collect image information in the park in real time, but the digital image collected by a single image collection device cannot meet the actual application requirements, and multiple sensors need to work together, and the fusion of three-dimensional point cloud data and image data is widely used in the unmanned inspection vehicle system. Three-dimensional point cloud data is different from two-dimensional image application. Three-dimensional point cloud data often represents the depth, geometry and other three-dimensional information characteristics of the target, and two-dimensional image data represents the color, texture and other two-dimensional information characteristics of the target. Therefore, the fusion of three-dimensional point cloud data and two-dimensional image data can obtain a three-dimensional point cloud model with color information, which can accurately represent the objective characteristics of the target, so that the unmanned inspection vehicle system can accurately detect the target condition and provide better automatic inspection assistance.

[0003] However, in the existing method, the three-dimensional point cloud data is projected onto a two-dimensional plane according to the coordinate conversion relationship for fusion, and the spatial characteristics of the three-dimensional point cloud data are represented by the similar areas between the continuous frames of images. However, the projection process ignores the inconsistent scale between the continuous frames of images, and the layers between each projection image are different, and the images also have offset phenomenon. When constructing the three-dimensional point cloud data, the differences between different layers are ignored, which cannot meet the normal inspection image fusion requirements. SUMMARY

[0004] The embodiment of the present application provides a three-dimensional point cloud image fusion method based on computer vision, which solves the technical problem that the existing technology easily ignores the detail features in the conversion of the superimposed image, and improves the image fusion effect.

[0005] The embodiment of the present application provides a three-dimensional point cloud image fusion method based on computer vision, which includes:

[0006] Step S001, collecting image data and corresponding three-dimensional point cloud data, and obtaining the coordinate relationship of the image data according to the collection device;

[0007] Step S002, according to the distribution between the similar regions in the two-dimensional image data, the two-dimensional image data of different scales is superimposed on the largest two-dimensional image data; based on the largest two-dimensional image data, according to the number of superimposition, the point cloud data of different segments is set;

[0008] Step S003, the point cloud data distribution characteristics in each segment are extracted, the superimposed image and the superimposed region corresponding to the superimposed image are obtained; based on the sampling region density, the uniform region with consistent density is obtained; based on the point cloud data corresponding to the superimposed region and the uniform region, the structure difference degree is obtained; based on the superimposed region and the uniform region, the global feature and the pooled maximum feature are obtained, the first structure complexity and the key point corresponding to the first structure complexity are obtained;

[0009] Step S004, according to the first structure complexity corresponding to each superimposed point, the three-dimensional data conversion is carried out, the local feature is obtained, if there is a key point outside the local feature, the other region features are connected, the second structure complexity is obtained; the three-dimensional model is generated according to the second structure complexity conversion.

[0010] The one or more technical solutions provided in the embodiments of the application have at least the following technical effects or advantages:

[0011] The overlapping region is processed by multiple elements, and according to the obtained first structure complexity and second structure complexity, the rule of extension of the key points of the overlapping region during conversion is known, so that the blurred region in the superimposed region which is ignored due to less sampling and the identification effect of the accurate region are improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 The flowchart of the application is shown. DETAILED DESCRIPTION

[0013] In order to facilitate the understanding of the application, the application will be described more fully below with reference to the related drawings; the preferred embodiments of the application are shown in the drawings, but the application can be realized in many different forms, and is not limited to the embodiments described herein; on the contrary, the purpose of providing these embodiments is to make the disclosure of the application more thorough and comprehensive.

[0014] It should be noted that the terms "vertical", "horizontal", "up", "down", "left", "right" and similar expressions used herein are only for the purpose of illustration, and are not the only embodiment.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0016] like Figure 1 As shown, a computer vision-based 3D point cloud image fusion method includes:

[0017] Step S001: Acquire image data and corresponding 3D point cloud data, and obtain the coordinate relationship corresponding to the image data based on the acquisition device.

[0018] In this step, the image data is sampled multiple times by the sensor, and the coordinates corresponding to the image are obtained, transforming the visible two-dimensional image data into three-dimensional image data. After determining the image coordinate data, different images of the same target in each image are acquired multiple times to obtain more accurate three-dimensional coordinates.

[0019] Step S002: Based on the distribution of similar regions in the two-dimensional image data, superimpose two-dimensional image data of different proportions onto the largest two-dimensional image data; based on the largest two-dimensional image data, set point cloud data of different segments according to the number of superpositions.

[0020] In this step, in order to improve the accuracy of the collected data, the two-dimensional image data is continuously acquired to obtain similar regions between consecutive data. The similar regions are then superimposed according to the characteristics of the image, which can highlight some blurred image features in the two-dimensional image. This makes the spatial feature relationship and image details more complete when the two-dimensional image is converted into a three-dimensional image.

[0021] Secondly, objects within each similar region can essentially be considered as layers on the same visual plane. Overlaying multiple plane layers can make the converted 3D coordinates closer to the actual distribution effect.

[0022] In addition, when acquiring image data, the target objects will change to a certain extent depending on the acquisition time in different regions. At this time, we prioritize the acquisition of the more significant regions of interest in the target image and perform superposition and fusion processing on the pixels of the regions of interest. The obtained regions of interest are set into different forms of local segments according to the number of superpositions. The number of superpositions shows the number of times the target object changes within a certain period of time. The more changes there are, the more times they need to be superimposed. Processing the regions with more changes can reduce the noise and image shift generated during the three-dimensional fusion of images.

[0023] The collected image data is grayed according to the target object to be collected, and a gray image after gray processing is obtained. The Sobel operator is used to extract the edges of the gray image to obtain an edge image. The edge images extracted from consecutive frames are regarded as similar regions according to the image regions where each image with similar regions or target objects exists. Each similar region is set to a different layer according to the spatial distribution relationship between the objects existing in the region. When the spatial distribution and similarity are high, it is regarded as a continuous layer. Each layer is superimposed according to different proportions to obtain a superimposed image. Each superimposed image is divided into a plurality of segmented point cloud data according to the number of local superimpositions.

[0024] Specifically, the superimposed images in the continuous period are merged and processed. The image regions with the same superimposition number are regarded as a group and divided into different segments to obtain the point cloud data in the corresponding segments.

[0025] In step S003, the point cloud data distribution characteristics in each segment are extracted to obtain the superimposed image and the superimposed region corresponding to the superimposed image. Based on the sampling region density, a uniform region with consistent density is obtained. Based on the point cloud data corresponding to the superimposed region and the uniform region, the structural difference degree is obtained. Based on the superimposed region and the uniform region, the global feature and the maximum feature after pooling are obtained to obtain the first structural complexity and the key points corresponding to the first structural complexity.

[0026] Specifically, the noise points and offset points in the point cloud data are removed. The sparse point cloud data and the incomplete point cloud data in the point cloud data are superimposed to obtain the corresponding superimposed point cloud data. The sparse point cloud data and the incomplete point cloud data are unified in the same perspective. The superimposed point cloud data is merged to obtain the superimposed region corresponding to the point cloud data.

[0027] The point cloud data is iteratively sampled. According to the density of the current sampling region, adaptive sampling is performed to obtain a uniform region with consistent density. According to the distance between the nearest neighbor points between the superimposed region and the uniform region, the offset between the uniform region and the superimposed region is obtained.

[0028] The point cloud three-dimensional feature data, the view two-dimensional feature, and the superimposed region feature are fused to obtain a superimposed feature.

[0029] The point cloud three-dimensional feature data, the view two-dimensional feature, and the uniform region feature are fused to obtain a uniform feature.

[0030] The offset is obtained by calculating the average value of the distance changes between the superimposed region and the uniform region. The superimposed region and the uniform region are superimposed according to the offset to obtain the precise and enhanced point cloud data.

[0031] To construct offset prediction, for the overlay region, predict the offset at multiple scales based on the extracted overlay features. The offset is added to the spatial coordinates of the largest overlay image corresponding to the overlay region to obtain accurately enhanced point cloud data.

[0032] For uniform regions, multi-scale offsets are predicted based on the extracted uniform features. The offsets are added to the spatial coordinates of the smallest superimposed image corresponding to the uniform region to obtain accurately enhanced point cloud data.

[0033] At this point, processing the superimposed point cloud data according to different sampling methods can make the features of the obtained point cloud data more similar to the general feature representation. The point cloud data is offset according to the superimposed area and the uniform area respectively, and the two are merged to remove the data noise caused by residual point clouds and sparse point clouds in the point cloud data.

[0034] Furthermore, any point in each segment of point cloud data is designated as the first origin, which satisfies the following condition:

[0035] The first origin is located in the region with the highest density in the point cloud data, and the first origin is connected to all its nearest neighbor points in the current region.

[0036] The first origin and any two nearest neighbor points can form a triangle connection, and the average value of the pixels in each triangle is less than a preset threshold.

[0037] There is at least one first origin in each segment of point cloud data, and each first origin is a vertex of another.

[0038] At this point, the segmented point cloud data is divided into different forms of internal triangles. By distinguishing the differences between the internal triangles, the structural difference degree corresponding to the current point cloud data can be obtained. The connectable state described above means that, based on the texture and edges of the image, the triangles formed by the collected point cloud data are combined without destroying the overall image.

[0039] Furthermore, each segment of the point cloud data satisfies a certain degree of structural difference; the structural difference is manifested as follows:

[0040]

[0041] Where D(S1, S2) is the structural difference between inner triangle S1 and inner triangle S2. It represents the normal vector of one of the sides of the inner triangle S1. It is in the inner triangle S2 and The normal vector of the corresponding edge; based on the normal vectors between the internal triangles, the differences in the distribution of the superimposed image in different regions can be obtained.

[0042] Further, the mean value of the internal triangle in each region is taken as the structural difference of different regions;

[0043] Based on the superimposed region and the uniform region, the global feature and the pooled maximum feature are obtained, the maximum feature is superimposed to obtain the first structural complexity.

[0044] The first structural complexity is obtained as follows:

[0045] According to the score of each feature point, the first K feature points are sorted and output;

[0046] The key points greater than the first threshold value in the feature points are extracted to obtain the position information and geometric features corresponding to the key points;

[0047] According to the structural difference between the triangles combined by every three key points, the first structural complexity is obtained by combination;

[0048] The first structural complexity is represented as:

[0049]

[0050] Wherein, O is the first structural complexity, a is a set hyperparameter, D(S i , S i+1 ) refers to the structural difference between the internal triangle S i and the internal triangle S i+1 , and the structural difference between each internal triangle is combined to obtain the first structural complexity of the corresponding segmented point cloud data.

[0051] The first structural complexity is normalized to obtain the weight of each first structural complexity. When the first structural complexity is greater than the preset complexity, the weight is the ratio of the first structural complexity to the total first structural complexity. When it is less than the preset complexity, the weight is set to 0, which means that the current point cloud data changes beyond the set threshold, and has little effect on the overall three-dimensional image conversion, which can be ignored.

[0052] Step S004, according to the first structural complexity corresponding to each superimposed point, the three-dimensional data conversion is carried out to obtain the local feature, if the local feature still exists key point, then the other regional features are connected to obtain the second structural complexity; according to the second structural complexity conversion to generate three-dimensional model.

[0053] First, the initial local feature is obtained, if the local feature still exists key point, then the other regional features are connected to obtain the second structural complexity, and the key points in the second structural complexity cannot exceed the set distance threshold,

[0054] Each time the highest key point in the original image is obtained, and the adjacent key points are obtained for preliminary transformation, the corresponding relationship set is obtained according to the obtained preliminary key points, the area is further expanded, and the identification is increased;

[0055] The distribution characteristics of the point cloud data in each segment are extracted, and the area with too many repeated times and less occurrence times in the continuous frames is set as a viewing area;

[0056] The point cloud data of the viewing area is processed and filtered, and the point cloud data is fused according to different proportional superposition effects to obtain a three-dimensional image;

[0057] At this time, the different images are normalized in local positions to solve the error caused by the imbalance of the image proportion or the image offset.

[0058] Specifically, all continuous images are initialized, the initialization is to convert the point cloud data into hyper-volumes, the hyper-volumes are established based on density or grid, for each hyper-volume, the similarity between the hyper-volume and adjacent hyper-volumes is calculated, the adjacent hyper-volumes are aggregated together according to the similarity measurement to form larger hyper-volumes, until each hyper-volume has relevance;

[0059] For different styles of images, the three-dimensional reconstruction method is also different, the reconstruction is combined to generate images with different weights according to the generated hyper-volumes, for example, the concave-convex is set as a weight to form, each weight image is regarded as a layer view, and the images with different proportions and weights of the same thing are superimposed according to the set weight and proportion in a certain order based on the similar shape of the object; the feature score corresponding to each point cloud data is obtained, and the second structure complexity is obtained according to the feature score and the first structure complexity;

[0060] The feature score is extracted according to the frequency domain characteristics of each segment of point cloud data, and the change caused by the influence of light and dark changes of the image is extracted.

[0061] The second structure complexity is represented as:

[0062]

[0063] Wherein, ε is the feature score of the current point cloud data, L is the distance between the key points in the point cloud data, is the sum of the distances of each key point, O is the first structure complexity of the current point cloud data, and ω is the second structure complexity of the current point cloud data;

[0064] The second structure complexity can represent the error generated in the conversion process of the current point cloud data, and the three-dimensional model of the point cloud data is adjusted according to the second structure complexity to improve the effect of the generated model.

[0065] The technical solutions in the embodiments of the present application have at least the following technical effects or advantages:

[0066] The overlapping area is processed in multiple elements, and the obtained first structural complexity and second structural complexity show the extension rule of the key points in the overlapping area when converted, so that the blurred area ignored due to less sampling in the overlapping area can be obtained, and the identification effect of the accurate area is improved.

[0067] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A 3D point cloud image fusion method based on computer vision, characterized in that, include: Step S001: Acquire image data and corresponding 3D point cloud data, and obtain the coordinate relationship corresponding to the image data based on the acquisition device; Step S002: Based on the distribution of similar regions in the two-dimensional image data, superimpose two-dimensional image data of different proportions onto the largest two-dimensional image data; based on the largest two-dimensional image data, set point cloud data of different segments according to the number of superpositions. Step S003: Extract the distribution features of point cloud data in each segment to obtain the overlay image and the overlay region corresponding to the overlay image; Based on the sampling region density, a uniform region with consistent density is obtained; based on the point cloud data corresponding to the superimposed region and the uniform region, the structural difference is obtained; based on the superimposed region and the uniform region, the global features and the maximum feature after pooling are obtained, and the first structural complexity and the key points corresponding to the first structural complexity are obtained. The structural differences are manifested as follows: in, yes , It represents the inner triangle. The normal vector of one of the edges. Zhongyu The normal vector of the corresponding edge; The first structural complexity is as follows: in, It is the first structural complexity. These are the hyperparameters that are set. It means The first structural complexity of the point cloud data for the corresponding segment is obtained by combining the structural differences between each internal triangle. Step S004: Based on the first structural complexity corresponding to each superimposed point, perform three-dimensional data transformation to obtain local features. If there are key points outside the local features, connect the other regional features to obtain the second structural complexity; generate a three-dimensional model based on the second structural complexity. The second structural complexity is manifested as follows: in, This refers to the feature scores of the current point cloud data. It refers to the distance between key points in point cloud data. It is the sum of the distances to each key point. This refers to the first structural complexity of the current point cloud data. This refers to the second structural complexity of the current point cloud data.

2. The computer vision-based 3D point cloud image fusion method as described in claim 1, characterized in that, The superimposed images within a continuous time period are merged. Image regions with the same number of superimpositions are considered as a group, divided into different segments, and point cloud data within the corresponding segments are obtained.

3. The computer vision-based 3D point cloud image fusion method as described in claim 1, characterized in that, Remove noise and offset points from the point cloud data, overlay the sparse point cloud data and the residual point cloud data to obtain the corresponding overlaid point cloud data, and unify the sparse point cloud data and the residual point cloud data into the same view; merge with the overlaid point cloud data to obtain the overlay area corresponding to the point cloud data. Iterative sampling of point cloud data is performed, and adaptive sampling is performed based on the density of the current sampling area to obtain a uniform region with consistent density. The offset between the uniform region and the superimposed region is obtained based on the distance between the nearest neighbor points of the superimposed region and the uniform region. The overlay features are obtained by fusing point cloud 3D feature data, view 2D features, and overlay region features. Uniform features are obtained by fusing point cloud 3D feature data, view 2D features, and uniform region features. The offset is obtained by superimposing the superimposed area and the uniform area based on the average change in distance between similar areas of the superimposed area and the uniform area, thus obtaining the accurately enhanced point cloud data.

4. The computer vision-based 3D point cloud image fusion method as described in claim 1, characterized in that, Take any point in each segment of point cloud data and denote it as the first origin. The first origin satisfies the following condition: The first origin is located in the region with the highest density in the point cloud data, and the first origin is connected to all its nearest neighbors in the current region. The first origin and any two nearest neighbor points can form a triangle connection, and the average value of the pixels in each triangle is less than a preset threshold. There is at least one first origin in each segment of point cloud data, and each first origin is a vertex of another.

5. The computer vision-based 3D point cloud image fusion method as described in claim 1, characterized in that, When the complexity of the first structure is greater than the preset complexity, the weight is the ratio of the complexity of the first structure to the total complexity of the first structure; when it is less than the preset complexity, the weight is set to 0.

6. The computer vision-based 3D point cloud image fusion method as described in claim 1, characterized in that, The global features and the maximum features after pooling are obtained based on the superimposed regions and uniform regions. The maximum features are then superimposed to obtain the first structural complexity.

7. The computer vision-based 3D point cloud image fusion method as described in claim 1, characterized in that, If there are key points in addition to the local features, then the other regional features are connected to obtain the second structural complexity, and the key points within the second structural complexity do not exceed the set distance threshold.

Images