Three-dimensional occlusion voxel culling method, state estimation method, device, and medium

Abstract

The application relates to the field of data processing, and discloses a three-dimensional occlusion voxel elimination method, a state estimation method, equipment and a medium. The method comprises the following steps: acquiring continuous multiple image frames and continuous multiple point cloud frames in a carrier movement process; for each pixel block in each image frame, calculating the pixel intensity variance of each pixel block in each image frame to determine all stable regions in each image frame; back-projecting each stable region to a corresponding point cloud frame to determine occlusion voxels from each point cloud frame; eliminating the occlusion voxels in each point cloud frame, and eliminating the pixel blocks corresponding to the occlusion voxels in each image frame. The application combines image frames and point cloud frames to calculate the pixel-level standard deviation, and performs a back-projection operation to realize the fusion processing of the image and the point cloud, so as to identify the carrier structure noise therefrom; the whole processing process does not require training data and does not depend on any AI model or labeled sample, thereby saving a large amount of data processing cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a three-dimensional occlusion voxel elimination method, a state estimation method, a device and a medium. BACKGROUND

[0002] When using scanning devices (such as lidar, camera) for environment perception, if these devices are installed on a moving carrier (such as a vehicle, a drone, a handheld device), the structure of the carrier itself (such as a vehicle body, a rack, a gimbal) will constantly appear in the field of view of the sensor as it moves, causing "occlusion" or "interference" in the data, which affects subsequent tasks such as environment modeling, obstacle avoidance, and target detection.

[0003] The existing processing method is to rely on a large amount of labeled data to train an AI model to identify and eliminate these interferences. However, this method requires continuous collection and labeling of new samples, i.e., the data processing cost is high, which makes it difficult to adapt to different types of carriers and scenes, resulting in low processing efficiency and poor adaptability of carrier structure interference elimination. SUMMARY

[0004] Therefore, in order to solve the problems in the prior art, the present application provides a three-dimensional occlusion voxel elimination method, a state estimation method, a device and a medium.

[0005] In a first aspect, the present application provides a three-dimensional occlusion voxel elimination method, comprising:

[0006] obtaining continuous multiple image frames and continuous multiple point cloud frames in the motion process of a carrier;

[0007] For each pixel block in each image frame, calculating the pixel intensity variance of each pixel block in each image frame to determine all stable regions in each image frame;

[0008] back-projecting each stable region into a corresponding point cloud frame to determine occlusion voxels from each point cloud frame;

[0009] eliminating the occlusion voxels in each point cloud frame, and eliminating the pixel blocks corresponding to the occlusion voxels in each image frame.

[0010] In an optional implementation, the back-projection of each stable region into a corresponding point cloud frame to determine occlusion voxels from each point cloud frame comprises:

[0011] based on the transformation relationship between the radar coordinate system and the camera coordinate system, converting each stable region into a first unit vector in the camera coordinate system, and converting the first unit vector into a second unit vector in the radar coordinate system;

[0012] constructing a ray equation based on the second unit vector;

[0013] convert each frame of point cloud into a regular cubic network to obtain a plurality of voxel spaces; the voxel spaces contain a plurality of voxels;

[0014] analyze the passing information of the rays corresponding to the ray equation passing through voxels in each voxel space, and determine occluded voxels in each voxel according to the passing information.

[0015] In an optional implementation, the analyzing the passing information of the rays corresponding to the ray equation passing through voxels in each voxel space, and determining occluded voxels in each voxel according to the passing information, comprises:

[0016] for each voxel, counting the number of times of passing through each voxel by the rays corresponding to the ray equation after being emitted from the end point and the number of hits when first passing through each voxel in each voxel space;

[0017] constructing an occupancy grid for each voxel, and updating the value of the occupancy grid according to the number of times of passing through and the number of hits of each voxel;

[0018] determining occluded voxels from each voxel based on the value of each occupancy grid.

[0019] In an optional implementation, the determining occluded voxels from each voxel based on the value of each occupancy grid, comprises:

[0020] for each occupancy grid, calculating the occupancy probability of the number of hits in the occupancy grid;

[0021] if the occupancy probability is greater than a first preset threshold, determining the voxel corresponding to the occupancy grid as an occluded voxel.

[0022] In an optional implementation, the analyzing the passing information of the rays corresponding to the ray equation passing through voxels in each voxel space, and determining occluded voxels in each voxel according to the passing information, comprises:

[0023] for each voxel, determining the target voxel first passing through by the rays corresponding to the ray equation after being emitted from the end point in each voxel space, and counting the number of first passing through of each target voxel in each voxel space to obtain the number of hits;

[0024] determining occluded voxels in each target voxel based on the number of hits; wherein if the number of hits is greater than a second preset threshold, the target voxel is determined as an occluded voxel.

[0025] In an optional implementation, the calculating the pixel intensity variance of each pixel block in each image frame to determine all stable regions in each image frame comprises:

[0026] calculating a time domain mean value of each pixel block;

[0027] constructing a pixel intensity covariance matrix according to the time domain mean value;

[0028] determining stable regions in each image frame based on the pixel intensity variance of each pixel block in each image frame in the pixel intensity covariance matrix.

[0029] In an optional implementation, the calculating the pixel intensity variance of each pixel block in each image frame to determine all stable regions in each image frame comprises:

[0030] for a single image frame, if the pixel intensity variance is less than a third preset threshold, determining that the pixel block corresponding to the pixel intensity variance is a stable region in the image frame;

[0031] and / or, after the determining all stable regions in each image frame, further comprising:

[0032] respectively performing binary image labeling on the stable regions and the non-stable regions in each image frame to obtain a binary mask image.

[0033] In a second aspect, the present application provides a self-state estimation method based on dynamic view angle difference, comprising:

[0034] obtaining continuous multiple frames of point clouds and continuous multiple frames of images in a carrier movement process, using the three-dimensional occlusion voxel removal method as described above to identify and remove occlusion voxels in the continuous multiple frames of point clouds, and identify and remove pixel blocks corresponding to the occlusion voxels in the continuous multiple frames of images;

[0035] based on the continuous multiple frames of point clouds and the continuous multiple frames of images after the removal processing, determining self-state information of the carrier at different time points.

[0036] In a third aspect, the present application provides a carrier device, comprising a processor and a memory, the memory stores a computer program, and the processor is configured to execute the computer program to implement the three-dimensional occlusion voxel removal method of the first aspect or the self-state estimation method based on dynamic view angle difference of the second aspect.

[0037] In a fourth aspect, the present application provides a computer storage medium storing a computer program, which, when executed on a processor, implements the three-dimensional occlusion voxel removal method according to the first aspect or the self-state estimation method based on dynamic view angle difference according to the second aspect.

[0038] The embodiments of the present application have the following beneficial effects:

[0039] The embodiments of the present application provide a three-dimensional occlusion voxel removal method, which utilizes the difference between the stability of the carrier structure and the dynamics of the environmental object, analyzes the change intensity of the pixel block or the voxel in the multi-frame image or the point cloud data collected by the sensor, extracts the stable occlusion features in the multi-frame data, identifies the structure noise fixed on the carrier, and removes it from the multi-frame image and the multi-frame point cloud, so as to retain the real environmental information for subsequent processing. The entire implementation process does not require training data and does not depend on any AI model or labeled samples, thereby saving a large amount of data processing cost. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope of protection of the present application. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0041] Figure 1 A flowchart of the three-dimensional occlusion voxel removal method in the embodiments of the present application is shown;

[0042] Figure 2a An untreated original image frame diagram in the embodiments of the present application is shown;

[0043] Figure 2b An image frame diagram of the original image frame after removing the pixel block corresponding to the occlusion voxel in the embodiments of the present application is shown;

[0044] Figure 3a An untreated original point cloud frame diagram in the embodiments of the present application is shown;

[0045] Figure 3b A point cloud frame diagram of the original point cloud frame after removing the occlusion voxel in the embodiments of the present application is shown;

[0046] Figure 4 A flowchart of the self-state estimation method based on dynamic view angle difference in the embodiments of the present application is shown. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application.

[0048] The components of the embodiments of the present application generally described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0049] Hereinafter, the terms "include", "have", and their conjugates, used in the various embodiments of the present application, are merely intended to denote that specific features, numbers, steps, operations, elements, components, or combinations thereof, are present or are added, and are not intended to exclude the presence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof.

[0050] In addition, the terms "first", "second", "third", and the like are used only to distinguish descriptions, and are not understood to indicate or imply relative importance.

[0051] Unless otherwise defined, all terms used herein (including technical terms and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in a generally used dictionary) will be interpreted to have the same meaning as the contextual meaning in the relevant technical field and will not be interpreted to have an idealized or overly formal meaning, unless clearly defined in the various embodiments of the present application.

[0052] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0053] Pixel intensity variance refers to the difference in gray value or color value of corresponding pixels in multiple images.

[0054] A voxel is a basic unit in three-dimensional space, similar to a pixel in two-dimensional space, composed of a regular cubic grid, and each voxel contains the center of gravity or average value of points in the region, approximating the spatial structure by a cubic unit.

[0055] In a mobile carrier system or carrier device equipped with sensors such as LiDAR, camera, etc., for example, a vehicle-mounted system (automatic driving, environment perception), an airborne system (unmanned aerial vehicle surveying), a handheld device (AR / VR headset, scanner), these devices collect data (images or point clouds) of the surrounding environment through sensors during movement for subsequent tasks such as environment modeling, dynamic obstacle avoidance, and target detection.

[0056] When the sensor moves with the carrier device, the structure of the carrier device itself (such as the vehicle body, support, gimbal, etc.) will also appear in the sensor's field of view. The movement of the carrier device causes the carrier's own structure to constantly deform in the sensor data, and these own states will affect subsequent environment perception, obstacle avoidance, and target detection tasks. Since these structures are fixed or relatively static, they will exhibit a certain "stability" in continuous multiple frames of data, but will also exhibit "occlusion" characteristics as the viewing angle changes.

[0057] The prior art usually relies on AI training samples, which requires a large amount of labeled data to identify which parts belong to carrier noise, but this process has problems such as high data cost, poor generalization ability (difficult to adapt to all scenarios), and dependence on human intervention.

[0058] Based on this, the embodiments of the present application provide a three-dimensional occlusion voxel removal method that uses dynamic viewing angle difference characteristics to distinguish between stable occlusions and real environment information in sensor data, and uses voxel blocks or voxels of stable occlusions as identifiers of carrier structures to automatically identify and remove carrier noise in sensor data.

[0059] It can be understood that the carrier part is fixedly connected with the sensor, and the occlusion area remains stable in the viewing angle change, and thus exhibits low pixel intensity variance (i.e., stability) in multiple frames of data; while the environment part is the surrounding real object, which exhibits diversity in the viewing angle change, and exhibits high pixel intensity variance (i.e., instability) in multiple frames of data. Based on the analysis of this stable or unstable feature, the embodiments can achieve identification and removal of the carrier structure; and by utilizing the difference between multiple dynamic views collected by the sensor, the noise points caused by the carrier structure are automatically identified and removed from the perspective of visual stability and point cloud statistics, and the real environment information is preserved.

[0060] In short, the embodiments identify the areas that change little and appear repeatedly by analyzing the changes of pixel blocks or voxels in multiple image / point cloud frames, which are likely to be caused by occlusion of the carrier's own structure. Then, the identified "stable occlusion" area is used as the carrier structure to be removed from subsequent processing, thereby improving the accuracy of environment perception. As can be seen, the entire processing process does not require training data and does not rely on any AI model or labeled samples, thereby saving a large amount of data processing cost.

[0061] See also Figure 1 , exemplary, the three-dimensional occlusion voxel culling method includes the following steps:

[0062] S110 , acquiring continuous multi-frame images and continuous multi-frame point clouds during the movement of the carrier.

[0063] First, a sensor is used to collect continuous multi-frame scanning data while the carrier is in motion. The sensor includes but is not limited to a camera, a video camera, and a lidar, and the carrier includes but is not limited to a vehicle, an unmanned aerial vehicle, a handheld device, etc. Optionally, the scanning data includes continuous multi-frame images and continuous multi-frame point clouds.

[0064] S120 , for each pixel block in each image frame, calculating the pixel intensity variance of each pixel block in each image frame to determine all stable regions in each image frame.

[0065] In this embodiment, each image frame is divided into pixel blocks, each of which is considered an independent unit. Each pixel block can be represented by its coordinates in the image frame, which are expressed as (x, y). In other words, the pixel blocks referred to herein are pixels in the image.

[0066] For each pixel block (x, y), the variance of its pixel intensity across multiple consecutive image frames is calculated. This pixel intensity variance is used to measure the stability of each pixel block; in other words, the variance of the pixel intensity of each pixel block across multiple consecutive image frames can be used to determine whether the pixel block is a stable region in the image frame.

[0067] For reference, the time domain mean (i.e., the three-dimensional pixel mean) of each pixel block in multiple consecutive frames of images is calculated; a pixel intensity covariance matrix is ​​constructed based on the time domain mean; a stability measure (i.e., a stable region) is extracted from the pixel intensity covariance matrix; that is, based on the pixel intensity variance of each pixel block in each image frame within the pixel intensity covariance matrix, the stable region in each image frame is determined.

[0068] Among them, the image frame sequence { I k Taking |k=1,…,N} as an example, for each pixel block (x, y), the time domain average value is calculated to obtain:

[0069]

[0070] Where, The temporal mean of the pixel block with coordinates (x, y) is a three-dimensional vector. is the RGB vector of the pixel block with coordinates (x, y); N is the total number of image frames, I k (x, y ) is the RGB value of the pixel block with coordinates (x, y) in the kth image frame; k represents the kth image frame.

[0071] Based on the time domain mean of each pixel block, a pixel intensity covariance matrix can be constructed:

[0072]

[0073] In the formula, N is the total number of image frames, I k x, y ) is the GRB value of the pixel block with coordinates (x, y) in the kth image frame; k represents the kth image frame. The covariance matrix is a 3*3 real matrix.

[0074] Then, a stability measure is extracted from the pixel intensity covariance matrix:

[0075]

[0076] In the formula, is the matrix trace, which can represent the stability degree of the pixel block; the greater the value of the matrix trace, the more unstable the pixel block, and the smaller the value of the matrix trace, the more stable the pixel block, and the more likely the pixel block is the carrier itself.

[0077] It can be understood that the stable region in all image frames is determined from the continuous multiple image frames through the pixel intensity variance of each pixel block in the continuous multiple image frames.

[0078] If the pixel intensity variance of the pixel block is less than a third preset threshold (denoted as ), it is determined that the pixel block corresponding to the pixel intensity variance is a stable region, and it is considered that the change of the pixel block is very small, which belongs to the stable region, and the stable region can be a carrier structure (such as a vehicle body, a support, etc.). The value of the third preset threshold is not limited, and the value of the preset threshold can be adaptively adjusted according to the scene complexity, light change, etc.; for example, the third preset threshold can be 0.01~0.05.

[0079] In other words, the region with a pixel intensity variance lower than the third preset threshold in each image is the stable region.

[0080] ​​​In an example, after determining all stable regions in each image frame, a binary mask can be generated according to the stability metric, specifically, the stable regions and non-stable regions in each image frame are respectively marked by binary image marking to obtain a binary mask image. That is, the pixel region that may be the carrier structure is marked in the image frame by binary image marking.

[0081] The digital expression of the binary mask image is:

[0082]

[0083] In the formula, is a third preset threshold value; is a stable region; is a binary function, representing a region less than is a stable region.

[0084] S130, each stable region is back projected into the corresponding point cloud frame to determine the occluded voxels from each point cloud frame.

[0085] In the present embodiment, the stable region identified in the image frame is back projected into the corresponding three-dimensional point cloud frame, that is, a single stable region is converted from a two-dimensional camera coordinate system to a three-dimensional radar coordinate system.

[0086] Wherein, the origin of the camera coordinate system (denoted as C) is at the optical center, and the Z-axis points to the imaging direction; the origin of the radar coordinate system (denoted as L) is at the radar center.

[0087] The transformation relationship expression between the camera coordinate system and the radar coordinate system is:

[0088]

[0089] In the formula, represents the transformation relationship of the radar coordinate system converted to the camera coordinate system; is the rotation of the radar coordinate system converted to the camera coordinate system; t is the position of the camera in the radar system.

[0090] Further, based on the transformation relationship between the radar coordinate system and the camera coordinate system, each stable region is converted into a first unit vector in the camera coordinate system. Wherein, the expression of the first unit vector (denoted as ) is:

[0091]

[0092] In the formula, K is the camera intrinsic parameter; (x, y) is the coordinate of the stable region.

[0093] The first unit vector is converted into a second unit vector in the radar coordinate system. The second unit vector (denoted as The expression of the equation is:

[0094]

[0095] In the formula, T represents transposition.

[0096] Next, in the three-dimensional point cloud frame, the nearest occlusion voxel is searched in each view cone direction, and the occlusion voxel is a candidate voxel that may belong to the carrier structure.

[0097] Specifically, each frame of point cloud is correspondingly converted into a regular cubic network to obtain a plurality of voxel spaces; each voxel space contains a plurality of voxels.

[0098] The voxel grid parameters of each voxel space include a grid origin coordinate, a voxel size, and a grid dimension. For example, the grid origin coordinate in a single voxel space is: , the voxel size is , and the grid dimension is .

[0099] Based on different second unit vectors, a normalized ray equation is constructed. The ray corresponding to each ray equation is a view cone direction.

[0100] The expression of the ray equation is:

[0101]

[0102] In the formula, is the origin of the radar coordinate system, P represents a sampling point in the view cone direction, t is a sampling distance on the ray equation, t ≥ 0.

[0103] Further, the passing information of the ray corresponding to the ray equation in each voxel space is analyzed, and the occlusion voxels are determined in each voxel according to the passing information. For each frame of point cloud, the number of hits of each voxel is counted, and if a voxel is hit multiple times in multiple frames, it is considered to have a high probability of belonging to the carrier structure. The hit means that the ray passes through the voxel.

[0104] It can be understood that whether a voxel is an occlusion voxel is determined according to the case that the ray equation passes through the voxel (i.e., hits the voxel) in each voxel space.

[0105] In an embodiment, for each voxel, in each voxel space, the number of paths of the ray corresponding to the ray equation that passes through and does not pass through each voxel after being emitted from the end point and the number of hits when first passing through each voxel are counted.

[0106] That is, in each voxel space, a ray is extended from the starting point (such as the sensor position) along the direction until it hits the voxel, and the voxels passed by the ray (Miss Voxels), the voxels finally hit (Hit Voxels), and the number of passes and the number of hits of each voxel are recorded in the process.

[0107] Next, an occupancy grid is constructed for each voxel, and the value of the occupancy grid is updated according to the number of passes and the number of hits of each voxel. Then, based on the value of each occupancy grid, an occluded voxel is determined from each voxel.

[0108] In this embodiment, for each occupancy grid, the value of the occupancy grid includes the number of hits and the number of passes, that is, one occupancy grid corresponds to one voxel, and the value in the occupancy grid is adjusted according to the number of hits and the number of passes of the voxel. In an example, the occupancy grid is a data structure for representing the occupancy state of the space, which includes a plurality of grid cells, and each grid cell records whether the position is occupied. If the number of hits of the voxel is increased by one, one grid cell in the occupancy grid is marked as occupied, and the occupancy probability of the number of hits is determined by combining the empty grid cells (i.e., unmarked grid cells, representing the number of passes) in the occupancy grid.

[0109] The occupancy probability of the number of hits in the occupancy grid is calculated, wherein the occupancy probability is the proportion of the number of hits to the sum of the number of hits and the number of passes. If the occupancy probability is greater than a first preset threshold, the voxel corresponding to the occupancy grid is determined as an occluded voxel. The value of the first preset threshold can be set according to actual needs, and the present embodiment is not limited in this regard.

[0110] In another embodiment, the determination process of the occluded voxel can be represented by a mathematical formula.

[0111] In an example, based on the expression formula of the voxel grid parameters and the ray equation described above, the occluded voxel can be identified by discrete step detection.

[0112] In this embodiment, the step size is The expression of each point cloud coordinate is: .

[0113] The point cloud coordinates are converted into voxel indexes:

[0114]

[0115] Based on the voxel indexes described above, the sampling distance (i.e., t) on the ray equation is sequentially traversed from 0, 1 to positive infinity to obtain the position of the sampling point. It is determined whether there is a point in the current point cloud frame that is located in the same occupancy grid as the sampling point. If there is such a point, it is considered that the point is hit for the first time. The mathematical expression of the voxel first hit is:

[0116]

[0117] That is, in each point cloud frame, the value of the first hit of each voxel is counted. The value of the minimum corresponding voxel, which may be an occlusion voxel.

[0118] It can be understood that, in the voxel space corresponding to each frame of point cloud, the first hit of each voxel is counted. Once the same voxel is also first hit in the subsequent point cloud frame, the value of is incremented. After counting all the point cloud frames, the occupancy probability of the voxel in the occupancy grid is determined.

[0119] If the occupancy probability of the voxel is greater than a threshold value, it indicates that the voxel is a fixed structure (such as a roof, a bracket, etc.) of the carrier, i.e., the voxel is an occlusion voxel.

[0120] In another example, the present embodiment can also directly count the first hit times of each voxel to determine whether the voxel is an occlusion voxel.

[0121] Among them, for each voxel, in each voxel space, the target voxel first hit by the ray corresponding to the ray equation is determined, and the first hit times of each target voxel in each voxel space are counted to obtain the hit times. Based on the hit times, the occlusion voxel is determined among the target voxels; wherein, if the hit times of the voxel are greater than a second preset threshold value, the target voxel is determined as an occlusion voxel. The value of the second preset threshold value can be set according to actual needs, which is not limited here.

[0122] In some examples, an occlusion voxel list can be constructed based on the occlusion voxels in each point cloud frame, and the occlusion voxels in each point cloud frame can be searched and removed according to the occlusion voxel list.

[0123] S140, remove the occlusion voxels in each point cloud frame, and remove the pixel blocks corresponding to the occlusion voxels in each image frame.

[0124] After the occlusion voxels are identified, the occlusion voxels are removed from the continuous multiple frames of point clouds that have been collected, and the pixel blocks corresponding to the occlusion voxels are removed from the continuous multiple frames of images. In addition, the corresponding occlusion voxels and voxel blocks can also be removed from the multiple frames of point clouds and multiple frames of images collected subsequently. The point cloud frames and image frames containing only real environment data are obtained, so as to realize automatic identification and removal of the carrier structure (such as the roof and the bracket), and only the dynamic point cloud and image data of the environment objects are left.

[0125] For example, as shown in Figure 2a , after the original image frame is subjected to occlusion pixel block removal, the image frame as shown in Figure 2bThe shown image frame; obviously, the original image frame has an occlusion part, and the processed image frame only retains the dynamically changing real environment part.

[0126] As Figure 3a shown (the occlusion pixel noise in the block in the figure), after the original point cloud frame is subjected to occlusion pixel block removal, the point cloud frame as Figure 3b shown is obtained; wherein the original point cloud frame contains carrier structure noise, and the processed point cloud frame only retains the point cloud of the real environment.

[0127] In this embodiment, the pixel-level standard deviation is calculated in combination with the image frame and the point cloud frame, and the fusion processing of the image and the point cloud is implemented through the back projection operation to identify the carrier structure noise therefrom, which facilitates the subsequent denoising processing; the entire noise identification process does not require training data and does not depend on any AI model or labeled sample, and is completely processed based on physical dynamic characteristics, thereby saving a large amount of data processing cost; and the adaptability and universality of the entire scheme are relatively strong, and the scheme can be applied to various platforms such as vehicle-mounted, airborne, handheld, etc., and can be applied in real time; and the application range is relatively wide, and the scheme can be applied to multiple fields such as robot environment perception, automatic driving, unmanned aerial vehicle surveying and mapping, AR / VR head-mounted display, etc.

[0128] The embodiment of the present application also provides a self-state estimation method based on dynamic view angle difference, which is used to determine the running state of the carrier in the movement process.

[0129] Exemplarily, as Figure 4 shown, the method comprises the following steps:

[0130] S410, acquiring continuous multiple frames of point clouds and continuous multiple frames of images in the movement process of the carrier, and using a three-dimensional occlusion voxel removal method to identify and remove occlusion voxels in the continuous multiple frames of point clouds, and identify and remove pixel blocks corresponding to the occlusion voxels in the continuous multiple frames of images.

[0131] S420, determining self-state information of the carrier at different time points based on the continuous multiple frames of point clouds and the continuous multiple frames of images subjected to the removal processing.

[0132] It can be understood that, in the embodiment, the three-dimensional occlusion voxel removal method of the foregoing embodiment is used to remove the carrier structure noise (i.e., the occlusion voxels and the pixel blocks corresponding thereto) from the collected multiple frames of images and multiple frames of point clouds, and then the data fusion is performed on the image frames and the point cloud frames subjected to the denoising processing to identify the state information of the carrier at different time points.

[0133] Since the image frames and the point cloud frames are subjected to fine denoising processing before the state estimation, the interference of the carrier structure noise in the data processing process is eliminated, thereby improving the accuracy and reliability of the subsequent state estimation.

[0134] The embodiment of the present application further provides a three-dimensional occlusion voxel elimination device, which exemplarily comprises:

[0135] The acquisition module is configured to acquire continuous multiple frames of images and continuous multiple frames of point clouds in the carrier movement process.

[0136] The first determination module is configured to calculate, for each pixel block in each frame of image, a pixel intensity variance of each pixel block in each frame of image, so as to determine all stable regions in each frame of image.

[0137] The second determination module is configured to back-project each stable region into a corresponding frame of point cloud, so as to determine occlusion voxels from each frame of point cloud.

[0138] The de-noising module is configured to eliminate the occlusion voxels in each frame of point cloud, and eliminate pixel blocks corresponding to the occlusion voxels in each frame of image.

[0139] It can be understood that the three-dimensional occlusion voxel elimination device of the embodiment corresponds to the three-dimensional occlusion voxel elimination method of the above-mentioned embodiment, and the optional items in the above-mentioned embodiment are also applicable to the present embodiment, and thus will not be described herein again.

[0140] The embodiment of the present application further provides a state estimation device based on dynamic visual angle difference, which exemplarily comprises:

[0141] The de-noising module is configured to acquire continuous multiple frames of point clouds and continuous multiple frames of images in the carrier movement process, and eliminate occlusion voxels in the continuous multiple frames of point clouds and pixel blocks corresponding to the occlusion voxels in the continuous multiple frames of images by using a three-dimensional occlusion voxel elimination method.

[0142] The state estimation module is configured to determine self-state information of the carrier at different time based on the continuous multiple frames of point clouds and the continuous multiple frames of images after the elimination processing.

[0143] It can be understood that the state estimation device based on dynamic visual angle difference of the embodiment corresponds to the state estimation method based on dynamic visual angle difference of the above-mentioned embodiment, and the optional items in the above-mentioned embodiment are also applicable to the present embodiment, and thus will not be described herein again.

[0144] The present application further provides a carrier device, which exemplarily comprises a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program, so that the carrier device executes the above-mentioned three-dimensional occlusion voxel elimination method, or the self-state estimation method based on dynamic visual angle difference, or the functions of each module in the above-mentioned three-dimensional occlusion voxel elimination device, or the functions of each module in the above-mentioned self-state estimation device based on dynamic visual angle difference.

[0145] The processor can be an integrated circuit chip with signal processing capability. The processor can be a general purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), and a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or at least one of the above. The general purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application.

[0146] The memory can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor can execute the computer program accordingly after receiving an execution instruction.

[0147] It should be understood that all the functional modules or units in the embodiments of the present application can be integrated or can exist alone, and two or more functional modules or units can be integrated to form a single functional module or unit.

[0148] In addition, each functional module or unit in the embodiments of the present application can be integrated together to form a separate part, or each functional module can exist alone, or two or more functional modules can be integrated to form a separate part.

[0149] The functions described above if realized in the form of software functional modules and sold or used as independent products can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a plurality of instructions for causing a carrier device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application.

[0150] The above describes only the specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application.

Claims

1. A three-dimensional occluded voxel culling method, characterized in that: include: Acquire continuous multi-frame images and continuous multi-frame point clouds during the carrier's motion; For each pixel block in each image frame, calculating the pixel intensity variance of each pixel block in each image frame to determine all stable areas in each image frame; Back-projecting each of the stable regions into a corresponding point cloud frame to determine occluded voxels from each of the point cloud frames; Eliminating the occluding voxels in each of the point cloud frames, and eliminating the pixel blocks corresponding to the occluding voxels in each of the image frames; The back-projecting of each stable region into a corresponding point cloud frame to determine occluded voxels from each point cloud frame includes: Based on a transformation relationship between the radar coordinate system and the camera coordinate system, each of the stable regions is converted into a first unit vector in the camera coordinate system, and the first unit vector is converted into a second unit vector in the radar coordinate system; constructing a ray equation based on the second unit vector; Each frame of point cloud is converted into a regular cube network to obtain a plurality of voxel spaces; the voxel space contains a plurality of voxels; analyzing the passing information of the ray corresponding to the ray equation passing through the voxels in each voxel space, and determining the blocked voxels in each voxel according to the passing information; The analyzing the passing information of the ray corresponding to the ray equation passing through the voxels in each voxel space, and determining the occluded voxels in each voxel according to the passing information, includes: For each voxel, within each voxel space, counting the number of times the ray corresponding to the ray equation passes through each voxel after being emitted from the endpoint and does not pass through each voxel, and the number of hits when passing through each voxel for the first time; constructing an occupancy grid for each voxel, and updating a value of the occupancy grid according to the number of paths and the number of hits of each voxel; determining occluded voxels from each of the voxels based on a value of each of the occupancy grids; The analyzing the passing information of the ray corresponding to the ray equation passing through the voxels in each voxel space, and determining the occluded voxels in each voxel according to the passing information, includes: For each voxel, in each voxel space, determining the target voxel that the ray corresponding to the ray equation passes through for the first time after being emitted from the endpoint, and counting the number of first passes of each target voxel in each voxel space to obtain a hit count; Based on the number of hits, an occluding voxel is determined in each of the target voxels; wherein, if the number of hits is greater than a second preset threshold, the target voxel is determined to be an occluding voxel.

2. The three-dimensional occlusion voxel culling method according to claim 1, characterized in that: The determining of occluded voxels from the voxels based on the values ​​of the occupied grids includes: For each of the occupied grids, calculating the occupancy probability of the hit count within the occupied grid; If the occupancy probability is greater than a first preset threshold, the voxel corresponding to the occupancy grid is determined to be an occluded voxel.

3. The three-dimensional occlusion voxel culling method according to claim 1, characterized in that: Calculating the pixel intensity variance of each pixel block in each image frame to determine all stable areas in each image frame includes: Calculating a temporal mean of each pixel block; constructing a pixel intensity covariance matrix based on the temporal mean; Based on the pixel intensity variance of each pixel block in each image frame in the pixel intensity covariance matrix, a stable region in each image frame is determined.

4. The three-dimensional occlusion voxel culling method according to claim 1 or 3, characterized in that: Calculating the pixel intensity variance of each pixel block in each image frame to determine all stable areas in each image frame includes: For a single image frame, if the pixel intensity variance is less than a third preset threshold, determining a pixel block corresponding to the pixel intensity variance as a stable area in the image frame; And / or, after determining all stable areas in each of the image frames, the method further includes: Binary image marking is performed on the stable area and the unstable area in each image frame to obtain a binary mask image.

5. A state estimation method based on dynamic view difference, characterized in that: include: Acquire continuous multi-frame point clouds and continuous multi-frame images during the movement of the carrier, and use the three-dimensional occlusion voxel culling method according to any one of claims 1 to 4 to identify and eliminate occlusion voxels in the continuous multi-frame point clouds, and identify and eliminate pixel blocks corresponding to the occlusion voxels in the continuous multi-frame images; Based on the continuous multi-frame point clouds and the continuous multi-frame images after the elimination process, the state information of the carrier at different times is determined.

6. A carrier device, characterized in that: The carrier device includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the three-dimensional occlusion voxel culling method according to any one of claims 1 to 4 or the state estimation method based on dynamic perspective difference according to claim 5.

7. A computer storage medium, characterized in that The computer program is stored therein, and when the computer program is executed on a processor, the computer program implements the three-dimensional occlusion voxel culling method according to any one of claims 1 to 4 or the state estimation method based on dynamic view difference according to claim 5.

Images