A method and related apparatus for generating conditional images based on point rasterization

Abstract

The present application provides a kind of conditional image generation method and related device based on point rasterization, according to the environment image corresponding to target time, target point cloud set is colored processing;The target point cloud set after coloring processing is carried out front and background separation processing, to determine the dynamic foreground region and static background region therein;Combining the historical point cloud set in time window, the incomplete area is completed;Using perspective projection mapping relationship, the target point cloud set after completion is projected to the image coordinate system of vehicle, to obtain first projection image;The raster area in first projection image is rendered, to obtain the conditional image of point rasterization.It is analyzed frame by frame to original laser radar point cloud data and image plane mapping, and combined with point rasterization processing, constructs the geometric conditional image with pixel level precision, provides structure priori support for subsequent new view synthesis and controllable video generation.

Description

Technical Field

[0001] This invention relates to the field of images, and more specifically, to a conditional image generation method and related apparatus based on point rasterization. Background Technology

[0002] With the deep integration of artificial intelligence and digital twin technology, the accuracy and real-time performance of environmental perception and scene modeling have become core elements for improving the reliability of autonomous driving systems. Especially in dynamic urban scenarios with dense traffic participants, variable lighting, and frequent perspective shifts, achieving efficient fusion of multimodal data and instance-level spatiotemporal consistency remains a key bottleneck restricting the implementation of autonomous driving technology. Vector maps, as the core carrier connecting environmental perception and decision-making, directly impact the depth of understanding and responsiveness of autonomous driving systems to complex scenarios.

[0003] Currently, vector maps can be generated using neural network models or deep learning models. However, the quality of generated vector maps heavily depends on the model's training performance, which requires conditional images. Therefore, how to quickly and efficiently generate high-quality conditional images has become one of the key challenges for those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a conditional image generation method and related apparatus based on point rasterization to improve the above-mentioned problems.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0006] In a first aspect, embodiments of the present invention provide a conditional image generation method based on point rasterization, the method comprising:

[0007] The target point cloud set is colored based on the environmental image corresponding to the target time.

[0008] The target point cloud set is the set of point cloud data collected by the lidar deployed on the vehicle at the target time, and the coloring process refers to adding the color value corresponding to the environment image to each point in the target point cloud set.

[0009] The target point cloud set after coloring is subjected to foreground-background separation processing to determine the dynamic foreground region and the static background region.

[0010] By combining the historical point cloud set within the time window, the incomplete area is completed, wherein the incomplete area is a dynamic foreground area in the target point cloud set that is occluded and incomplete;

[0011] Using perspective projection mapping, the completed target point cloud set is projected onto the vehicle's image coordinate system to obtain the first projected image. The perspective projection mapping relationship is the mapping relationship between the lidar coordinate system and the vehicle's image coordinate system.

[0012] The raster region in the first projected image is rendered to obtain a point rasterized conditional image.

[0013] In a second aspect, embodiments of the present invention provide a conditional image generation apparatus based on dot rasterization, the apparatus comprising:

[0014] The first processing unit is used to perform coloring processing on the target point cloud set according to the environmental image corresponding to the target time.

[0015] The target point cloud set is the set of point cloud data collected by the lidar deployed on the vehicle at the target time, and the coloring process refers to adding the color value corresponding to the environment image to each point in the target point cloud set.

[0016] The first processing unit is also used to perform foreground-background separation processing on the colored target point cloud set to determine the dynamic foreground region and static background region therein;

[0017] The first processing unit is also used to combine the historical point cloud set within the time window to complete the incomplete region, wherein the incomplete region is a dynamic foreground region in the target point cloud set that has occlusion and incompleteness;

[0018] The first processing unit is further configured to use perspective projection mapping relationship to project the completed target point cloud set onto the vehicle's image coordinate system to obtain a first projected image, wherein the perspective projection mapping relationship is the mapping relationship between the lidar coordinate system and the vehicle's image coordinate system;

[0019] The second processing unit is used to render the raster region in the first projected image to obtain a point rasterized conditional image.

[0020] Thirdly, embodiments of the present invention provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0021] Fourthly, embodiments of the present invention provide an electronic device, the electronic device comprising: a processor and a memory, the memory being used to store one or more programs; when the one or more programs are executed by the processor, the above-described method is implemented.

[0022] Compared to existing technologies, the present invention provides a conditional image generation method and related apparatus based on point rasterization. The method involves coloring a target point cloud set according to the environmental image corresponding to the target time. The target point cloud set is the set of point cloud data collected by a LiDAR deployed on a vehicle at the target time. Coloring refers to adding the color value corresponding to the environmental image to each point in the target point cloud set. The colored target point cloud set undergoes foreground-background separation processing to determine the dynamic foreground region and static background region. Incomplete regions are filled in using historical point cloud sets within a time window. The incomplete regions are dynamic foreground regions in the target point cloud set that are occluded or incomplete. The completed target point cloud set is projected onto the vehicle's image coordinate system using perspective projection mapping to obtain a first projected image. The perspective projection mapping relationship is the mapping relationship between the LiDAR coordinate system and the vehicle's image coordinate system. The raster regions in the first projected image are rendered to obtain a point rasterized conditional image. By performing frame-by-frame analysis and image plane mapping on the original LiDAR point cloud data, and combining it with point rasterization processing, a geometric condition image with pixel-level accuracy is constructed, providing structural prior support for subsequent new view synthesis and controllable video generation.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0026] Figure 2 This is a flowchart illustrating the conditional image generation method based on point rasterization provided in an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of a unit of a conditional image generation device based on point rasterization provided in an embodiment of the present invention.

[0028] In the diagram: 10-Processor; 11-Memory; 12-Bus; 13-Communication interface; 601-First processing unit; 602-Second processing unit. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0030] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0031] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0033] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0034] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0035] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0036] This invention provides an electronic device, which may be a vehicle computer or a server. Please refer to... Figure 1 This is a schematic diagram of the structure of an electronic device. The electronic device includes a processor 10, a memory 11, and a bus 12. The processor 10 and the memory 11 are connected via the bus 12. The processor 10 is used to execute executable modules, such as computer programs, stored in the memory 11.

[0037] Processor 10 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the point rasterization-based conditional image generation method can be completed through integrated logic circuits in the hardware or software instructions within processor 10. Processor 10 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0038] The memory 11 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0039] Bus 12 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. Figure 1 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus 12 or one type of bus 12.

[0040] The memory 11 is used to store programs, such as programs corresponding to a point rasterization-based conditional image generation device. The point rasterization-based conditional image generation device includes at least one software functional module that can be stored in the memory 11 in the form of software or firmware, or embedded in the operating system (OS) of the electronic device. Upon receiving an execution instruction, the processor 10 executes the program to implement the point rasterization-based conditional image generation method.

[0041] The electronic device provided in this embodiment of the invention may further include a communication interface 13. The communication interface 13 is connected to the processor 10 via a bus.

[0042] It should be understood that, Figure 1 The structure shown is only a partial schematic diagram of the electronic device; the electronic device may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0043] The conditional image generation method based on dot rasterization provided in this invention can be applied to, but is not limited to, applications such as... Figure 1 For the specific process of the electronic devices shown, please refer to [link / reference]. Figure 2 The conditional image generation method based on point rasterization includes S10, S20, S30, S40 and S50, which are described in detail below.

[0044] S10: Color the target point cloud set according to the environmental image corresponding to the target time.

[0045] The target point cloud set is the set of point cloud data collected by the LiDAR deployed on the vehicle at the target time. The coloring process refers to adding the color value corresponding to the environment image to each point in the target point cloud set. The environment image is the two-dimensional visual image acquired by the vehicle's image acquisition system at the target time, and the environment image includes the color value of each pixel.

[0046] This coloring process not only preserves the geometric realism of the scene structure, but also introduces texture information to enhance the perceptual quality of the conditional image, effectively improving the ability of subsequent vector map generation models to identify geometric structures and semantic regions.

[0047] Optionally, in step S10, the target point cloud set is colored according to the environmental image corresponding to the target time, including steps S11, S12, S13 and S14, which are described in detail below.

[0048] S11, using perspective projection mapping, the target point cloud set is projected onto the vehicle's image coordinate system to obtain the second projected image.

[0049] The perspective projection mapping relationship refers to the mapping relationship between the lidar coordinate system and the vehicle's image coordinate system. A second projected image is obtained through projection, completing the mapping from a three-dimensional spatial structure to a two-dimensional image structure.

[0050] In some alternative implementations, the perspective projection mapping relationship can be determined by using the intrinsic and extrinsic parameter models of the lidar during the calibration process.

[0051] S12, use the color value of the i-th pixel in the environment image as the color value of the i-th pixel in the second projection image.

[0052] Where 1≤i≤I, and I is the total number of pixels in the environment image.

[0053] It should be noted that, in order to enhance the representation ability of point clouds in the semantic space of images, the environmental image can be an image obtained by multi-source alignment of images captured by multiple cameras on the vehicle at different locations at the target time.

[0054] S13, Based on the perspective projection mapping relationship, determine the corresponding point of the i-th pixel in the second projected image in the target point cloud set.

[0055] S14, add the color value of the i-th pixel in the second projected image to its corresponding point in the target point cloud set.

[0056] Through coloring processing, spatial-image joint encoding is completed. This coloring process not only preserves the geometric realism of the scene structure, but also introduces texture information to enhance the perceptual quality of the conditional image, effectively improving the ability of subsequent vector map generation models to identify geometric structures and semantic regions.

[0057] S20, perform foreground-background separation processing on the colored target point cloud set to determine the dynamic foreground region and static background region.

[0058] To highlight the salience of target objects (moving objects) in dynamic foreground regions within point cloud conditional images, a foreground-background separation mechanism combining geometric segmentation and semantic guidance (annotation) is introduced to perform foreground-background separation processing on the colored target point cloud set. This foreground-background separation process ensures the overall structural coherence of the conditional image while enhancing the expressive power of local regions, enabling the vector map generation model to maintain structural consistency for key target objects even when considering changes in viewpoint.

[0059] Optionally, the environment image is an image with dynamic object annotations (e.g., cars, motorcycles, pedestrians, and bicycles) and static object annotations (e.g., roads, buildings, and trees). Based on this, in step S20, foreground-background separation processing is performed on the colored target point cloud set to determine the dynamic foreground region and static background region, including steps S21, S22, and S23, as follows.

[0060] S21 uses spatial density, depth gradient, and normal vector as clustering reference factors (low-order geometric features) to perform preliminary clustering on the colored target point cloud set to obtain preliminary clustering regions.

[0061] The initial clustering region is the region where the spatial density is greater than the density threshold, the fluctuation value of the depth gradient (i.e., the frequency of change of depth) is greater than the gradient threshold, and the change value of the normal vector is greater than the vector threshold.

[0062] S22, when the proportion of dynamic points in the preliminary clustering region is greater than or equal to a preset ratio, it is determined as a dynamic foreground region.

[0063] In this context, the coordinates of points in the initial clustered region after being projected onto the vehicle's image coordinate system are called projected coordinates. When the pixel point corresponding to the projected coordinates in the environment image is labeled as a dynamic object, the point corresponding to the projected coordinates is called a dynamic point.

[0064] S23. When the proportion of dynamic points in the preliminary clustering region is less than the preset ratio, it is determined to be a static background region.

[0065] By using the annotations of dynamic and static objects in environmental images as weak label semantic cues, multi-scale filtering and label refinement are performed on the initial clustered regions, thereby accurately distinguishing dynamic foreground regions (such as vehicles, pedestrians, bicycles, etc.) from static background regions (such as roads, buildings, trees, etc.).

[0066] S30, combined with the historical point cloud set within the time window, completes the missing areas.

[0067] The time window is a window of preset length between target times, and the incomplete area is a dynamic foreground area in the target point cloud set that is occluded or incomplete.

[0068] Because single-frame LiDAR point cloud sets may suffer from problems such as sparse sampling, severe occlusion, and missing textures, it is difficult to achieve a continuous and complete geometric representation in image space. To address this, this invention introduces a cross-frame point cloud aggregation mechanism to complete the target point cloud set and uniformly construct a dense point cloud that integrates observation information from multiple time points.

[0069] Optionally, S30, combining the historical point cloud set within the time window, completes the incomplete area, including: S31, S32 and S33, as follows.

[0070] S31, combining the vehicle's motion trajectory, compare the foreground dynamic region and background static region of the historical point cloud frame within the time window with the foreground dynamic region and background static region of the target point cloud set to determine the missing region.

[0071] S32, combining pose change reference information and vehicle motion trajectory, obtains supplementary content for the missing area from historical point cloud frames within the time window.

[0072] S33, add the supplementary content to the missing area in the target point cloud set to complete the missing area.

[0073] During the completion of incomplete regions, a time window control strategy is introduced to limit the number of point cloud aggregation frames, avoiding geometric drift caused by long-term trajectory changes. For dynamic foreground regions, a dynamic binding framework for vehicle motion trajectories is constructed, and the foreground point clouds corresponding to the dynamic foreground regions are transformed and aggregated according to the vehicle motion trajectory to ensure their geometric continuity and physical consistency.

[0074] Finally, this stage outputs a dense, structurally complete, and texture-enhanced world-class point cloud ensemble, providing global support for subsequent rendering, image generation, and 3D reconstruction. This point cloud ensemble not only possesses high spatial density but also exhibits strong occlusion robustness and multi-view consistency, demonstrating high adaptability and generalization capabilities in dynamic urban scenes.

[0075] S40, using perspective projection mapping, the completed target point cloud set is projected onto the vehicle's image coordinate system to obtain the first projected image.

[0076] S50, render the raster region in the first projected image to obtain a point rasterized conditional image.

[0077] Among them, the point rasterization conditional image is used as a training reference image for the vector map generation model to improve the training effect of the model and improve the quality of the vector map it generates.

[0078] To overcome the hole phenomenon caused by the sparsity of LiDAR point clouds in image space, this invention provides a conditional image generation method based on point rasterization. Point rasterization allows each 3D point to be projected onto the image plane as a kernel function (such as a Gaussian kernel) with spatial coverage, thereby achieving diffusion of influence on neighboring pixels. This not only preserves the positional accuracy of geometric points but also effectively fills the image structure discontinuity problem caused by projection sparsity.

[0079] Optionally, in step S50, the raster region in the first projected image is rendered to obtain a point rasterized conditional image, including steps S51 and S52, which are described in detail below.

[0080] S51, determine the grating region in the first projected image.

[0081] Optionally, in step S51, the raster region in the first projected image is determined, including steps S511, S512, S513, and S514, as follows.

[0082] S511, combined with the pose information of the LiDAR at the target moment, constructs a target frustum model.

[0083] By cropping the target point cloud set according to the target view frustum model, we can obtain the points that fall into the target view frustum model.

[0084] S512, project the raster points onto the vehicle's image coordinate system to obtain raster pixels, where the raster points are the points in the target point cloud set that fall into the target view frustum model.

[0085] S513 constructs a two-dimensional Gaussian kernel at the position of the grating pixel and determines the scale range of the two-dimensional Gaussian kernel based on the depth of the grating pixel.

[0086] The scale of this two-dimensional Gaussian kernel is adaptively adjusted by the depth of the point on the image plane. The scale range corresponding to the near area (small depth) is larger to enhance detail coverage, while the scale range corresponding to the far area (large depth) is reduced to preserve boundary accuracy.

[0087] S514 uses the region within the scale range corresponding to the two-dimensional Gaussian kernel as the grating region.

[0088] S52 takes the RGB, depth, point density, and normal vector corresponding to the raster region as the rendering target, and performs weighted fusion on a unit of raster region to complete the raster region rendering, thereby generating a conditional image that integrates geometric, texture, and shape information.

[0089] By rendering the raster region in the first projected image, the final output point rasterized conditional image is significantly better than the traditional point cloud visualization results in terms of resolution, spatial consistency and information density. It can be directly input into the vector map generation model as a high-quality prior condition and play a key role in controlling viewpoint changes and lighting optimization.

[0090] In an alternative implementation, after completing the missing region by combining the historical point cloud set within the time window, the conditional image generation method based on point rasterization further includes: S34, as follows.

[0091] S34 introduces a depth consistency filtering mechanism, which, based on the joint constraints of Euclidean distance and angular similarity, suppresses and deletes redundant points in the repeated regions of the completed target point cloud set, while preserving structurally stable regional information.

[0092] Optionally, to further enhance the expressive power of the generated image in structural and semantic space, the conditional image generation method based on point rasterization provided in this embodiment of the invention introduces a multi-scale processing mechanism in the conditional image generation process. Specifically, when rendering the raster region in the first projected image to obtain the point rasterized conditional image, i.e., the point rasterization output stage, multiple resolution-level image versions (such as 1×, 1 / 2×, 1 / 4×) are output simultaneously, corresponding to the requirements of detail enhancement, structural reconstruction, and semantic alignment, respectively. The multi-scale images are fused as a conditional pyramid in the subsequent video generation module to improve the network's ability to model image content at different levels.

[0093] Optionally, to maintain temporal consistency and semantic continuity of target tracking, the conditional image generation method based on point rasterization provided in this embodiment of the invention also incorporates a cross-frame semantic calibration mechanism. This mechanism, based on optical flow estimation and semantic propagation algorithms, ensures that the geometric structure and semantic distribution of the same target remain stable under different viewpoints, thereby avoiding image jitter, structural drift, and semantic mismatch caused by viewpoint shifts or occlusions.

[0094] Please see Figure 3 , Figure 3 The present invention provides a conditional image generation apparatus based on point rasterization, which is optionally applied to the electronic device described above.

[0095] The conditional image generation apparatus based on point rasterization includes: a first processing unit 601 and a second processing unit 602.

[0096] The first processing unit 601 is used to perform coloring processing on the target point cloud set according to the environmental image corresponding to the target time.

[0097] Among them, the target point cloud set is the set of point cloud data collected by the LiDAR deployed on the vehicle at the target time, and the coloring process refers to adding the color value corresponding to the environment image to each point in the target point cloud set.

[0098] The first processing unit 601 is also used to perform foreground-background separation processing on the target point cloud set after coloring processing, so as to determine the dynamic foreground region and the static background region therein;

[0099] The first processing unit 601 is also used to combine the historical point cloud set within the time window to complete the incomplete region, wherein the incomplete region is a dynamic foreground region in the target point cloud set that has occlusion and incompleteness.

[0100] The first processing unit 601 is also used to project the completed target point cloud set onto the vehicle's image coordinate system using the perspective projection mapping relationship to obtain the first projected image, wherein the perspective projection mapping relationship is the mapping relationship between the lidar coordinate system and the vehicle's image coordinate system.

[0101] The second processing unit 602 is used to render the raster region in the first projected image to obtain a point rasterized conditional image.

[0102] It should be noted that the conditional image generation apparatus based on dot rasterization provided in this embodiment can execute the method flow shown in the above method flow embodiment to achieve the corresponding technical effects. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0103] This invention also provides a storage medium storing computer instructions and programs that, when read and executed, perform the point rasterization-based conditional image generation method described above. The storage medium may include memory, flash memory, registers, or a combination thereof.

[0104] The following describes an electronic device, which can be a vehicle computer or a server device. Figure 1 As shown, the above-described conditional image generation method based on point rasterization can be implemented. Specifically, the electronic device includes: a processor 10, a memory 11, and a bus 12. The processor 10 may be a CPU. The memory 11 is used to store one or more programs, which, when executed by the processor 10, execute the conditional image generation method based on point rasterization described in the above embodiment.

[0105] In summary, the present invention provides a conditional image generation method and related apparatus based on point rasterization. The method involves coloring a target point cloud set based on the environmental image corresponding to the target time. The target point cloud set is the set of point cloud data collected by a LiDAR deployed on a vehicle at the target time. Coloring refers to adding the color value corresponding to the environmental image to each point in the target point cloud set. The colored target point cloud set undergoes foreground-background separation processing to determine the dynamic foreground region and static background region. Combined with historical point cloud sets within a time window, incomplete regions are filled in, where the incomplete regions are dynamic foreground regions in the target point cloud set that are occluded or incomplete. Using perspective projection mapping, the filled target point cloud set is projected onto the vehicle's image coordinate system to obtain a first projected image, where the perspective projection mapping is the mapping relationship between the LiDAR coordinate system and the vehicle's image coordinate system. The raster region in the first projected image is rendered to obtain a point rasterized conditional image. By performing frame-by-frame analysis and image plane mapping on the original LiDAR point cloud data, and combining it with point rasterization processing, a geometric condition image with pixel-level accuracy is constructed, providing structural prior support for subsequent new view synthesis and controllable video generation.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0107] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A conditional image generation method based on point rasterization, characterized in that, The method includes: The target point cloud set is colored based on the environmental image corresponding to the target time. The target point cloud set is the set of point cloud data collected by the lidar deployed on the vehicle at the target time, and the coloring process refers to adding the color value corresponding to the environment image to each point in the target point cloud set. The target point cloud set after coloring is subjected to foreground-background separation processing to determine the dynamic foreground region and the static background region. The missing region is filled by combining the historical point cloud set within the time window, including: comparing the foreground dynamic region and background static region of the historical point cloud frame within the time window with the foreground dynamic region and background static region of the target point cloud set based on the vehicle motion trajectory to determine the missing region; obtaining supplementary content for the missing region from the historical point cloud frame within the time window based on pose change reference information and the vehicle motion trajectory; and adding the supplementary content to the missing region in the target point cloud set to complete the missing region filling, wherein the missing region is a dynamic foreground region in the target point cloud set that is occluded and incomplete. A depth consistency filtering mechanism is introduced, which suppresses and deletes redundant points in the repeated regions of the completed target point cloud set based on the joint constraints of Euclidean distance and angular similarity. Using perspective projection mapping, the completed target point cloud set is projected onto the vehicle's image coordinate system to obtain the first projected image. The perspective projection mapping relationship is the mapping relationship between the lidar coordinate system and the vehicle's image coordinate system. The raster region in the first projected image is rendered to obtain a point rasterized conditional image; The environmental image is an image with both dynamic and static object annotations. The foreground-background separation process performed on the colored target point cloud set to determine the dynamic foreground and static background regions includes: using spatial density, depth gradient, and normal vector as clustering reference factors to perform preliminary clustering on the colored target point cloud set to obtain preliminary clustered regions; wherein, the preliminary clustered regions are regions with spatial density greater than a density threshold, depth gradient fluctuation values ​​greater than a gradient threshold, and normal vector change values ​​greater than a vector threshold; when the proportion of dynamic points in the preliminary clustered regions is greater than or equal to a preset ratio, it is determined to be a dynamic foreground region; wherein, the coordinates of points in the preliminary clustered regions projected onto the vehicle's image coordinate system are projected coordinates, and when the pixel point corresponding to the projected coordinates in the environmental image is labeled as a dynamic object, the point corresponding to the projected coordinates is a dynamic point; when the proportion of dynamic points in the preliminary clustered regions is less than a preset ratio, it is determined to be a static background region.

2. The conditional image generation method based on point rasterization as described in claim 1, characterized in that, The step of coloring the target point cloud set based on the environmental image corresponding to the target time includes: Using perspective projection mapping, the target point cloud set is projected onto the vehicle's image coordinate system to obtain a second projected image; The color value of the i-th pixel in the environmental image is used as the color value of the i-th pixel in the second projected image; Based on the perspective projection mapping relationship, determine the corresponding point of the i-th pixel in the second projection image in the target point cloud set; Add the color value of the i-th pixel in the second projected image to its corresponding point in the target point cloud set.

3. The conditional image generation method based on point rasterization as described in claim 1, characterized in that, The step of rendering the raster region in the first projected image to obtain a point rasterized conditional image includes: Determine the raster region in the first projected image; The RGB, depth, point density, and normal vector corresponding to the raster region are used as the rendering target, and weighted fusion is performed on the raster region as a unit to complete the raster region rendering.

4. The conditional image generation method based on point rasterization as described in claim 3, characterized in that, Determining the raster region in the first projected image includes: By combining the pose information of the target at the time of the LiDAR, a target frustum model is constructed; The raster points are projected onto the vehicle's image coordinate system to obtain raster pixels, wherein the raster points are points in the target point cloud set that fall into the target view frustum model; A two-dimensional Gaussian kernel is constructed at the location of the grating pixel, and the scale range of the two-dimensional Gaussian kernel is determined based on the depth of the grating pixel; The region within the scale range corresponding to the two-dimensional Gaussian kernel is taken as the grating region.

5. A conditional image generation device based on dot rasterization, characterized in that, The device includes: The first processing unit is used to perform coloring processing on the target point cloud set according to the environmental image corresponding to the target time. The target point cloud set is the set of point cloud data collected by the lidar deployed on the vehicle at the target time, and the coloring process refers to adding the color value corresponding to the environment image to each point in the target point cloud set. The first processing unit is also used to perform foreground-background separation processing on the colored target point cloud set to determine the dynamic foreground region and static background region therein; The first processing unit is further configured to complete the missing region by combining the historical point cloud set within the time window, including: comparing the foreground dynamic region and background static region of the historical point cloud frame within the time window with the foreground dynamic region and background static region of the target point cloud set based on the vehicle motion trajectory to determine the missing region; obtaining supplementary content of the missing region from the historical point cloud frame within the time window based on pose change reference information and the vehicle motion trajectory; and adding the supplementary content to the missing region in the target point cloud set to complete the missing region completion, wherein the missing region is a dynamic foreground region in the target point cloud set that is occluded and incomplete; The conditional image generation device based on point rasterization is also used to introduce a depth consistency filtering mechanism, which suppresses and deletes redundant points in the repeated regions of the completed target point cloud set based on the joint constraints of Euclidean distance and angular similarity. The first processing unit is further configured to use perspective projection mapping relationship to project the completed target point cloud set onto the vehicle's image coordinate system to obtain a first projected image, wherein the perspective projection mapping relationship is the mapping relationship between the lidar coordinate system and the vehicle's image coordinate system; The second processing unit is used to render the raster region in the first projected image to obtain a point rasterized conditional image. The environmental image is an image with both dynamic and static object annotations. The foreground-background separation process performed on the colored target point cloud set to determine the dynamic foreground and static background regions includes: using spatial density, depth gradient, and normal vector as clustering reference factors to perform preliminary clustering on the colored target point cloud set to obtain preliminary clustered regions; wherein, the preliminary clustered regions are regions with spatial density greater than a density threshold, depth gradient fluctuation values ​​greater than a gradient threshold, and normal vector change values ​​greater than a vector threshold; when the proportion of dynamic points in the preliminary clustered regions is greater than or equal to a preset ratio, it is determined to be a dynamic foreground region; wherein, the coordinates of points in the preliminary clustered regions projected onto the vehicle's image coordinate system are projected coordinates, and when the pixel point corresponding to the projected coordinates in the environmental image is labeled as a dynamic object, the point corresponding to the projected coordinates is a dynamic point; when the proportion of dynamic points in the preliminary clustered regions is less than a preset ratio, it is determined to be a static background region.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-4.

7. An electronic device, characterized in that, include: Processor and memory, the memory being used to store one or more programs; When the one or more programs are executed by the processor, the method as described in any one of claims 1-4 is implemented.

Images