Conditional image generation method based on point rasterization and related device

Abstract

The invention provides a condition image generation method based on point rasterization and a related device, and the method comprises the steps: carrying out the coloring processing of a target point cloud set according to an environment image corresponding to a target moment; performing foreground and background separation processing on the target point cloud set after coloring processing to determine a dynamic foreground region and a static background region in the target point cloud set; complementing the incomplete area in combination with a historical point cloud set in the time window; projecting the complemented target point cloud set to an image coordinate system of the vehicle by using a perspective projection mapping relation to obtain a first projection image; and rendering the grating area in the first projection image to obtain a point rasterization condition image. According to the method, frame-by-frame analysis and image plane mapping are carried out on original laser radar point cloud data, point rasterization processing is combined, a geometric condition image with pixel-level precision is constructed, and structural prior support is provided for subsequent new view synthesis and controllable video generation.

Description

Technical Field

[0001] The present invention relates to the field of images, and in particular to a conditional image generation method based on point rasterization and a related device. Background Art

[0002] With the deep integration of artificial intelligence and digital twin technology, the accuracy and real-time performance of environmental perception and scene modeling have become core elements for improving the reliability of autonomous driving systems. Especially in dynamic urban scenarios with dense traffic, variable lighting, and frequent perspective switching, the efficient fusion of multimodal data and instance-level spatiotemporal consistency remain key bottlenecks hindering the implementation of autonomous driving technology. As the core vehicle connecting environmental perception and decision-making planning, the quality of vector maps directly impacts the autonomous driving system's depth of understanding and responsiveness to complex scenarios.

[0003] Currently, vector maps can be generated using neural network models or deep learning models. However, the quality of these generated vector maps is highly dependent on the effectiveness of model training, which requires the use of conditional images. Therefore, how to quickly and efficiently generate high-quality conditional images has become a major challenge for those skilled in the art. Summary of the Invention

[0004] The object of the present invention is to provide a conditional image generation method based on point rasterization and a related device to improve the above-mentioned problem.

[0005] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for generating a conditional image based on point rasterization, the method comprising:

[0007] Coloring the target point cloud set according to the environment image corresponding to the target moment;

[0008] The target point cloud set is a point cloud data set collected by a laser radar deployed on the vehicle at the target time, and the coloring process refers to adding a color value corresponding to each point in the target point cloud set in the environment image;

[0009] Perform foreground and background separation on the target point cloud set after coloring to determine the dynamic foreground area and static background area;

[0010] Combine the historical point cloud set within the time window to complete the incomplete area, where the incomplete area is the dynamic foreground area in the target point cloud set that is occluded and incomplete;

[0011] Projecting the completed target point cloud set into the vehicle's image coordinate system using a perspective projection mapping relationship to obtain a first projected image, wherein the perspective projection mapping relationship is a mapping relationship between the lidar coordinate system and the vehicle's image coordinate system;

[0012] The raster area in the first projection image is rendered to obtain a point rasterized conditional image.

[0013] In a second aspect, an embodiment of the present invention provides a conditional image generation device based on point rasterization, the device comprising:

[0014] A first processing unit is used to perform coloring processing on the target point cloud set according to the environment image corresponding to the target moment;

[0015] The target point cloud set is a point cloud data set collected by a laser radar deployed on the vehicle at the target time, and the coloring process refers to adding a color value corresponding to each point in the target point cloud set in the environment image;

[0016] The first processing unit is further configured to perform foreground-background separation processing on the target point cloud set after the coloring processing to determine a dynamic foreground area and a static background area therein;

[0017] The first processing unit is further configured to complete the incomplete area by combining the historical point cloud set within the time window, wherein the incomplete area is a dynamic foreground area in the target point cloud set that is occluded and incomplete;

[0018] The first processing unit is further configured to project the completed target point cloud set into the vehicle's image coordinate system using a perspective projection mapping relationship to obtain a first projection image, wherein the perspective projection mapping relationship is a mapping relationship between the lidar coordinate system and the vehicle's image coordinate system;

[0019] The second processing unit is configured to render the raster area in the first projection image to obtain a point rasterized conditional image.

[0020] In a third aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, which implements the above method when executed by a processor.

[0021] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store one or more programs; when the one or more programs are executed by the processor, the above method is implemented.

[0022] Compared with the prior art, the embodiment of the present invention provides a conditional image generation method and related device based on point rasterization, which performs coloring processing on the target point cloud set according to the environmental image corresponding to the target moment; wherein the target point cloud set is a point cloud data set collected by the laser radar deployed on the vehicle at the target moment, and the coloring processing refers to adding the color value corresponding to each point in the target point cloud set in the environmental image; the target point cloud set after coloring is subjected to foreground and background separation processing to determine the dynamic foreground area and the static background area therein; the incomplete area is completed in combination with the historical point cloud set within the time window, wherein the incomplete area is the dynamic foreground area in the target point cloud set that is occluded and incomplete; the completed target point cloud set is projected to the image coordinate system of the vehicle using the perspective projection mapping relationship to obtain a first projection image, wherein the perspective projection mapping relationship is the mapping relationship between the laser radar coordinate system and the image coordinate system of the vehicle; the raster area in the first projection image is rendered to obtain a point rasterized conditional image. By performing frame-by-frame parsing and image plane mapping on the original lidar point cloud data, and combining it with point rasterization processing, a geometric conditional image with pixel-level accuracy is constructed, providing structural prior support for subsequent new view synthesis and controllable video generation.

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0026] Figure 2 A schematic flow chart of a method for generating a conditional image based on point rasterization provided in an embodiment of the present invention.

[0027] Figure 3 A schematic diagram of units of a conditional image generation device based on point rasterization provided by an embodiment of the present invention.

[0028] In the figure: 10 - processor; 11 - memory; 12 - bus; 13 - communication interface; 601 - first processing unit; 602 - second processing unit. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0030] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0031] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are used only to distinguish the description and should not be understood as indicating or implying relative importance.

[0032] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0033] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they should not be understood as limiting the present invention.

[0034] In the description of the present invention, it should also be noted that, unless otherwise expressly specified or limited, the terms "disposed" and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections, or electrical connections; direct connections, indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0035] The following embodiments of the present invention are described in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0036] An embodiment of the present invention provides an electronic device, which may be a vehicle computer device or a server device. Figure 1 , a schematic diagram of the structure of an electronic device. The electronic device includes a processor 10, a memory 11, and a bus 12. The processor 10 and the memory 11 are connected via the bus 12. The processor 10 is used to execute executable modules stored in the memory 11, such as computer programs.

[0037] The processor 10 can be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the conditional image generation method based on point rasterization can be completed by the hardware integrated logic circuit in the processor 10 or the instructions in the form of software. The above-mentioned processor 10 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0038] The memory 11 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0039] The bus 12 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. Figure 1 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus 12 or one type of bus 12.

[0040] The memory 11 is used to store programs, such as a program corresponding to a point rasterization-based conditional image generation device. The point rasterization-based conditional image generation device includes at least one software functional module that can be stored in the memory 11 in the form of software or firmware, or embedded in the operating system (OS) of the electronic device. Upon receiving an execution instruction, the processor 10 executes the program to implement the point rasterization-based conditional image generation method.

[0041] Possibly, the electronic device provided by the embodiment of the present invention further includes a communication interface 13. The communication interface 13 is connected to the processor 10 via a bus.

[0042] It should be understood that Figure 1 The structure shown is only a schematic diagram of a portion of the electronic device. The electronic device may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0043] The present invention provides a method for generating conditional images based on point rasterization, which can be applied to, but is not limited to, Figure 1 For detailed procedures, please refer to the electronic equipment shown in Figure 2 The conditional image generation method based on point rasterization includes: S10, S20, S30, S40 and S50, which are specifically described as follows.

[0044] S10, performing coloring processing on the target point cloud set according to the environment image corresponding to the target moment.

[0045] Among them, the target point cloud set is the point cloud data set collected by the laser radar deployed on the vehicle at the target time. The coloring processing refers to adding the corresponding color value in the environmental image to each point in the target point cloud set. The environmental image is the two-dimensional visual image obtained by the vehicle's image acquisition system at the target time, and the environmental image includes the color value of each pixel.

[0046] This colorization process not only preserves the geometric authenticity of the scene structure, but also introduces texture information to enhance the perceptual quality of the conditional image, effectively improving the subsequent vector map generation model's ability to recognize geometric structures and semantic areas.

[0047] Optionally, S10, coloring the target point cloud set according to the environment image corresponding to the target moment, includes: S11, S12, S13 and S14, which are specifically described as follows.

[0048] S11 , using a perspective projection mapping relationship, projecting the target point cloud set into the vehicle's image coordinate system to obtain a second projected image.

[0049] The perspective projection mapping relationship is the mapping relationship between the LiDAR coordinate system and the vehicle's image coordinate system. The second projection image is obtained by projection, completing the mapping from the three-dimensional space structure to the two-dimensional image structure.

[0050] In some optional implementations, the perspective projection mapping relationship may be determined through the internal reference model and the external reference model of the laser radar during the calibration process.

[0051] S12: Using the color value of the i-th pixel in the environment image as the color value of the i-th pixel in the second projection image.

[0052] Wherein, 1≤i≤I, I is the total number of pixels in the environment image.

[0053] It should be noted that, in order to enhance the representation capability of the point cloud in the image semantic space, the environment image may be an image obtained by multi-source alignment of images captured by multiple cameras in different orientations on the vehicle at the target moment.

[0054] S13: Determine the corresponding point of the i-th pixel point in the second projection image in the target point cloud set according to the perspective projection mapping relationship.

[0055] S14: Add the color value of the i-th pixel in the second projection image to its corresponding point in the target point cloud set.

[0056] Through shading, joint spatial-image encoding is achieved. This shading process not only preserves the geometric authenticity of the scene structure, but also introduces texture information to enhance the perceptual quality of the conditional image, effectively improving the subsequent vector map generation model's ability to identify geometric structures and semantic regions.

[0057] S20, performing foreground-background separation processing on the target point cloud set after the coloring processing to determine the dynamic foreground area and the static background area therein.

[0058] To highlight the saliency of target objects (moving objects) in dynamic foreground regions within the point cloud conditional image, a foreground-background separation mechanism based on geometric segmentation and semantic guidance (annotation) is introduced. This separation is performed on the colorized target point cloud set. This separation process not only ensures the overall structural coherence of the conditional image but also enhances the expressiveness of local regions, enabling the raw vector map generation model to maintain structural consistency for key target objects while focusing on perspective changes.

[0059] Optionally, the environment image is an image that has been annotated with dynamic objects (e.g., cars, motorcycles, pedestrians, and bicycles) and static objects (e.g., roads, buildings, and trees). Based on this, S20 performs foreground and background separation on the colored target point cloud set to determine the dynamic foreground region and the static background region therein, including S21, S22, and S23, as detailed below.

[0060] S21, using spatial density, depth gradient and normal vector as clustering reference factors (low-order geometric features), the target point cloud set after coloring is preliminarily clustered to obtain a preliminary clustering area.

[0061] Among them, the preliminary clustering area is the area where the spatial density is greater than the density threshold, the fluctuation value of the depth gradient (ie, the frequency of depth change) is greater than the gradient threshold, and the normal vector change value is greater than the vector threshold.

[0062] S22: When the proportion of dynamic points in the preliminary clustering area is greater than or equal to a preset ratio, determine it as a dynamic foreground area.

[0063] Among them, the coordinates of the midpoints in the preliminary clustering area after being projected into the image coordinate system of the vehicle are the projection coordinates. When the annotation of the pixel point corresponding to the projection coordinate in the environment image is a dynamic object annotation, the point corresponding to the projection coordinate is a dynamic point.

[0064] S23, when the proportion of dynamic points in the preliminary clustering area is less than a preset ratio, determine it as a static background area.

[0065] The dynamic and static object annotations in the environment image are used as weak label semantic cues to perform multi-scale screening and label refinement on the initial clustered areas, thereby accurately distinguishing dynamic foreground areas (such as vehicles, pedestrians, bicycles, etc.) from static background areas (such as roads, buildings, trees, etc.).

[0066] S30, combining the historical point cloud set within the time window to complete the incomplete area.

[0067] Among them, the time window is a window of preset length between target moments, and the incomplete area is a dynamic foreground area with occlusion and incompleteness in the target point cloud set.

[0068] Because a single-frame LiDAR point cloud collection may suffer from sparse sampling, severe occlusion, and texture loss, it is difficult to achieve continuous and complete geometric representation in image space. To this end, the present invention introduces a cross-frame point cloud aggregation mechanism to complete the target point cloud collection and uniformly construct a dense point cloud that integrates observation information from multiple moments.

[0069] Optionally, S30, combining the historical point cloud set within the time window to complete the incomplete area, includes: S31, S32 and S33, as follows.

[0070] S31, combining the vehicle motion trajectory, compares the foreground dynamic area and the background static area of ​​the historical point cloud frame within the time window with the foreground dynamic area and the background static area of ​​the target point cloud set to determine the incomplete area.

[0071] S32, combining the pose change reference information and the vehicle motion trajectory, obtaining supplementary content of the incomplete area from the historical point cloud frames within the time window.

[0072] S33, adding the supplementary content to the incomplete area in the target point cloud set to complete the incomplete area completion.

[0073] When completing incomplete areas, a time window control strategy is introduced to limit the number of point cloud aggregation frames to avoid geometric drift caused by long-term trajectory changes. For dynamic foreground areas, a dynamic binding framework is constructed based on the vehicle's motion trajectory. The foreground point clouds corresponding to the dynamic foreground areas are transformed and aggregated according to the vehicle's motion trajectory to ensure their geometric continuity and physical consistency.

[0074] Ultimately, this stage outputs a dense, structurally complete, and texture-enhanced world-class point cloud, providing global support for subsequent rendering, image generation, and 3D reconstruction. This point cloud boasts high spatial density, strong occlusion robustness, and multi-view consistency, demonstrating high adaptability and generalizability in dynamic urban scenes.

[0075] S40 , using a perspective projection mapping relationship, projecting the completed target point cloud set into the vehicle's image coordinate system to obtain a first projection image.

[0076] S50 , rendering the raster area in the first projection image to obtain a point rasterized conditional image.

[0077] Among them, the point rasterized conditional image is used as a training reference image for the vector map generation model to improve the training effect of the model and improve the quality of the vector map it generates.

[0078] To overcome the voids caused by the sparsity of LiDAR point clouds in image space, an embodiment of the present invention provides a conditional image generation method based on point rasterization. Point rasterization allows each 3D point to be projected onto the image plane as a kernel function with spatial coverage (such as a Gaussian distribution kernel), thereby diffusing its influence on neighboring pixels. This not only preserves the positional accuracy of the geometric points but also effectively fills the image structure discontinuities caused by projection sparsity.

[0079] Optionally, S50, rendering the raster area in the first projection image to obtain a point rasterized conditional image, includes: S51 and S52, which are specifically described as follows.

[0080] S51: Determine a grating area in a first projection image.

[0081] Optionally, S51, determining a grating area in the first projection image, includes: S511, S512, S513 and S514, as follows.

[0082] S511: Build a target visual cone model based on the corresponding position information of the laser radar at the target time.

[0083] According to the target view cone model, the target point cloud set is clipped to obtain the points that fall into the target view cone model.

[0084] S512 , projecting the grating points to the vehicle's image coordinate system to obtain grating pixel points, wherein the grating points are points in the target point cloud set that fall into the target viewing cone model.

[0085] S513: construct a two-dimensional Gaussian kernel at the position of the grating pixel point, and determine the scale range of the two-dimensional Gaussian kernel based on the depth of the grating point.

[0086] The scale of the two-dimensional Gaussian kernel is adaptively adjusted by the depth of the point on the image plane. The scale range corresponding to the near view area (small depth) is larger to enhance detail coverage, and the scale range corresponding to the distant view area (large depth) is reduced to retain boundary accuracy.

[0087] S514, taking the area within the scale range corresponding to the two-dimensional Gaussian kernel as the grating area.

[0088] S52, taking the RGB, depth, point density, and normal vector corresponding to the raster area as rendering targets, and performing weighted fusion in units of the raster area to complete the raster area rendering, thereby generating a conditional image that fuses geometry, texture, and shape information.

[0089] By rendering the raster area in the first projection image, the final output point rasterized conditional image is significantly superior to traditional point cloud visualization results in terms of resolution, spatial consistency and information density. It can be directly input into the vector map generation model as a high-quality prior condition, playing a key role in controlling perspective changes and lighting optimization.

[0090] In an optional implementation, after completing the incomplete area by combining the historical point cloud set within the time window, the conditional image generation method based on point rasterization further includes: S34, as follows.

[0091] S34 introduces a deep consistency filtering mechanism based on the joint constraints of Euclidean distance and angular similarity to suppress and delete redundant points in repeated areas of the completed target point cloud set, thereby retaining the information of structurally stable areas.

[0092] Optionally, in order to further improve the expressive ability of the generated image in the structural and semantic space, the conditional image generation method based on point rasterization provided by an embodiment of the present invention introduces a multi-scale processing mechanism in the conditional image generation. Specifically, when rendering the raster area in the first projection image to obtain a point rasterized conditional image, that is, the point rasterization output stage, multiple resolution levels of image versions (such as 1×, 1 / 2×, 1 / 4×) are output simultaneously, corresponding to detail enhancement, structure reconstruction and semantic alignment requirements respectively. The multi-scale images are fused as conditional pyramids in the subsequent video generation module to enhance the network's modeling capabilities for image content at different levels.

[0093] Optionally, to maintain temporal consistency and semantic continuity in target tracking, the point rasterization-based conditional image generation method provided in this embodiment of the present invention also incorporates a cross-frame semantic calibration mechanism. This mechanism, based on optical flow estimation and semantic propagation algorithms, ensures that the geometric structure and semantic distribution of the same target remain stable across different viewpoints, thereby avoiding jitter, structural drift, and semantic mismatch in the generated image caused by viewpoint offset or occlusion.

[0094] See also Figure 3 , Figure 3 An embodiment of the present invention provides a conditional image generation device based on point rasterization. Optionally, the conditional image generation device based on point rasterization is applied to the electronic device described above.

[0095] The conditional image generating apparatus based on point rasterization includes: a first processing unit 601 and a second processing unit 602 .

[0096] The first processing unit 601 is configured to perform coloring processing on the target point cloud set according to the environment image corresponding to the target moment;

[0097] The target point cloud set is the point cloud data set collected by the laser radar deployed on the vehicle at the target time. The coloring process refers to adding the corresponding color value in the environment image to each point in the target point cloud set.

[0098] The first processing unit 601 is further configured to perform foreground-background separation processing on the target point cloud set after the coloring processing to determine a dynamic foreground area and a static background area therein;

[0099] The first processing unit 601 is further configured to complete the incomplete region by combining the historical point cloud set within the time window, wherein the incomplete region is a dynamic foreground region in the target point cloud set that is occluded and incomplete;

[0100] The first processing unit 601 is further configured to project the completed target point cloud set into the vehicle's image coordinate system using a perspective projection mapping relationship to obtain a first projection image, wherein the perspective projection mapping relationship is a mapping relationship between the lidar coordinate system and the vehicle's image coordinate system;

[0101] The second processing unit 602 is configured to render the raster area in the first projection image to obtain a point rasterized conditional image.

[0102] It should be noted that the conditional image generation device based on point rasterization provided in this embodiment can implement the method flow shown in the above method flow embodiment to achieve the corresponding technical effects. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiment.

[0103] An embodiment of the present invention further provides a storage medium storing computer instructions or programs that, when read and executed, execute the method for conditional image generation based on point rasterization described in the above embodiment. The storage medium may include memory, flash memory, registers, or a combination thereof.

[0104] The following provides an electronic device, which can be a vehicle computer device or a server device. Figure 1 As shown, the above-mentioned conditional image generation method based on point rasterization can be implemented. Specifically, the electronic device includes: a processor 10, a memory 11, and a bus 12. The processor 10 may be a CPU. The memory 11 is used to store one or more programs. When the one or more programs are executed by the processor 10, the conditional image generation method based on point rasterization of the above-mentioned embodiment is executed.

[0105] In summary, an embodiment of the present invention provides a conditional image generation method and related device based on point rasterization, which performs coloring processing on a target point cloud set according to an environmental image corresponding to the target moment; wherein the target point cloud set is a point cloud data set collected by a laser radar deployed on a vehicle at the target moment, and the coloring processing refers to adding a color value corresponding to each point in the target point cloud set in the environmental image; performing foreground and background separation processing on the target point cloud set after coloring to determine the dynamic foreground area and the static background area therein; combining the historical point cloud set within the time window, the incomplete area is completed, wherein the incomplete area is a dynamic foreground area in the target point cloud set that is occluded and incomplete; using the perspective projection mapping relationship, the completed target point cloud set is projected to the image coordinate system of the vehicle to obtain a first projection image, wherein the perspective projection mapping relationship is the mapping relationship between the laser radar coordinate system and the image coordinate system of the vehicle; rendering the raster area in the first projection image to obtain a point rasterized conditional image. By performing frame-by-frame parsing and image plane mapping on the original lidar point cloud data, and combining it with point rasterization processing, a geometric conditional image with pixel-level accuracy is constructed, providing structural prior support for subsequent new view synthesis and controllable video generation.

[0106] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

[0107] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A conditional image generation method based on point rasterization, characterized in that: The method comprises: Coloring the target point cloud set according to the environment image corresponding to the target moment; The target point cloud set is a point cloud data set collected by a laser radar deployed on the vehicle at the target time, and the coloring process refers to adding a color value corresponding to each point in the target point cloud set in the environment image; Perform foreground and background separation on the target point cloud set after coloring to determine the dynamic foreground area and static background area; Combine the historical point cloud set within the time window to complete the incomplete area, where the incomplete area is the dynamic foreground area in the target point cloud set that is occluded and incomplete; Projecting the completed target point cloud set into the vehicle's image coordinate system using a perspective projection mapping relationship to obtain a first projected image, wherein the perspective projection mapping relationship is a mapping relationship between the lidar coordinate system and the vehicle's image coordinate system; The raster area in the first projection image is rendered to obtain a point rasterized conditional image.

2. The conditional image generation method based on point rasterization according to claim 1, characterized in that: The coloring process of the target point cloud set according to the environment image corresponding to the target moment includes: Projecting the target point cloud set onto the vehicle's image coordinate system using a perspective projection mapping relationship to obtain a second projected image; Using the color value of the i-th pixel in the environment image as the color value of the i-th pixel in the second projection image; Determine, according to the perspective projection mapping relationship, a corresponding point of the i-th pixel point in the second projection image in the target point cloud set; The color value of the i-th pixel in the second projection image is added to its corresponding point in the target point cloud set.

3. The conditional image generation method based on point rasterization according to claim 1, characterized in that: The environment image is an image that has completed dynamic object annotation and static object annotation. The foreground-background separation process is performed on the target point cloud set after the coloring process to determine the dynamic foreground area and the static background area therein, including: Using spatial density, depth gradient, and normal vector as clustering reference factors, the target point cloud set after coloring is preliminarily clustered to obtain a preliminary clustering region; wherein the preliminary clustering region is a region where the spatial density is greater than the density threshold, the fluctuation value of the depth gradient is greater than the gradient threshold, and the change value of the normal vector is greater than the vector threshold; When the proportion of dynamic points in the preliminary clustering area is greater than or equal to a preset ratio, determining it as a dynamic foreground area; The coordinates of the points in the preliminary clustering area after being projected onto the vehicle's image coordinate system are projected coordinates. When the annotation of the pixel point corresponding to the projected coordinate in the environment image is a dynamic object annotation, the point corresponding to the projected coordinate is a dynamic point. When the proportion of dynamic points in the preliminary clustering area is less than a preset ratio, it is determined to be a static background area.

4. The conditional image generation method based on point rasterization according to claim 1, wherein: The method of combining the historical point cloud set within the time window to complete the incomplete area includes: In combination with the vehicle motion trajectory, the foreground dynamic area and the background static area of ​​the historical point cloud frame within the time window are compared with the foreground dynamic area and the background static area of ​​the target point cloud set to determine the incomplete area; Acquire supplementary content of the incomplete area from historical point cloud frames within the time window in combination with the pose change reference information and the vehicle motion trajectory; The supplementary content is added to the incomplete area in the target point cloud set to complete the incomplete area completion.

5. The conditional image generation method based on point rasterization according to claim 4, characterized in that: After completing the incomplete area by combining the historical point cloud set within the time window, the method further includes: A deep consistency filtering mechanism is introduced to suppress and delete redundant points in repeated areas of the completed target point cloud set based on the joint constraints of Euclidean distance and angular similarity.

6. The method for generating conditional images based on point rasterization according to claim 1, wherein: Rendering the raster area in the first projection image to obtain a point rasterized conditional image includes: determining a grating area in the first projection image; The RGB, depth, point density, and normal vector corresponding to the grating area are used as rendering targets, and weighted fusion is performed on the grating area as a unit to complete the grating area rendering.

7. The conditional image generation method based on point rasterization according to claim 6, characterized in that: The determining of the grating area in the first projection image includes: Combined with the laser radar's corresponding pose information at the target time, a target cone model is constructed; Projecting the raster points to the vehicle's image coordinate system to obtain raster pixel points, wherein the raster points are points in the target point cloud set that fall into the target viewing cone model; Constructing a two-dimensional Gaussian kernel at the position of the grating pixel point, and determining the scale range of the two-dimensional Gaussian kernel based on the depth of the grating point; The area within the scale range corresponding to the two-dimensional Gaussian kernel is used as the grating area.

8. A conditional image generation device based on point rasterization, characterized in that: The device comprises: A first processing unit is used to perform coloring processing on the target point cloud set according to the environment image corresponding to the target moment; The target point cloud set is a point cloud data set collected by a laser radar deployed on the vehicle at the target time, and the coloring process refers to adding a color value corresponding to each point in the target point cloud set in the environment image; The first processing unit is further configured to perform foreground-background separation processing on the target point cloud set after the coloring processing to determine a dynamic foreground area and a static background area therein; The first processing unit is further configured to complete the incomplete area by combining the historical point cloud set within the time window, wherein the incomplete area is a dynamic foreground area in the target point cloud set that is occluded and incomplete; The first processing unit is further configured to project the completed target point cloud set into the vehicle's image coordinate system using a perspective projection mapping relationship to obtain a first projection image, wherein the perspective projection mapping relationship is a mapping relationship between the lidar coordinate system and the vehicle's image coordinate system; The second processing unit is configured to render the raster area in the first projection image to obtain a point rasterized conditional image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: a processor and a memory, the memory being configured to store one or more programs; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images