Depth occlusion constraint-based unstructured road driving video generation method

Abstract

The invention relates to an unstructured road driving video generation method based on depth occlusion constraint, and belongs to the technical field of automatic driving. Semantic structure migration is carried out on driving videos of structured road scenes such as cities or expressways, and semantic layout representation adaptive to mining areas is constructed; on this basis, layered layout representation of depth and occlusion sorting is introduced, and in combination with layout taking a Diffusion Transform as a backbone network and a depth condition diffusion video generation architecture, driving video data conforming to mining area terrain and occlusion geometric constraints are automatically synthesized on the premise of not depending on real mining area point cloud and a BEV world model. The method is used for providing controllable and extensible training and testing samples for the mining area automatic driving perception model.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and specifically to a method for generating unstructured road driving videos based on depth occlusion constraints. Background Technology

[0002] With the widespread application of deep learning in autonomous driving perception and decision-making, models are increasingly reliant on large-scale, diverse, and highly labeled video data. In real-world road environments, urban roads and highways possess structured elements such as lane lines, curbs, and traffic signs, resulting in a large accumulation of public and private real-world driving datasets. However, unstructured road scenarios such as open-pit mines and construction sites present unique challenges: temporary road morphology, significant terrain undulations, and massive operational equipment, leading to high safety risks. Collecting video data from actual vehicles is costly and time-consuming, and long-tail hazardous conditions are even more difficult to cover systematically through live filming. To alleviate the data scarcity problem, existing technologies have proposed various video generation methods to generate synthetic videos from text, images, or multimodal inputs, expanding the distribution of training samples.

[0003] Patent CN120835216A proposes a video generation method and system based on cross-modal collaborative learning. It receives multimodal inputs such as text and images, extracts multimodal features, and designs a cross-modal collaborative module within a diffusion model to weightedly fuse latent video features, thereby improving the consistency between the generated video and the text semantics and the overall image quality. This type of method primarily focuses on multimodal semantic alignment and visual effects, and is applicable to general video generation scenarios. However, it lacks specific modeling and control over 3D geometric relationships, occlusion relationships, and driving safety constraints within the scene, making it difficult to guarantee the strict rationality of spatial structure and motion logic when using training data for autonomous driving.

[0004] Patent CN120897105A proposes a video generation method based on temporal similarity. This method selects key video frames from textual information, uses a diffusion or generative network to generate initial intermediate frames between these key frames, and then optimizes the intermediate frames based on the cosine similarity of visual features between adjacent frames to improve the inter-frame coherence and temporal consistency of the generated video. While this approach emphasizes smoothness and similarity constraints in the temporal dimension, which can alleviate problems such as flickering and frame skipping, it still primarily measures similarity in a two-dimensional pixel space. It does not explicitly characterize the depth distribution and occlusion relationships of the scene, and lacks the ability to express the three-dimensional geometric constraints between the vehicle and the environment. Therefore, it is not suitable for direct application in autonomous driving training scenarios that require precise spatial relationships.

[0005] In the field of autonomous driving, some studies attempt to generate driving videos through simulation platforms or world models, utilizing virtual environments to construct various weather, traffic density, and accident scenarios, reducing the burden of real-world data collection. Some of these methods use bird's-eye view representations or BEV features to unify terrain and traffic elements, combining them with diffusion generative networks or Transformer structures to generate multi-view videos. These solutions often rely on existing high-precision 3D maps and finely modeled simulation scenarios, or depend on large-scale sensor data from the mining area itself to construct a semantic terrain world model. Given the abundance of urban road and highway driving data in reality, but the lack of open mining area scene data, existing methods have not yet provided an effective path for automatically migrating from "structured urban scenes" to "unstructured mining area scenes," and lack specific solutions to combine the migrated mining area layout with deep occlusion relationships to drive video generation.

[0006] On the other hand, while general video generation technologies improve visual quality, they mostly rely on implicit learning of the entire geometric relationship—scene layout, depth distribution, and occlusion order—through networks. While some geometric errors are tolerable in general entertainment content, in autonomous driving training in mining areas, the minimum distance between the vehicle and slopes, potholes, and material piles, as well as the location of occlusion boundaries, directly affect the performance of tasks such as object detection and distance estimation. The lack of explicit depth and occlusion modeling easily leads to problems such as vehicles being "occluded" by distant backgrounds, objects appearing suspended in mid-air, and inconsistent terrain undulations, reducing the training value of synthetic samples for perception models.

[0007] In summary, existing technologies still have shortcomings in the following aspects: (1) Multimodal or temporal similarity-driven video generation methods focus on semantic consistency and temporal continuity, but lack three-dimensional geometric and occlusion constraint design for autonomous driving. (2) There is no mature solution to automatically construct unstructured scene layouts with semantic elements and spatial structure of mining areas using existing urban scenes and large-scale structured road data for synthesizing mining area video data; (3) Existing diffusion video generation architectures mostly work under two-dimensional pixels or simple conditions, lacking a special design that encodes semantic layout, depth map and occlusion level as conditions to constrain the relationship between vehicles and terrain over long time series.

[0008] Therefore, this invention proposes a technical solution: based on urban scene videos, construct a semantic layout adapted to mining areas, introduce explicit depth and occlusion ordering representations and combine them with a DiT-based conditional diffusion video generation network to generate synthetic driving videos that conform to the unstructured semantic features of mining areas and satisfy depth occlusion geometric constraints, thereby improving the training and evaluation effect of autonomous driving systems in mining areas under complex scenarios. Summary of the Invention

[0009] In view of the above problems, this invention proposes an unstructured road driving video generation method based on depth occlusion constraints. By performing semantic structure transfer on driving videos of structured road scenes such as cities or highways, a semantic layout representation adapted to mining areas is constructed. On this basis, a hierarchical layout representation based on depth and occlusion order is introduced. Combined with a layout and depth conditional diffusion video generation architecture with DiffusionTransformer (DiT) as the backbone network, driving video data that conforms to the terrain and occlusion geometric constraints of mining areas can be automatically synthesized without relying on real mining area point clouds and BEV world models. This provides controllable and scalable training and testing samples for autonomous driving perception models in mining areas.

[0010] This invention provides a method for generating unstructured road driving videos based on depth occlusion constraints, including: S1: Acquire driving images in a structured scene; After time synchronization and calibration alignment of driving images, a semantic segmentation network trained for structured road scenes is introduced to perform semantic annotation and instance segmentation to obtain the semantic category and instance contour of the driving images; the semantic category and instance contour of the driving images are discretized into a regular grid to construct a semantic layout map of the urban scene. S2. Based on the preset semantic mapping rules, the structured road scene in the urban scene semantic layout map is transferred to the mining area scene to generate a mining area semantic layout map. S3: Semantic layout map of the mining area Input the layout into the depth prediction network, and output the predicted depth map of the mining area; Multiple semantic instances were located in the semantic layout map of the mining area using instance segmentation technology; For each semantic instance, depth statistics are performed to obtain an average depth, and then a global sort is performed to obtain the instance depth sorting result; Based on the sorting results, clarify the occlusion logic between instances and mark the occlusion relationships: Based on the instance depth sorting results, divide the data into depth layers and generate the corresponding structured data; Based on structured data from multiple depth layers, a layered semantic layout representation of the mining area is formed; S4: Under the condition of layered semantic layout representation of the mining area, a DiT denoising network is used to generate driving videos that conform to the geometric and occlusion constraints of the mining area; the network architecture of the denoising network adopts Diffusion Transformer.

[0011] Optionally, step S1 specifically includes the following steps: S1.1 Select driving images containing multi-view camera and vehicle pose information, and read the intrinsic and extrinsic parameters, timestamps and vehicle trajectory data of each view camera. Distortion correction, temporal alignment and resolution unification are performed on the driving images. Samples with severe occlusion, blurred images or missing annotations are removed to obtain the filtered images. The images of consecutive frames are combined into several urban driving scenes according to the scene division; S1.2 Utilize a semantic segmentation network to perform pixel-level semantic classification on the filtered images and label the category of each pixel in the driving image; An instance segmentation network is used to extract instance-level contours and locations from the filtered images, distinguishing different individuals under the same semantic category and obtaining the contour boundaries and spatial locations of each instance in the filtered images. The double segmentation result of the filtered image is based on the category label of each pixel and the contour boundary and spatial location of each instance. The double segmentation results of the filtered images are projected onto a uniform fixed-resolution regular raster to form a two-dimensional grid map, which serves as a semantic layout map of the urban scene.

[0012] Optionally, step S2 specifically includes the following steps: S2.1: Based on the characteristics of open-pit mining operations, establish a mapping table from the structured road scene semantic categories to the mining area semantic categories in the urban scene semantic layout map, and obtain the semantic mapping results; An end-to-end mapping network is constructed. A batch of urban scene semantic layout maps are taken as input. The preliminary semantic layout of the mining area is output according to the above semantic mapping results. At the same time, the mapping residual is calculated to iteratively update the mapping network and obtain the first updated semantic layout of the mining area. S2.2: Based on the semantic mapping results, the semantic layout of the mining area updated for the first time is structurally adjusted and geometrically reconstructed to generate a reconstructed mining area layout; the reconstructed mining area layout is a semantic layout of the mining area with geometric features and connectivity. S2.3: Based on the operating mode and safety distance requirements of vehicles in the mining area, the main operating entities are automatically arranged on the reconstructed mining area layout to obtain a semantic layout map of the mining area.

[0013] Optionally, the specific steps of S2.1 include: S2.11 Let k=1, when k=1, it represents the first iteration; S2.12 Input the semantic layout map of the city scene in the kth round into the mapping network in the kth round, complete the semantic category conversion, and output the preliminary semantic layout of the mining area in the kth round; S2.13 Obtain the k-th round mapping residual between the initial semantic layout of the mining area and the ideal semantic layout of the mining area; S2.14 Input the k-th round mapping residual and the k-th round urban scene semantic layout map into the residual learning network, train the residual learning network based on the loss function, and obtain the k-th round updated residual learning network as the k+1 round residual learning network; S2.15 Update the k-th round mapping network based on the k-th round updated residual learning network to obtain the k-th round updated mapping network, which serves as the k+1-th round mapping network; S2.16 Input the k-th round city scene semantic layout graph into the k-th round update mapping network to obtain the k-th round optimized mining area semantic layout, which is used as the k+1 round city scene semantic layout graph; S2.17 Determine whether k is greater than or equal to K, where K represents the total number of iterations. If yes, obtain the semantic layout of the mining area updated for the first time. If no, let k = k + 1 and return to step S2.12.

[0014] Optionally, the specific steps of step S2.2 include: The width of urban lanes is scaled up to ensure that the passable area is wide enough to meet the needs of mining vehicles for passing and turning. According to the preset elevation difference and slope parameters, the positions of platforms, slopes between platforms and side slopes are arranged to form a mining area geometric structure with stepped or multi-platform characteristics. Generate the outline of the stockpile block in the specified area, and mark the toe of the stockpile slope and the top area of ​​the stockpile; While ensuring accessibility and safe distances, connectivity checks are conducted on accessible areas to ensure that vehicles can form closed or semi-closed traffic flows between loading and unloading areas.

[0015] Optionally, step S3 specifically includes the following steps: S3.1: Construct the original layout to the deep prediction network with the encoder-decoder structure as the backbone; The original layout-to-depth prediction network is trained to obtain the layout-to-depth prediction network. semantic layout map of the mining area Input the layout into the depth prediction network, output the predicted depth map of the mining area. ; S3.2: Predicted depth map of the mining area Semantic layout map of mining area L Multiple semantic instances were located using instance segmentation technology; For each semantic instance, depth statistics are performed within its corresponding region to obtain several core metrics, including: average depth, depth range, and depth distribution shape. Based on the average depth, a global sort of all semantic instances is performed to obtain the instance depth sorting result; Based on the sorting results, clarify the occlusion logic between instances and mark the occlusion relationships: S3.3: Divide the depth layers according to the instance depth sorting results; For each depth layer, generate corresponding structured data; Based on structured data from multiple depth layers, a layered semantic layout representation of the mining area is formed, which includes semantic categories, relative depths, and hierarchical labels. This representation not only preserves the original semantic structure of the mining area's terrain and equipment, but also explicitly gives the occlusion order relationship between different layers and different instances.

[0016] Optionally, step S4 specifically includes the following steps: S4.1: Select the driving video of the mining area from the target perspective, input each frame of the image into the encoder, and generate a spatiotemporal token sequence; The semantic layout map, depth map, and layered occlusion labels of the mining area are fused together and then input into the encoder to generate a conditional token sequence. S4.2: Add Gaussian noise to the spatiotemporal token sequence and the conditional token sequence and then fuse them to obtain a noisy video token sequence; S4.4: Order t =1, when t =1 indicates the initial time step; S4.5: Input the concatenated token sequence into the time step. t DiT denoising network A t ; The conditional token sequence is injected into each Transformer layer as a time step using cross-attention. t Global constraints; Based on time step t Global constraints, to obtain time steps t Predictive noise; Calculate time steps t The loss values ​​of predicted noise and actual noise are compared based on the noise loss function and the loss value update time step. t Denoising network parameters to obtain time steps t The updated noise reduction network serves as a time step. t +1 noise reduction network; S4.6: Judgment t Is it greater than or equal to? T , T Let represent the total time steps. If yes, the final denoising network is obtained; otherwise, let . t = t +1, return to step S4.5; S4.7: Obtain a clean sequence of potential video tokens based on the final denoising network; The clean latent video token sequence is input into the convolutional decoder. Through multiple deconvolution and upsampling operations, the low-dimensional latent features are restored into high-resolution pixel images. All frames are concatenated to obtain a driving video that conforms to the geometry and occlusion constraints of the mining area.

[0017] Optionally, the noisy video token sequence expression is:

[0018] in, These are the fidelity coefficients obtained by multiplying the noisy scheduling sequence. Indicates time step t Noisy video potential token sequence, Represents a clean sequence of potential video tokens. Standard Gaussian noise added during the forward diffusion process. Indicates a normal distribution. This represents the unit covariance matrix.

[0019] Optionally, the occlusion logic is that a foreground instance occludes a background instance, and the foreground subject, the occluded subject, and the completely occluded instance are marked one by one; The foreground subject is closest to the camera and is either unobstructed or only obstructed by a closer instance, such as the mining truck in front. The obscured entity is partially obscured by a close-up instance, such as a section of a slope obscured by a mining card; the completely obscured instance is completely covered by a close-up instance with no visible area, such as equipment behind a tall stockpile of materials.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects: (1) This invention utilizes the semantic structure transfer mechanism of “urban scene / highway scene - mining area scene” to make full use of existing public road datasets. Even in the absence of large-scale real mining area data, it can still construct a rich variety of mining area driving scenarios, significantly reducing the cost of mining area data collection and annotation. (2) Based on the semantic layout of the mining area, this invention introduces a layered layout representation of depth and occlusion order, explicitly models the distance relationship and occlusion order between objects such as mining cards, excavators, stockpiles, and slopes, and combines the inter-layer synthesis strategy in the generation stage to effectively improve the geometric rationality of the generated video in terms of vehicle-terrain occlusion relationship, which is convenient for training a perception model that is sensitive to occlusion and distance. (3) The present invention adopts a layout and depth conditional video diffusion generation architecture with Diffusion Transformer as the backbone, and uniformly models the semantic layout, depth occlusion relationship and video content of the mining area at the spatiotemporal token level. Compared with the traditional convolutional UNet type diffusion network, it has stronger long-term sequence modeling ability and multi-condition fusion ability, and can generate long-term continuous mining area driving videos with consistent structure and occlusion relationship, providing high-quality synthetic data support for autonomous driving perception and simulation in mining areas. Attached Figure Description The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0021] Figure 1 This is a flowchart illustrating a method for generating driving videos in mining areas based on depth occlusion constraints. Detailed Implementation To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0022] A specific embodiment of the present invention, such as Figure 1 A method for generating depth occlusion-aware video data for unstructured mining scenes is disclosed, including the following steps: S1: Acquire driving images in a structured scene; After time synchronization and calibration alignment of the driving images, a semantic segmentation network trained for structured road scenes is introduced to perform semantic annotation and instance segmentation to obtain the semantic category and instance contour of the driving images. Discretize the semantic categories and instance contours of driving images into regular grids or pixel-level rasters to construct urban scene semantic layout maps and structured scene semantic layout maps; This invention forms a complete scene structured modeling process by constructing multi-source data, parsing semantic layout, and extracting topological relationships, providing high-precision and interpretable semantic prior knowledge for subsequent model training and autonomous driving decision-making.

[0023] Optionally, step S1 specifically includes the following steps: S1.1: Construct a driving image dataset covering diverse scenarios and multimodal features; The scenarios include urban roads, highways, and park roads; The multimodal data includes various traffic flows, weather conditions, and time periods. The driving images include a variety of lane patterns, roadside environment, and combinations of traffic participants; S1.2: A semantic segmentation network is used to perform pixel-level semantic classification on driving images, and category labels are annotated to obtain the semantic segmentation results of driving images; An instance segmentation network is used to perform instance-level segmentation on driving images to obtain the instance segmentation results of driving images, including the contour and position of each instance in the driving image; The semantic segmentation results and instance segmentation results of the driving images are projected onto a fixed-resolution rule grid to obtain a two-dimensional grid map, which serves as the city semantic layout map, including category labels, instance outlines, and topological relationships. S1.3: Perform connectivity and topology analysis on the semantic layout graph to obtain lane topology and traffic rule priors; Based on the lane topology, the main lanes, auxiliary roads, pedestrian crossings, and intersection areas are extracted, and the traffic flow direction and priority relationship are recorded to provide structural priors for subsequent semantic structure transfer. Optionally, connectivity analysis includes the following steps: The graph search algorithm is used to identify the connection relationships between main lanes, auxiliary roads, and ramps; and the intersection area and its associated lanes are divided based on the location of pedestrian crossings, stop lines, and traffic lights.

[0024] Optionally, the specific steps for topology modeling include: Lane lines, intersections, and traffic participants are modeled as graph nodes, and connection relationships are modeled as graph edges; attributes are labeled for graph nodes / edges to form a structured topology graph.

[0025] Optionally, the specific steps for extracting traffic rules include: By analyzing the direction of lane lines and vehicle trajectories, the direction of travel in a lane can be inferred.

[0026] Priority traffic rules: Based on the stop line position, traffic light signals, and yield signs, the priority traffic rules at intersections are marked.

[0027] Optionally, the specific steps for generating the structural prior include: Template library construction: Clustering the topological structures of common scenarios into templates for subsequent semantic transfer; Rule coding: Traffic rules are encoded as logical constraints and embedded into the planning / control module.

[0028] Optionally, step S1 specifically includes the following steps: S1.1 Select driving images from the nuScenes dataset that contain multi-view camera and vehicle pose information, and read the intrinsic and extrinsic parameters, timestamps and vehicle trajectory data of each view camera. Distortion correction, temporal alignment and resolution unification are performed on the driving images. Samples with severe occlusion, blurred images or missing annotations are removed to obtain the filtered images. The images of consecutive frames are combined into several urban driving scenes according to the scene division; S1.2 Utilize a semantic segmentation network to perform pixel-level semantic classification on the filtered images and label the category of each pixel in the driving image; An instance segmentation network is used to extract instance-level contours and locations from the filtered images, distinguishing different individuals under the same semantic category and obtaining the contour boundaries and spatial locations of each instance in the filtered images. The double segmentation result of the filtered image is based on the category label of each pixel and the contour boundary and spatial location of each instance. The double segmentation results of the filtered images are projected onto a uniform fixed-resolution regular raster to form a two-dimensional grid map, which serves as a semantic layout map of the urban scene. Optionally, it also includes projection rules: based on the vehicle pose or camera calibration parameters, the image pixel coordinates are mapped to the BEV grid coordinates of the top view to ensure that the spatial scale is consistent after different images of the same scene are projected. Raster information integration: Each raster unit synchronously retains semantic category labels and instance outline identifiers, and implicitly implies topological relationships through raster connectivity.

[0029] Semantic categories include, but are not limited to: lane lines, motor vehicle lanes, non-motor vehicle lanes, sidewalks, curbs, buildings, green belts, vehicles, pedestrians, and bicycles; Optionally, the expression for the mapping is:

[0030] in, For the camera intrinsic parameter matrix, This is the extrinsic transformation matrix from camera to vehicle. This is the extrinsic transformation matrix from the vehicle to the world coordinate system. u Represents the x-coordinate of a pixel. v The vertical coordinate of a pixel is represented by X, which is the x-axis coordinate of the world coordinate system, Y is the y-axis coordinate of the world coordinate system, and Z is the z-axis coordinate of the world coordinate system.

[0031] Optionally, the expression for the semantic category is:

[0032] in, For semantic categories, To segment the network for semantic categories at this grid The predicted probability, Represents raster coordinates ( x , y The final semantic category label at ) This indicates that the parameter corresponding to the maximum value of all semantic categories c is taken.

[0033] Through the above projection and discretization, the semantic information of lanes, curbs, roadside environment and dynamic traffic participants is projected onto a two-dimensional grid with a fixed resolution to obtain a semantic layout map of the urban scene. Connectivity and topology analysis are performed on the semantic layout map to identify structures such as main lanes, auxiliary roads, pedestrian crossings, and intersection areas, and to record information such as traffic flow direction, lane connection relationship and candidate parking areas.

[0034] This invention obtains a semantic layout representation of a city scene in a two-dimensional grid space that simultaneously includes semantic categories, instance outlines, and road topological relationships, providing input for subsequent semantic structure migration from cities to mining areas.

[0035] S2. Based on the preset semantic mapping rules, the structured road scene in the urban scene semantic layout map is transferred to the mining area scene to generate a mining area semantic layout map. For example, structured environmental elements such as "driveway - curb - building - green belt" can be mapped to core mining area elements such as "accessible area - platform edge - slope - stockpile - work area". Passenger cars, trucks, and other vehicle categories are mapped to mining dump trucks, mining trucks, and other operational vehicles. Map some buildings or roadside facilities to areas such as excavators, loaders, and fixed equipment.

[0036] By rewriting topological relationships and scaling geometric parameters, the original lane width and intersection shape are adjusted to make the passable area more in line with the road width and platform shape of the mining area. By arranging objects such as mining trucks, excavators, and stockpiles within a specific area, a semantic layout map of the mining area that satisfies connectivity and safety constraints is constructed, thereby completing the semantic structure migration from a structured road scene to an unstructured mining area scene.

[0037] Optionally, step S2 specifically includes the following steps: S2.1: Based on the characteristics of open-pit mining operations, establish a mapping table from the structured road scene semantic categories to the mining area semantic categories in the urban scene semantic layout map, and obtain the semantic mapping results; An end-to-end mapping network is constructed. A batch of urban scene semantic layout maps are taken as input. The preliminary semantic layout of the mining area is output according to the above semantic mapping results. At the same time, the mapping residual is calculated to iteratively update the mapping network and obtain the first updated semantic layout of the mining area. Optionally, the specific steps of S2.1 include: S2.11 Let k=1, when k=1, it represents the first iteration; S2.12 Input the semantic layout map of the city scene in the kth round into the mapping network in the kth round, complete the semantic category conversion, and output the preliminary semantic layout of the mining area in the kth round; S2.13 Obtain the k-th round mapping residual between the initial semantic layout of the mining area and the ideal semantic layout of the mining area; S2.14 Input the k-th round mapping residual and the k-th round urban scene semantic layout map into the residual learning network, train the residual learning network based on the loss function, and obtain the k-th round updated residual learning network as the k+1 round residual learning network; S2.15 Update the k-th round mapping network based on the k-th round updated residual learning network to obtain the k-th round updated mapping network, which serves as the k+1-th round mapping network; S2.16 Input the k-th round city scene semantic layout graph into the k-th round update mapping network to obtain the k-th round optimized mining area semantic layout, which is used as the k+1 round city scene semantic layout graph; S2.17 Determine whether k is greater than or equal to K, where K represents the total number of iterations. If yes, obtain the semantic layout of the mining area updated for the first time. If no, let k = k + 1 and return to step S2.12.

[0038] The residual learning network can be a convolutional network (CNN) or a graph neural network (GNN). By learning the mapping residuals, it corrects the morphological inconsistencies caused by relying solely on rule mapping, so that the final generated mining area layout strictly follows the semantic mapping rules and has a spatial form that conforms to reality.

[0039] It is understandable that the morphological incongruity may be the difference between the linear form of urban roads and the blocky form of mining platforms; the actual spatial form may be that the platform size is appropriate, the passageway is reasonable, and the equipment distribution is coordinated.

[0040] Optionally, the mapping table includes: The driveway, curb, roadside green belt, and roadside open space are respectively mapped to the core elements of the mining area, such as the platform's passable area, platform edge, slope area, stockpile area, and work area; The vehicle categories, such as passenger cars, trucks, and buses, are mapped to mining operation vehicles such as mining dump trucks, mining trucks, and service vehicles. Map some roadside buildings or facilities as areas for equipment such as excavators, loaders, and crushing plants; Optionally, convolutional networks or graph neural networks can be introduced to perform residual learning on the structural differences between the urban layout and the target mining area layout, so that the relocated mining area layout is more in line with the actual mining area road and platform structure in terms of overall shape.

[0041] S2.2: Based on the semantic mapping results, the semantic layout of the mining area updated for the first time is structurally adjusted and geometrically reconstructed to generate a reconstructed mining area layout; the reconstructed mining area layout is a semantic layout of the mining area with geometric features and connectivity. Optionally, the specific steps of step S2.2 include: The width of urban lanes is scaled up to ensure that the passable area is wide enough to meet the needs of mining vehicles for passing and turning. According to the preset elevation difference and slope parameters, the positions of platforms, slopes between platforms and side slopes are arranged to form a mining area geometric structure with stepped or multi-platform characteristics. Generate the outline of the stockpile block in the specified area, and mark the toe of the stockpile slope and the top area of ​​the stockpile; While ensuring accessibility and safe distances, connectivity checks are conducted on accessible areas to ensure that vehicles can form closed or semi-closed traffic flows between loading and unloading areas.

[0042] S2.3: Based on the operating mode and safety distance requirements of vehicles in the mining area, the main operating entities are automatically arranged on the reconstructed mining area layout to obtain a semantic layout map of the mining area. Optionally, the specific steps of step S2.3 include: Excavators and loaders are deployed near the loading area, and a queue of mining trucks waiting to be loaded is set up in front of or to the side of them; Arrange unloading vehicles and discharge equipment near the unloading area or crushing station; The initial positions and driving directions of vehicles entering, exiting, and meeting on the platform channel are generated; through a rule system or statistical model, the minimum distance between vehicles, the safe distance from the platform edge, and the safe buffer zone from the stockpile are constrained to obtain a semantic layout map of the mining area that satisfies connectivity, safety, and operational rationality.

[0043] This invention successfully migrates urban structured road scenes into two-dimensional semantic layouts with mining operation semantics and road morphology features, laying the foundation for mining area depth and occlusion modeling.

[0044] S3: Input the semantic layout map of the mining area into the depth prediction network to predict the relative depth value of each position in the layout, and obtain a mining area depth map from an approximate camera view. The average depth and depth distribution are calculated based on the depth map of the mining area. All instances are sorted and divided into several layers according to a preset depth threshold or clustering strategy. A layered semantic layout representation of the mining area containing semantic categories, relative depths and layered labels is constructed. The layered semantic layout of the mining area not only preserves the semantic structure of the mining area's terrain and equipment, but also explicitly gives the occlusion order relationship between different objects.

[0045] Optionally, step S3 specifically includes the following steps: S3.1: Construct the original layout to the deep prediction network with the encoder-decoder structure as the backbone; The original layout-to-depth prediction network is trained to obtain the layout-to-depth prediction network. semantic layout map of the mining area Input the layout into the depth prediction network, output the predicted depth map of the mining area. The expression is:

[0046] in, For having trainable parameters From layout to deep prediction networks, This represents a depth map of the mining area. This represents a semantic layout map of the mining area.

[0047] Optionally, a joint loss function is used to train the original layout-to-depth prediction network. The expression for the joint loss function is as follows:

[0048] in, This indicates that gradient operations are used to constrain deep edge structures. For deep loss, The weighting coefficient is one. The weighting coefficient is two. This is a map showing the actual depth of the mining area.

[0049] Optionally, the input to the layout-to-depth prediction network is a semantic layout map of the mining area. , The depth map of the mining area reflects the relative distances of the platform, slope, stockpile, and vehicles from the camera's perspective.

[0050] S3.2: Predicted depth map of the mining area Semantic layout map of mining area L Multiple semantic instances were located using instance segmentation technology; For each semantic instance, depth statistics are performed within its corresponding region to obtain several core metrics, including: average depth, depth range, and depth distribution shape. Based on the average depth, a global sort of all semantic instances is performed to obtain the instance depth sorting result; Based on the sorting results, clarify the occlusion logic between instances and mark the occlusion relationships: Optionally, the average depth is the arithmetic mean of the depth values ​​of all pixels within the instance region, reflecting the overall relative distance of the instance; The depth range is the difference between the maximum and minimum depth values ​​within the instance area, reflecting the geometric shape of the instance itself. For example, the depth range of a tall slope is larger, while the depth range of a small mining car is smaller. The depth distribution shape is analyzed by histograms, Gaussian fitting, and other methods to determine the distribution pattern of depth values ​​within an instance, reflecting the spatial orientation of the instance. For example, the depth distribution of an inclined slope increases linearly, while the depth distribution of a flat platform is relatively concentrated.

[0051] Optionally, the occlusion logic includes: foreground instances occlude background instances, and foreground subjects, occluded subjects, and completely occluded instances are marked one by one; The foreground subject is closest to the camera and is either unobstructed or only obstructed by a closer instance, such as the mining truck in front. The obscured entity is partially obscured by a close-up instance, such as a section of a slope obscured by a mining card; the completely obscured instance is completely covered by a close-up instance with no visible area, such as equipment behind a tall stockpile of materials.

[0052] S3.3: Divide the depth layers according to the instance depth sorting results; For each depth layer, generate corresponding structured data; Based on structured data from multiple depth layers, a layered semantic layout representation of the mining area is formed, which includes semantic categories, relative depths, and hierarchical labels. This representation not only preserves the original semantic structure of the mining area's terrain and equipment, but also explicitly gives the occlusion order relationship between different layers and different instances.

[0053] Optionally, the depth layer includes a foreground layer, a midground layer, and a background layer; Optionally, a preset depth threshold strategy and a clustering strategy can be used to divide the depth layers; The specific steps for dividing the depth layers include: for mining areas with relatively fixed terrain and regular distribution of instance depth, several depth thresholds are pre-set, instances with an average depth less than the near-field threshold D1 are classified as the foreground layer, instances between the near-field threshold D1 and the mid-field threshold D2 are classified as the mid-field layer, and instances greater than the mid-field threshold D2 are classified as the background layer. The specific steps for dividing the depth layer include: for mining areas with complex terrain and no obvious pattern in instance depth distribution, K-Means and other clustering algorithms are used to automatically cluster instances into K′ classes based on the average depth of instances, thereby achieving adaptive layering.

[0054] Optionally, the structured data includes a layer semantic layout graph, a layer-corresponding depth graph, and inter-layer occlusion relationships; The specific steps for generating a layer semantic layout graph include: extracting all semantic instances within the layer, retaining their category, position, and shape information, removing instances from other layers, and forming a semantic layout graph that only contains the content of the layer; The specific steps for generating the depth map corresponding to the layer include: extracting the depth region corresponding to the layer instance from the global depth map D, retaining its relative depth information, and forming a depth map that is aligned with the layer semantic layout graph space; The specific steps for recording the interlayer occlusion relationship include: clarifying the stacking order between each layer (e.g., "foreground layer > midground layer > background layer", indicating that the foreground layer occludes the midground layer, and the midground layer occludes the background layer), and marking the boundary areas between layers (e.g., the boundary between the foreground mine car and the midground slope).

[0055] Optionally, the semantic layout map and corresponding depth map of each layer are encoded into multi-channel feature maps or multi-resolution pyramid representations to construct a hierarchical layout with different resolutions, which can be adapted to the multi-scale feature processing of Diffusion Transformer.

[0056] This invention preserves the semantic structure of the mining area while explicitly modeling the occlusion order between vehicles and objects such as slopes and stockpiles, providing constraints for geometric consistency in the video generation process.

[0057] S4: Under the condition of layered semantic layout representation of the mining area, a DiT denoising network is used to generate driving videos that conform to the geometric and occlusion constraints of the mining area.

[0058] Optionally, in the generated image space, according to the layered layout obtained in step S3, the generated results of each frame can be synthesized layer by layer in the order of background layer, mid-ground layer, and foreground layer, and the boundary areas between layers can be reconstructed and smoothed in a small range to further ensure the consistency of the occlusion relationship between vehicles, operating equipment and slopes and stockpiles at the pixel level, and suppress geometrically unreasonable phenomena such as vehicles being incorrectly occluded by distant terrain or objects being suspended in mid-air.

[0059] Optionally, step S4 specifically includes the following steps: S4.1: Select the driving video of the mining area from the target perspective, input each frame of the image into a convolutional encoder or variational autoencoder, map it to a low-dimensional latent feature space, and obtain a sequence of latent feature frames; The latent features of each frame are divided into several fixed-size non-overlapping patches. Each patch corresponds to a spatial location. The corresponding patches of all frames are concatenated along the time dimension to form a spatiotemporal token sequence. After fusing the semantic layout map, depth map, and hierarchical occlusion labels of the mining area, the data is input into a convolutional encoder to extract a low-dimensional conditional feature map. The conditional feature map is divided into several fixed-size non-overlapping patches, each patch corresponding to a spatial location. The corresponding patches of all frames are concatenated along the time dimension to form a conditional token sequence. S4.2: Gaussian noise is progressively added to the spatiotemporal token sequence and the conditional token sequence, and then fused to obtain a noisy video token sequence. The expression is:

[0060] in, These are the fidelity coefficients obtained by multiplying the noisy scheduling sequence. Indicates time step t Noisy video potential token sequence, Represents a clean sequence of potential video tokens. Standard Gaussian noise added during the forward diffusion process. Indicates a normal distribution. Represents the unit covariance matrix. S4.3: Concatenate the noise token sequence, the spatiotemporal token sequence, and the conditional token sequence to generate a concatenated token sequence; S4.4: Order t =1, when t =1 indicates the initial time step; S4.5: Input the concatenated token sequence into the time step. t DiT denoising network A t ; The conditional token sequence is injected into each Transformer layer as a time step using cross-attention. t Global constraints; Based on time step t Global constraints, to obtain time steps t Predictive noise; Calculate time steps t The loss values ​​of predicted noise and actual noise are compared based on the noise loss function and the loss value update time step. t Denoising network parameters to obtain time steps t The updated noise reduction network serves as a time step. t +1 noise reduction network; S4.6: Judgment t Is it greater than or equal to? T , T Let represent the total time steps. If yes, the final denoising network is obtained; otherwise, let . t = t +1, return to step S4.5; S4.7: Obtain a clean sequence of potential video tokens based on the final denoising network; The clean latent video token sequence is input into the convolutional decoder. Through multiple deconvolution and upsampling operations, the low-dimensional latent features are restored into high-resolution pixel images. All frames are concatenated to obtain a driving video that conforms to the geometry and occlusion constraints of the mining area.

[0061] Optionally, the expression for the noise loss function is:

[0062] in, This represents the trainable parameters of the Diffusion Transformer. This represents a condition vector or sequence of condition tokens obtained from layout and depth conditional encoding. This represents the core training loss of the diffusion model. It represents the mathematical expectation, which is the joint expectation of three random variables: clean latent token, real noise, and time step. This represents the standard Gaussian noise added during the forward diffusion process. This represents a denoising network. Represents a noisy video latent token sequence at time step t.

[0063] Optionally, the network architecture of the denoising network adopts a diffusion transformer. The denoising network possesses the ability to model spatiotemporal dependencies, the ability to constrain mining conditions, and the ability to distinguish deep hierarchical levels.

[0064] The denoising network adopts a stacked structure of multi-layer Transformer encoders, with each layer containing two core sub-modules: Submodule 1: Self-Attention. Function: Models pairwise dependencies within the spatiotemporal token sequence. It can capture the correlation between different spatial regions within the same frame (such as the connection between a slope and a road) and the spatiotemporal correlation across frames (such as the continuous movement of vehicles and the inter-frame consistency of slopes), thus solving the problems of inter-frame flickering and inconsistent motion in video generation.

[0065] Submodule 2: Feedforward Network (FFN) Function: Performs independent nonlinear feature transformation on each token to improve the network's feature representation ability and refine the detailed features of the mining scene (such as the texture of the material pile and cracks in the road).

[0066] The denoising network employs a Diffusion Transformer structure with multiple layers of self-attention and feedforward sub-layers stacked together. By modeling the spatiotemporal dependencies within the video using a self-attention module, cross-frame motion and object deformation information can be captured. By injecting layout and depth condition tokens into each Transformer layer through cross-attention, the network is always constrained by the semantic structure of the mining area and the depth occlusion information during the denoising process. In the attention weight calculation, a decay term based on depth difference is introduced, which gives higher attention weight to tokens with similar depths, while the association between foreground and background tokens with large depth differences is suppressed.

[0067] The expression for depth-constrained self-attention is:

[0068] in, These are the query and key vectors, respectively. For feature dimension, This corresponds to the depth value of the token. The hyperparameters for controlling the intensity of the depth difference penalty.

[0069] For the first i The video token is for the first j The original attention score of each video token. for Optionally, the image space is composed layer by layer in the order of foreground layer, midground layer and background layer. Local reconstruction and edge smoothing strategies are used in the boundary areas between layers to further improve the pixel-level consistency of the occlusion relationship between the vehicle and the slope and the material pile, and suppress phenomena such as objects being suspended or incorrectly occluded by distant terrain.

[0070] In this invention, the target video sequence is encoded into a corresponding spatiotemporal feature sequence in the latent space, and a forward diffusion process is applied to introduce progressively increasing Gaussian noise, resulting in a noisy latent token sequence at each time step. Simultaneously, the semantic layout map, depth map, and hierarchical occlusion labels of the mining area are mapped to conditional feature maps via a convolutional encoder, then divided into spatiotemporal patches and flattened into conditional token sequences. The denoising network employs a DiffusionTransformer structure with stacked multi-layer self-attention and feedforward networks. The noisy latent video tokens are superimposed with the time step encoding and viewpoint encoding and input into the denoising network, using self-attention to model the spatiotemporal dependencies within the video. Cross-attention injects layout and depth-based conditional tokens into each Transformer layer, outputting the prediction result for the current noise component to guide subsequent progressive denoising. Depth-related weights or biases are introduced into the attention calculation, giving tokens at similar depth layers higher attention weights while limiting erroneous associations between foreground and background, thus explicitly considering occlusion relationships during generation. The latent video tokens are updated after depth-constrained attention. This means that during the back-diffusion stage, the potential token sequence starting from random noise is gradually denoised under conditional guidance, and is finally restored by the decoder into a video frame sequence that conforms to the semantic layout and occlusion geometric constraints of the mining area.

[0071] Through step S4, this embodiment achieves automatic synthesis of driving videos that conform to the semantics and occlusion geometric constraints of the mining area by utilizing the layout and depth conditions obtained from the migration of structured road datasets such as nuScenes, without relying on real mining area point clouds and BEV world models.

[0072] Through the above implementation methods, this invention utilizes structured road scene data such as nuScenes, and through semantic structure transfer, depth and occlusion perception layout modeling, and conditional video diffusion generation based on Diffusion Transformer, it achieves driving video synthesis with realistic occlusion geometry and mining operation semantics without relying on real mining area 3D point clouds and BEV world models. This can provide an efficient, low-cost, and controllable source of training and testing data for autonomous driving perception and simulation in mining areas.

[0073] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating unstructured road driving videos based on depth occlusion constraints, characterized in that, include: S1: Acquire driving images in a structured scene; After time synchronization and calibration alignment of driving images, a semantic segmentation network trained for structured road scenes is introduced to perform semantic annotation and instance segmentation to obtain the semantic category and instance contour of the driving images; the semantic category and instance contour of the driving images are discretized into a regular grid to construct a semantic layout map of the urban scene. S2. Based on the preset semantic mapping rules, the structured road scene in the urban scene semantic layout map is transferred to the mining area scene to generate a mining area semantic layout map. S3: Semantic layout map of the mining area Input the layout into the depth prediction network, and output the predicted depth map of the mining area; Multiple semantic instances were located in the semantic layout map of the mining area using instance segmentation technology; For each semantic instance, depth statistics are performed to obtain an average depth, and then a global sort is performed to obtain the instance depth sorting result; Based on the sorting results, clarify the occlusion logic between instances and mark the occlusion relationships: Based on the instance depth sorting results, divide the data into depth layers and generate the corresponding structured data; Based on structured data from multiple depth layers, a layered semantic layout representation of the mining area is formed; S4: Under the condition of layered semantic layout representation of the mining area, a DiT denoising network is used to generate driving videos that conform to the geometric and occlusion constraints of the mining area; the network architecture of the denoising network adopts Diffusion Transformer.

2. The method for generating unstructured road driving videos based on depth occlusion constraints according to claim 1, characterized in that, Step S1 specifically includes the following steps: S1.1 Select driving images containing multi-view camera and vehicle pose information, and read the intrinsic and extrinsic parameters, timestamps and vehicle trajectory data of each view camera. Distortion correction, temporal alignment and resolution unification are performed on the driving images. Samples with severe occlusion, blurred images or missing annotations are removed to obtain the filtered images. The images of consecutive frames are combined into several urban driving scenes according to the scene division; S1.2 Utilize a semantic segmentation network to perform pixel-level semantic classification on the filtered images and label the category of each pixel in the driving image; An instance segmentation network is used to extract instance-level contours and locations from the filtered images, distinguishing different individuals under the same semantic category and obtaining the contour boundaries and spatial locations of each instance in the filtered images. The double segmentation result of the filtered image is based on the category label of each pixel and the contour boundary and spatial location of each instance. The double segmentation results of the filtered images are projected onto a uniform fixed-resolution regular raster to form a two-dimensional grid map, which serves as a semantic layout map of the urban scene.

3. The method for generating unstructured road driving videos based on depth occlusion constraints according to claim 1, characterized in that, Step S2 specifically includes the following steps: S2.1: Based on the characteristics of open-pit mining operations, establish a mapping table from the structured road scene semantic categories to the mining area semantic categories in the urban scene semantic layout map, and obtain the semantic mapping results; An end-to-end mapping network is constructed. A batch of urban scene semantic layout maps are taken as input. The preliminary semantic layout of the mining area is output according to the above semantic mapping results. At the same time, the mapping residual is calculated to iteratively update the mapping network and obtain the first updated semantic layout of the mining area. S2.2: Based on the semantic mapping results, the semantic layout of the mining area updated for the first time is structurally adjusted and geometrically reconstructed to generate a reconstructed mining area layout; the reconstructed mining area layout is a semantic layout of the mining area with geometric features and connectivity. S2.3: Based on the operating mode and safety distance requirements of vehicles in the mining area, the main operating entities are automatically arranged on the reconstructed mining area layout to obtain a semantic layout map of the mining area.

4. The method for generating unstructured road driving videos based on depth occlusion constraints according to claim 3, characterized in that, The specific steps in S2.1 include: S2.11 Let k=1, when k=1, it represents the first iteration; S2.12 Input the semantic layout map of the city scene in the kth round into the mapping network in the kth round, complete the semantic category conversion, and output the preliminary semantic layout of the mining area in the kth round; S2.13 Obtain the k-th round mapping residual between the initial semantic layout of the mining area and the ideal semantic layout of the mining area; S2.14 Input the k-th round mapping residual and the k-th round urban scene semantic layout map into the residual learning network, train the residual learning network based on the loss function, and obtain the k-th round updated residual learning network as the k+1 round residual learning network; S2.15 Update the k-th round mapping network based on the k-th round updated residual learning network to obtain the k-th round updated mapping network, which serves as the k+1-th round mapping network; S2.16 Input the k-th round city scene semantic layout graph into the k-th round update mapping network to obtain the k-th round optimized mining area semantic layout, which is used as the k+1 round city scene semantic layout graph; S2.17 Determine whether k is greater than or equal to K, where K represents the total number of iterations. If yes, obtain the semantic layout of the mining area updated for the first time. If no, let k = k + 1 and return to step S2.

12.

5. The method for generating unstructured road driving videos based on depth occlusion constraints according to claim 3, characterized in that, The specific steps of step S2.2 include: The width of urban lanes is scaled up to ensure that the passable area is wide enough to meet the needs of mining vehicles for passing and turning. According to the preset elevation difference and slope parameters, the positions of platforms, slopes between platforms and side slopes are arranged to form a mining area geometric structure with stepped or multi-platform characteristics. Generate the outline of the stockpile block in the specified area, and mark the toe of the stockpile slope and the top area of ​​the stockpile; While ensuring accessibility and safe distances, connectivity checks are conducted on accessible areas to ensure that vehicles can form closed or semi-closed traffic flows between loading and unloading areas.

6. The method for generating unstructured road driving videos based on depth occlusion constraints according to claim 1, characterized in that, Step S3 specifically includes the following steps: S3.1: Construct the original layout to the deep prediction network with the encoder-decoder structure as the backbone; The original layout-to-depth prediction network is trained to obtain the layout-to-depth prediction network. semantic layout map of the mining area Input the layout into the depth prediction network, output the predicted depth map of the mining area. ; S3.2: Predicted depth map of the mining area Semantic layout map of mining area L Multiple semantic instances were located using instance segmentation technology; For each semantic instance, depth statistics are performed within its corresponding region to obtain several core metrics, including: average depth, depth range, and depth distribution shape. Based on the average depth, a global sort of all semantic instances is performed to obtain the instance depth sorting result; Based on the sorting results, clarify the occlusion logic between instances and mark the occlusion relationships: S3.3: Divide the depth layers according to the instance depth sorting results; For each depth layer, generate corresponding structured data; Based on structured data from multiple depth layers, a layered semantic layout representation of the mining area is formed, which includes semantic categories, relative depths, and hierarchical labels. This representation not only preserves the original semantic structure of the mining area's terrain and equipment, but also explicitly gives the occlusion order relationship between different layers and different instances.

7. The method for generating unstructured road driving videos based on depth occlusion constraints according to claim 1, characterized in that, Step S4 specifically includes the following steps: S4.1: Select the driving video of the mining area from the target perspective, input each frame of the image into the encoder, and generate a spatiotemporal token sequence; The semantic layout map, depth map, and layered occlusion labels of the mining area are fused together and then input into the encoder to generate a conditional token sequence. S4.2: Add Gaussian noise to the spatiotemporal token sequence and the conditional token sequence and then fuse them to obtain a noisy video token sequence; S4.4: Order t =1, when t =1 indicates the initial time step; S4.5: Input the concatenated token sequence into the time step. t DiT denoising network A t ; The conditional token sequence is injected into each Transformer layer as a time step using cross-attention. t Global constraints; Based on time step t Global constraints, to obtain time steps t Predictive noise; Calculate time steps t The loss values ​​of predicted noise and actual noise are compared based on the noise loss function and the loss value update time step. t Denoising network parameters to obtain time steps t The updated noise reduction network serves as a time step. t +1 noise reduction network; S4.6: Judgment t Is it greater than or equal to? T , T Let represent the total time steps. If yes, the final denoising network is obtained; otherwise, let . t = t +1, return to step S4.5; S4.7: Obtain a clean sequence of potential video tokens based on the final denoising network; The clean latent video token sequence is input into the convolutional decoder. Through multiple deconvolution and upsampling operations, the low-dimensional latent features are restored into high-resolution pixel images. All frames are concatenated to obtain a driving video that conforms to the geometry and occlusion constraints of the mining area.

8. The method for generating unstructured road driving videos based on depth occlusion constraints according to claim 7, characterized in that, The token sequence expression for noisy video is: in, These are the fidelity coefficients obtained by multiplying the noisy scheduling sequence. Indicates time step t Noisy video potential token sequence, Represents a clean sequence of potential video tokens. Standard Gaussian noise added during the forward diffusion process. Indicates a normal distribution. This represents the unit covariance matrix.

9. The method for generating unstructured road driving videos based on depth occlusion constraints according to claim 1, characterized in that, The occlusion logic is that foreground instances occlude background instances, and foreground subjects, occluded subjects and completely occluded instances are marked one by one; The foreground subject is closest to the camera and is either unobstructed or only obstructed by a closer instance, such as the mining truck in front. The obscured entity is partially obscured by a close-up instance, such as a section of a slope obscured by a mining card; the completely obscured instance is completely covered by a close-up instance with no visible area, such as equipment behind a tall stockpile of materials.

Images