Controllable video generation method and device for off-road driving

Abstract

The application discloses a controllable video generation method and device for off-road driving. Real off-road driving reference images and target control conditions are obtained, normalized depth maps, binary edge maps, drivable area masks and road vanishing points are extracted, target condition change degrees and scene structure complexities are calculated, and migration strength, depth control strength and edge control strength are initialized within a preset upper and lower limit. The above strengths are updated according to the target condition consistency and structure retention result feedback of the candidate initial frame; a candidate video is generated taking the qualified initial frame as a visual anchor point, motion rationality is calculated according to the forward direction consistency of road area optical flow relative to the vanishing point, the motion amplitude, the speed change and the camera motion, and the video control strength is updated in combination with the timing stability feedback. The first frame and the video feedback are terminated when the corresponding threshold or the upper limit of the number of times is reached, and the generated samples fed back are subjected to quality screening, near-duplicate detection and metadata recording to form a long-tail off-road driving sample set.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and intelligent driving data generation technology, and in particular to a controllable video generation method and apparatus for off-road driving. Background Technology

[0002] Off-road driving scenarios are widespread in agriculture, mining areas, field inspections, emergency rescue, and wilderness robotics. Compared to urban roads, off-road roads typically lack clear lane markings and regular road topology, have indistinct road boundaries, complex terrain textures, irregular drivable areas, and are easily affected by environmental factors such as rain, snow, fog, dust storms, water accumulation, and low light. In real-world data collection, severe weather, dangerous road surfaces, and rare terrains occur infrequently, requiring supplementary data collection typically involves long periods and necessitates the entry of equipment and personnel into hazardous environments.

[0003] CN117170560B discloses an image transformation method that acquires a target noise image associated with a first image, as well as current scene information such as weather, date, or time. It then forms a target text prompt based on the current scene information and the original text prompt corresponding to the first image, and uses a diffusion model to process the target text prompt and the target noise image to generate a transformed second image. CN117835012B discloses a controllable video generation method that extracts appearance guidance, motion guidance, and depth map conditional guidance from the input video and text. It generates I-frames based on appearance guidance and depth map conditional guidance, and generates B-frames based on motion guidance and I-frames to maintain inter-frame continuity and cross-shot temporal consistency in the generated video. While these methods can utilize scene information, text, depth maps, or optical flow to guide image or video generation, they do not disclose adaptively determining migration strength and structural control strength based on the quantization difference between the reference image and target control conditions, as well as the structural complexity of the reference scene. Furthermore, they do not disclose updating the corresponding control strength within a preset range and a limited number of iterations based on the target condition consistency and structural preservation results of the generated first frame, as well as the motion rationality and temporal stability of the generated video. When using preset generation parameters in practice, significant differences between the reference scene and the target conditions may result in insufficient representation of the target conditions, while smaller differences may lead to over-transfer. In off-road scenarios, vegetation, gravel, slopes, and weak road boundaries can make it difficult for uniform structural control parameters to simultaneously account for appearance changes and maintain road layout. Furthermore, when only using prompts to describe "stable progress," the generated data still lacks a deterministic calculation process that automatically identifies and corrects for stationary states, backward movements, sudden speed changes, and camera shake based on the actual results.

[0004] In summary, existing technologies still suffer from the following problems: The generation control parameters mainly rely on preset values ​​or manual adjustment, failing to adaptively determine migration and structural control strength based on the actual differences between the reference image and target control conditions, as well as the complexity of the scene structure; the quality evaluation results after generation are usually not fed back to the generation parameters, making it difficult to provide targeted corrections for issues such as insufficient expression of target conditions, road structure damage, video stillness, backward movement, sudden speed changes, camera shake, and inter-frame flicker; furthermore, the existing generation process lacks a parameter update mechanism constrained by preset upper and lower limits, a maximum number of feedback updates, and clear pass and fail exits, easily leading to repeated parameter trials, difficulty in determining the termination of the processing, and difficulty in tracking the generation results. Therefore, a controllable video generation method is needed that can adaptively initialize generation control parameters and update the corresponding control strength within a limited number of iterations based on the quantitative evaluation results of the first frame and the video. Summary of the Invention

[0005] The purpose of this invention is to provide a controllable video generation method and apparatus for off-road driving. The method uses the variability of target conditions and the complexity of the scene to initialize the generation parameters, and feeds back the consistency of target conditions and the structure preservation results of the initial frame, as well as the motion rationality and temporal stability results of the video, to the corresponding generation parameters, forming a two-level closed loop with an upper limit on the number of feedback updates and a failure exit.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A controllable video generation method for off-road driving, characterized by the following steps: S101, acquire a real off-road driving reference image and target control conditions, and perform size and color normalization processing on the real off-road driving reference image. The target control conditions include at least one of terrain, weather, road surface condition and lighting condition. S102, obtain the reference condition confidence vector corresponding to the real off-road driving reference image, extract the normalized depth map, binary edge map, drivable area mask and road vanishing point, calculate the target condition change degree U according to the difference between the reference condition confidence vector and the target condition state vector, and calculate the scene structure complexity R according to the grayscale distribution of the real off-road driving reference image, the binary edge map and the normalized depth map, where U, R∈[0,1]; S103, construct image migration control information and video control information according to the target control conditions, and initialize migration intensity α and depth control intensity between their respective preset upper and lower limits according to the target condition variability U and the scene structure complexity R. and edge control strength ; S104, set the first frame feedback update count n = 0, generate candidate target condition initial frames based on the image migration control information, the normalized depth map, the binary edge map, and the initialized control intensities, and calculate the target condition consistency score C, structure preservation score S, depth structure component score, and edge structure component score of the candidate target condition initial frames; when C or S does not reach the corresponding threshold and n is less than the maximum first frame feedback update count N. I At that time, based on the corresponding deficiency amount of the above scores, α is updated within the preset upper and lower limits. and Increment n by 1 and regenerate; determine the candidate target condition initial frame where both C and S reach the corresponding threshold as the target condition initial frame, and n equals If C or S still has not reached the corresponding threshold, a first frame generation failure flag will be output, and the current video generation process will be terminated. S105, re-extract the drivable area mask and road vanishing point of the initial frame of the target conditions, and combine the initial frame of the target conditions with the video control information obtained in step S103 as video generation input; construct a forward motion direction template based on the drivable area mask and road vanishing point of the initial frame of the target conditions, and initialize the forward motion control intensity within their respective preset upper and lower limits. and timing consistency control strength And generate candidate off-road driving videos; S106, set the video feedback update count j = 0, and calculate the motion rationality score M and temporal stability score T of the candidate off-road driving video; if either score does not reach the corresponding threshold and j is less than the maximum video feedback update count N, the video feedback update will be updated. V At that time, updates will be made within their respective preset upper and lower limits based on the corresponding insufficient scores. and Increment j by 1 and regenerate candidate off-road driving videos, recalculating the motion rationality score and temporal stability score; determine the candidate off-road driving videos whose motion rationality score M and temporal stability score T both reach the corresponding thresholds as target off-road driving videos, where j equals N. V If the threshold is still not reached, the candidate off-road driving video is removed, the video generation failure status is recorded, and the initial frame of the target condition is retained for processing in step S107. S107, Perform comprehensive quality screening and near-duplicate detection in the same batch on at least one of the target condition initial frame and the target off-road driving video; S108 records the target control condition labels, reference image identifiers, generation parameters, quality scores, and timestamps of the generated samples after filtering and deduplication, and constitutes a long-tail off-road driving sample set.

[0007] Furthermore, the target condition categories specified this time are formed into a non-empty set A of active conditions; for the k-th condition in A, let Let X be the candidate label probability vector output by the conditional classifier for image X, and let X be the reference conditional confidence vector. Target conditional state vector For target label The corresponding one-hot vector, the degree of change of the target condition for: ; in, The image represents the actual off-road driving reference image, X represents the image to be classified, and k represents the active condition index. This represents the number of conditions contained in the active condition set A. Describing the L1 norm, Indicates the first The class condition classifier applies to the real off-road driving reference image. Output target label The probability of an unspecified condition; conditions not specified are not added to the active condition set A and are not included in the calculation.

[0008] Furthermore, the real off-road driving reference image is divided into L grayscale intervals, where L≥2, and the pixel ratio of the l-th grayscale interval is... The image height H and width W are both not less than 2, and the number of edge pixels in the binary edge map is... Normalized depth map is Then the normalized gray entropy Edge density Normalized depth changes The scene structure complexity R is as follows: , , , ; in, =0 is the rule ;in, and These are the pixel row index and pixel column index, respectively. To normalize the depth map at pixel positions Depth value at that location, It represents the natural logarithm.

[0009] Furthermore, let the intermediate variables : The initial values ​​for the migration intensity, depth control intensity, and edge control intensity are as follows: , , , Among them, each minimum value is less than the corresponding maximum value; unenabled deep control branches or edge control branches do not have corresponding control strengths set and do not participate in subsequent updates.

[0010] Furthermore, for the nth generated candidate target conditional initial frame The scores for consistency of objective conditions and preservation of structure are as follows: , , ; in, for Normalized cosine similarity within the same semantic feature space as the target condition positive description. For reference image and Depth-related scores, The intersection ratio of their edge regions. The intersection ratio of their drivable areas. The inverse score is the normalized distance between the vanishing points of the two regions; when the union of the two regions to be compared is empty, the intersection-union ratio is set to 1. Initial frame for candidate target conditions The corresponding minimum target label confidence, For the first A conditional classifier conditions the initial frame of candidate targets. Output target label The probability of.

[0011] and, , , , ; , , Among them, threshold and All are greater than 0 and not greater than 1. This represents the larger value of z compared to 0; , , ; in, Each step size is a positive number; only when ≥ and ≥ Video generation begins at that time; and These are the scores for the depth structure and the edge structure, respectively. , , and These are respectively: insufficient consistency of target conditions, insufficient structure preservation, insufficient deep structure, and insufficient edge structure. and These are the target condition consistency threshold and the structure preservation threshold, respectively; , and These are the migration intensity update step size, depth control intensity update step size, and edge control intensity update step size, respectively.

[0012] Furthermore, let the road vanishing point of the initial frame of the target condition be... , >0, the template for the fixed forward motion direction at position p is: , The candidate off-road driving videos include Frame and ≥3, The actual optical flow from frame t to frame t+1. For the corresponding road vanishing point, For the optical flow evaluation region within the drivable area, excluding the neighborhood of the vanishing point, the frame-by-frame forward motion direction template is: When any When M is empty, let M = 0. When neither is empty, calculate each component of motion using the following formula: , , ; in, For the first The region-normalized average optical flow amplitude corresponding to the frame. The normalized average optical flow amplitude over time is the average value of the region. To preset the minimum motion amplitude, Scoring exists for each sport; The image diagonal >0; Forward direction score From actual optical flow and The consistency of the nonnegative weighted cosines between them is determined; , ; in, and These are the global translation magnitude and rotation angle magnitude, normalized to [0,1], respectively. For speed variation error, To score points for consistent speed; ; ; The optical flow compensation residual, the change in adjacent optical flow, and the difference in the color histogram of adjacent frames were normalized to: ; , , ; , ; ; ; in, For camera motion variation error, To ensure the camera scores a stable score, and These are insufficient exercise and insufficient timing, respectively. and These are the motion rationality threshold and the temporal stability threshold, respectively. and These are the forward motion control intensity update step size and the temporal consistency control intensity update step size, respectively; threshold. and All are greater than 0 and not greater than 1. >0, >0; , This represents the larger value of z compared to 0; < , < ; Only when M and T respectively reach and Output the target off-road driving video in real time; in, The normalized average optical flow amplitude for the region. The normalized average optical flow amplitude over time is the average value of the region. To preset the minimum motion amplitude, Scoring exists for each sport; and These are the normalized global translation amplitude and the normalized global rotation angle amplitude, respectively.

[0013] Furthermore, in S101, the real off-road driving reference image is cropped according to a preset aspect ratio, and the drivable road surface area, road boundary, road center direction and distant vanishing point are preserved in the cropping process; The image migration control information includes positive and negative prompts. The positive prompts include reference scene, target terrain, target weather, target road surface condition, target lighting condition, forward off-road driving perspective, and realistic description. The negative prompts include descriptions for suppressing incorrect weather, incorrect terrain, urban road features, vehicle interior components, non-target traffic participants, blur, overexposure, color shift, duplicate textures, road structure distortion, and generated artifacts. The video control information also includes scene structure constraint information for maintaining road position, road extension direction and vanishing point direction, as well as motion timing constraint information for maintaining forward motion and inter-frame consistency and suppressing stillness, backward movement, shaking, speed change, viewpoint jump, color change and background flicker.

[0014] Furthermore, the target condition consistency score C, structure preservation score S, sharpness score F, exposure and color score E, and sample difference score D are all normalized to [0,1]. For the target off-road driving video, C and S are taken from the initial frame score of the visual anchor point, F and E are taken from the arithmetic mean of the corresponding scores of each frame, and D is calculated based on the features of the entire video segment; the initial frame comprehensive quality score is... Overall video quality score Both the overall threshold and the individual threshold are located in [0,1]. When the overall quality score reaches the overall threshold and all applicable individual scores except D reach the individual threshold, the generated sample passes the quality screening. For samples that pass the test within the same batch and belong to the same sample type and have the same target control condition labels, they are arranged in descending order of comprehensive quality score, ascending order of timestamp, and ascending order of sample identifier; in the... In the round-robin repetition process, let the set of samples to be processed be... Select The sample ranked first in the middle is used as the representative sample. and form near-repeating sets. : ; Retain representative samples And remove near-duplicate sets from the current samples to be processed. Repeat the above process until there are no more samples to be processed. in, This indicates the sample currently awaiting processing. Indicates the current sample to be processed. Representative sample Normalized sample similarity between them This indicates the preset near-repetition threshold.

[0015] The present invention also provides a controllable video generation device for off-road driving. The device includes an image and condition processing module, an adaptive parameter initialization module, a first frame closed-loop generation module, a video closed-loop generation module, and a filtering and output module. Each module is used to collaboratively execute the above steps S101 to S108, and outputs a determined failure state when each level of feedback reaches the maximum number of feedback updates.

[0016] The beneficial effects achieved by this invention are: 1. This invention characterizes the degree of change of the real off-road driving reference image relative to the target control conditions through the target condition variability (U), and comprehensively characterizes the grayscale distribution, edge density and depth change of the reference image through the scene structure complexity (R). Based on these two factors, the migration strength, depth control strength and edge control strength are initialized within preset upper and lower limits, so that the generated parameters have a clear calculation basis and are easy to verify and repeat.

[0017] 2. In the first frame generation stage, the present invention uses the target condition consistency score and the structure preservation score as mutually restrictive feedback quantities. When the target condition expression is insufficient, the migration intensity is increased and the structural constraints are appropriately reduced. When the depth structure or edge structure preservation is insufficient, the excessive migration intensity is reduced and the corresponding structural control intensity is increased, thereby taking into account both the target condition migration and the preservation of the reference scene structure, and reducing the need for repeated manual adjustment of parameters.

[0018] 3. In the video generation stage, this invention does not use the overall average optical flow direction, which is prone to cancellation due to opposite motion directions in different areas. Instead, it compares the actual optical flow with the frame-by-frame forward motion direction template constructed based on the road vanishing point within the drivable area. It evaluates abnormal situations such as stillness, backward movement, sudden speed changes, and camera shake through motion presence score, forward direction score, speed stability score, and camera stability score, respectively. The lowest score in each sub-item determines the rationality of the motion, which helps to improve the rationality and stability of forward motion in off-road driving videos.

[0019] 4. This invention limits each control intensity to the corresponding preset upper and lower limits, and sets the maximum number of feedback updates and the output for successful generation and failed generation for the first frame feedback and video feedback, respectively; at the same time, the comprehensive quality screening and near-repetition detection are both performed on a limited number of generated samples, so that the image generation, video generation, feedback update, quality screening and sample output processes have clear stopping conditions, forming a generation closed loop that can determine termination.

[0020] 5. This invention records target control condition labels, reference image identifiers, generation parameters, quality scores, and timestamps for generated samples that have been filtered and deduplicated. This enables the generated samples and their generation process to be tracked and verified, and can form a long-tail off-road driving sample set to supplement real-world data collected in scenarios such as severe weather, dangerous road surfaces, and rare terrain. This helps to reduce the need for supplementary data collection for corresponding long-tail scenarios and the frequency of equipment and personnel entering dangerous environments. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the controllable video generation device and data stream for off-road driving according to the present invention.

[0022] Figure 2 This is a flowchart illustrating the overall process of the controllable video generation method for off-road driving according to the present invention.

[0023] Figure 3 A schematic diagram is constructed for the target control conditions, condition state vectors, and control information.

[0024] Figure 4 This is a schematic diagram of the target condition migration module and the input / output relationship of adaptive parameters.

[0025] Figure 5 The flowchart shows the evaluation and parameter feedback process for the initial frame of the candidate target conditions.

[0026] Figure 6 This is a flowchart for the evaluation and parameter feedback of candidate off-road driving videos under forward motion template constraints. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments. For ease of reading, two operators that are used throughout this embodiment are first given. For any number z and upper and lower bounds a and b, the truncation function clip and the positive part function are defined as follows: , (1); Equation (1) ensures that each parameter update falls within the preset range. The threshold, upper and lower limits of the parameters, step size, and maximum number of feedback updates mentioned below are all written into the configuration file before the generation begins and remain unchanged in the same generation task.

[0028] Example 1: A Controllable Video Generation Method for Off-Road Driving Combination Figure 2 In Example 1, steps S101 to S108 are executed sequentially.

[0029] Step S101: Reference image acquisition and preprocessing: A real off-road driving reference image I is acquired through a file reading interface, database interface, or camera data interface. I is then cropped according to the aspect ratio of the target video, with the cropping window prioritizing the drivable area at the bottom of the image, the left and right road boundaries, the road center extension direction, and the vanishing point in the distance. Subsequently, the resolution, RGB color space, pixel range, and file format are standardized. This processing does not require altering the original image content; it only ensures consistent input for subsequent conditional classification, structure extraction, and generation modules. The target control conditions must specify at least one of the following: terrain, weather, road surface condition, and lighting condition. Camera motion, viewpoint, and temporal constraints may also be added. The explicitly specified condition categories from the four appearance conditions (terrain, weather, road surface condition, and lighting condition) constitute a non-empty active condition set A. Unspecified conditions are not added to active condition set A and are not included in the target condition variability calculation, thus avoiding the misinterpretation of "no requirement" as "needs to be changed." Step S102: Calculation of the target condition variability U and the scene structure complexity R: For the k-th condition in the active condition set A, the confidence vector of the reference image I is obtained using the corresponding classifier. The vector's components are non-negative and sum to 1; the target label Encoded as one-hot vector The degree of variability for a single item is half the L1 distance between two probability vectors, and the overall degree of variability for the target condition is the average of the degrees of variability for each active condition. (2); Since the L1 distance between the two probability vectors does not exceed 2, U must be in the range [0,1]. U close to 0 indicates that the reference image already meets the target conditions relatively well, while U close to 1 indicates that the confidence of the target conditions in the reference image is low and more obvious appearance transfer is needed. Extract normalized depth maps from reference images Binary edge map Driveable area mask and road vanishing point The depth map is obtained by a monocular depth estimation network and linearly normalized to [0,1]; the edge map can be obtained by a gradient operator or an edge detection network; the drivable region can be obtained by a segmentation network or road region rules; the road vanishing point can be obtained by the intersection of road boundary extensions, a vanishing point detection network, or lane center direction estimation. The grayscale histogram of the reference image is divided into L intervals, where L≥2, and the pixel ratio of the l-th interval is... The image height and width are H and W respectively, and H≥2, W≥2. Let... Normalized gray entropy Edge density Normalized depth changes for: , (3); (4); in, =0 is the rule ; This represents the number of edge pixels. Since the depth map has been normalized, the depth difference between each adjacent pixel does not exceed 1. , ,and All values ​​are within [0,1]. The scene structure complexity is calculated as an equal-weighted average of the three factors: (5) Using equal-weighted averaging is a method that can be performed without sample fitting. Other implementations may also use pre-fixed non-negative weights that sum to 1, but these weights do not change with the generated results in the same task.

[0030] Step S103: Control information construction and parameter initialization: like Figure 3 As shown, positive prompts include a description of the reference scene, target terrain, target weather, target road surface, target lighting, forward off-road driving perspective, and realistic description; negative prompts are used to suppress incorrect weather, incorrect terrain, urban roads, dashboard, windshield, wipers, vehicle hood, non-target traffic participants, blur, overexposure, color shift, duplicate textures, road distortion, and generation artifacts; positive and negative prompts constitute image migration control information; scene structure constraint information is used to maintain road position, road extension direction, vanishing point direction, background geometry, and main contours; motion temporal constraint information is used to maintain stable forward movement and inter-frame continuity, and to suppress stillness, backward movement, large camera shake, sudden speed changes, perspective jumps, scene zooming out, color changes, and background flickering. The above information, combined with image migration control information, forms video control information; Let Z migration intensity Controls the degree to which the appearance of the reference image is altered by target conditions; depth control intensity. Injection weights for controlling deep structural priors; edge control strength Control the injection weights of local edge priors. The initial values ​​for the three are: (6); (7); (8); Equation (6) makes the migration stronger when the target conditions change more significantly, and appropriately reduces the migration increase when the structure is complex; Equation (7) strengthens the overall depth structure when either U or R is large; Equation (8) strengthens the local contour in complex scenes, but avoids completely locking the original reference image texture when U is large. The product terms of the three expressions are all in [0,1], so the initial values ​​naturally fall within the corresponding upper and lower limits. Unenabled structural branches do not have corresponding control strengths set and do not participate in updates.

[0031] Step S104: Initial frame generation and first frame feedback for target conditions: like Figure 4 and Figure 5 As shown, the target conditional transfer module includes an image encoding unit, a text encoding unit, a depth control branch, an edge control branch, and an initial frame generation unit. The reference image is encoded as image features, image transfer control information is encoded as text control features, and the depth map and edge map are encoded as structural constraint features, respectively; the initial frame generation unit... Generate the initial frame of the nth candidate target under weight control. .

[0032] Will visual features With positive prompts Text features Mapping to the same semantic space, calculate normalized cosine similarity. Simultaneously, the target label confidence is calculated using each active conditional classifier, and the minimum value is taken. This is to prevent a poorly expressed objective condition from being masked by other high scores. The objective condition consistency score is: ; , (9); From candidate frames The normalized depth map, binary edge map, drivable area mask, and road vanishing point are re-extracted, and the reference image and the reference image are calculated respectively. Depth-ordering related scores Intersection ratio of marginal regions drivable area intersection ratio and vanishing point retention score : , (10); s v,n =1-min{1,‖v I -v Yn ||2 / √(W²+H²)}, ,(11) in, This represents the rank correlation coefficient after the two depth maps are unfolded into sequences; This represents the intersection-union comparison, where the union of the two regions to be compared is empty. Set it to 1. To control the global depth structure and local edges separately, let... and Threshold and All are greater than 0 and not greater than 1. Define four insufficient normalization quantities: , , , (12); when Increase migration intensity when insufficient; appropriately reduce migration intensity when the overall structure is insufficient, and add corresponding structural controls based on the insufficiency of depth structure and edge structure respectively. The updated formula is: (13); ,(14) (15); The first candidate frame uses n=0 and the initial values ​​of equations (6) to (8). Only when ≥ and ≥ At that time, The initial frame is determined as the target condition and video generation begins. If it fails and... Then, calculate the (n+1)th parameter according to equations (13) to (15), increment n by 1, and repeat this step; if The time has not yet been reached simultaneously and If the first frame generation fails, a flag indicating failure will be output and video generation of the reference image combined with the target conditions will be stopped. This indicates the maximum number of first-frame feedback updates allowed; the first candidate is not counted as an update.

[0033] Steps S105 and S106: Candidate video generation, motion evaluation, and timing feedback: Extract the drivable area mask again from the initial frame based on the target conditions fed back from the first frame. and road vanishing point , and by A template for fixed forward motion direction is constructed around the image; this initial frame is combined with the video control information obtained in step S103 to form the video generation input. Two initial values ​​are used to generate the first candidate video. This fixed template is used to express the forward motion constraints to the generation module.

[0034] In the evaluation phase, drivable areas and road vanishing points are extracted frame by frame from the candidate videos. Let the number of candidate videos be N. F Frame and ≥3, the actual optical flow from frame t to frame t+1 is The vanishing point of the road in frame t is Excluding the vanishing point neighborhood within the drivable region of this frame yields the optical flow evaluation region. .make >0, the fixed template and the frame-by-frame template are respectively: , ,(16) When each When the image is not empty, the optical flow generated by the vehicle moving forward along the road typically diffuses from the vicinity of the road vanishing point to the periphery of the image. Therefore, this embodiment does not directly take the average direction of the optical flow vector of the entire image, but instead calculates the actual optical flow point by point. Consistency. Let the image diagonal... The region-normalized average optical flow amplitude is Its time average is Preset minimum motion amplitude >0, the exercise has a score. and forward direction score for: , , (17) , , (18); in, This is a non-negative weighted directional consistency cumulative amount. As the normalization factor for the optical flow amplitude weight, the contribution of the receding optical flow contracting towards the vanishing point to the numerator of equation (18) is 0. Let the global translation amplitude estimated from the feature points of adjacent frames be normalized according to... Normalization ∈[0,1], the absolute value of the global rotation angle is normalized to π as follows: ∈[0,1]. Velocity variation error Speed ​​and stability scoring Camera variation error And camera stability score for: , ; ; If any If the value is empty, the motion rationality score M is set to 0 and the process proceeds to feedback or failure judgment; otherwise, the motion rationality is assessed using the weakest link principle. ,(19) The minimum value rule ensures that any low score in a static, backward, sudden speed change, or obvious camera shake cannot be masked by the average of other high scores.

[0035] use For frame t, perform optical flow compensation and calculate the normalized pixel difference between the compensated image and frame t+1 within the effective non-occluded region. Normalize the differences in optical flow fields between adjacent and coordinate-aligned fields to The total variation distance of the normalized color histograms of adjacent frames is denoted as... All three terms are in the range [0,1], and the timing stability is: (20); threshold and All are greater than 0 and not greater than 1. >0, >0, and the lower limits of both sets of control strengths are less than their corresponding upper limits. Motion insufficiency and temporal insufficiency are defined as follows: , , Weighted motion constraints such as "maintaining forward movement, suppressing stillness and backward movement, and stabilizing the camera" are applied. Weighted temporal constraints such as "maintaining inter-frame continuity, suppressing flicker, and color abrupt changes" are applied, and both are updated independently and monotonically. , ,(twenty one); The initial candidate video uses j=0 and the aforementioned two midpoint initial values. When and Output the target off-road driving video in real time; otherwise, output the video in real time. The parameters are calculated according to equation (21) for the (j+1)th time, and j is incremented by 1 and then regenerated. If the video still fails to pass the test, the candidate video is removed and the video generation failure status is recorded, but the initial frame of the target condition that has already passed the test is retained and the process proceeds to step S107. This indicates the maximum number of video feedback updates allowed; the first candidate is not counted as an update. Since each update only increases the constraint strength corresponding to the non-compliant item, the update direction directly corresponds to the evaluation deficiency.

[0036] Steps S107 and S108: Quality screening, deduplication, and metadata recording: Online feedback determines whether to regenerate, while quality screening determines whether already generated samples should be added to the database; these two processes occur at different stages. For the initial frame or each frame in the video, the Laplacian response variance or high-frequency energy is normalized to obtain a sharpness score F. Exposure and color scores E are obtained based on overly bright pixels, overly dark pixels, and color channel deviations. When evaluating the video, C and S are calculated using the visual anchor point initial frame score, while F and E are calculated using the arithmetic mean of the corresponding scores for each frame. When evaluating the initial frame, the C, S, F, and E of that frame are directly used. The maximum similarity between the current initial frame or the entire video feature and historical samples of the same sample type with the same target control condition labels is used. Obtaining differences When the historical samples are empty, let D=1.

[0037] One way to implement weights without relying on manual fitting is to use an equal-weighted average: , (twenty two); To prevent the average score from masking key defects, a comprehensive threshold is applied to both the initial frame and the video. , With hard single threshold , The above thresholds are all located in [0,1]. Let and These are the initial frame quality screening pass indicators and the video quality screening pass indicators, respectively: ; ,(twenty three); Other situations or The value of D is 0 and the sample is removed. D reflects the value of the new sample rather than the correctness of the content, so it is not used as a hard threshold for equation (23). Near duplication is handled directly in subsequent steps.

[0038] For samples that pass the test within the same batch and belong to the same sample type and have the same target control condition labels, they are arranged in descending order of comprehensive quality score, ascending order of timestamp, and ascending order of sample identifier; in the... In the round-robin repetition process, let the set of samples to be processed be... Select The sample ranked first in the middle is used as the representative sample. and form near-repeating sets. : (twenty four); in =1, ∈[0,1]. Reserved And remove the entire B from the set to be processed. k Repeat until the set is empty. Because It must belong to Each round deletes at least one representative sample, thus terminating the deduplication process in a limited manner. Each deleted sample directly reaches the near-duplication threshold with the final retained representative sample, avoiding accidental deletion due to chain-like similarity of multiple samples. Image similarity can be achieved using perceptual hashing or visual feature cosine similarity; video similarity can be achieved using the average of keyframe visual similarity and motion trajectory similarity, with all similarities first normalized to [0,1].

[0039] For retained samples, record the target control condition labels, reference image identifiers, and the generated model name or version. The data includes feedback scores, feedback frequency, overall quality score, timestamps, and output status. This information can be associated with the samples in the form of data files, database records, or dataset indexes. The long-tail off-road driving sample set can be used for training, fine-tuning, or evaluation of at least one of the following: terrain recognition, weather recognition, drivable area recognition, scene classification, semantic segmentation, and off-road driving perception models.

[0040] A computational example for illustrating the data flow only: The following values ​​are used to demonstrate how the formula performs closed-loop calculations, and do not represent actual measurement accuracy or effect, nor restrict the parameter values. Suppose the task is to convert a dry gravel road reference image into a rainy, muddy road scene, and the active condition set includes four items: terrain, weather, road surface, and lighting. The classifier gives reference confidence scores of 0.75, 0.10, 0.15, and 0.25 for the four target labels, respectively, and then we can obtain the following from equation (2). .

[0041] like Therefore, R = 0.35. The preset parameter range is: , , , From equations (6) to (8), the following initial values ​​are obtained: ; All of the above values ​​are obtained directly from the input confidence level, image structure index, and preset interval.

[0042] set up = =0.70, Maximum number of first frame feedback updates Assuming the first candidate initial frame with n=0 is calculated as follows: ,but Approximately 0.086 Equation (13) increases If both the depth structure score and the edge structure score reach the threshold, then When the target condition consistency score does not reach the threshold, i.e. According to equations (14) and (15), depth controls intensity. and edge control strength The value will be appropriately reduced under the corresponding preset lower limit constraint to weaken the restriction of structural control on the migration of target conditions. After regenerating with n=1, if... =0.73 and If the score is 0.72, the first frame passes and the current level of feedback stops. The candidate scores here are only used to demonstrate the stopping decision and are not considered experimental results.

[0043] Video stage settings = =0.70, Maximum number of video feedback updates =4, and obtained by taking the midpoint of the upper and lower limits. ) and If the initial candidate video for j=0 has M=0.58 and T=0.74, then only... If greater than 0, increase according to formula (21). After regenerating with j=1, if both M and T reach 0.70, the video is output; otherwise, the process continues until the maximum number of feedback updates is reached. If the video still fails to pass after reaching the maximum, only the video is discarded; qualified initial frames can still be filtered. Thus, the reference image, target conditions, initial parameters, first frame feedback, video feedback, and final state form a data chain that can be recalculated item by item.

[0044] Optional model implementation in this embodiment: The above computational logic does not depend on a specific model brand. The target conditional transfer module can employ a conditional diffusion model with image conditions and structural control branches; for example, in a non-limiting implementation, an SDXL image-to-image architecture can be used, with depth prior obtained using DepthAnythingV2, edge prior obtained using the Canny operator, and a 25-step denoising process, a guiding coefficient of 6.5, and a DPM++2M-Karras sampler. The listed names and parameters illustrate only one alternative implementation. The initialization and feedback updates will still be performed according to the aforementioned formula.

[0045] The controllable video generation module can employ an image-to-video diffusion model, a diffusion network with temporal attention, or a video diffusion Transformer. For example, in a non-limiting implementation, the 14B image-to-video model from the Wan2.1 series can be used to output a 1280×720 video at 49 frames per second (fps). The target conditional initial frame can be directly fixed as the first frame, or it can be encoded as conditional features used in each denoising stage. and It acts on motion condition features and temporal consistency condition features, respectively.

[0046] The threshold and the upper limit for the number of feedback updates can be preset values ​​that can be executed without training, for example... This means that, beyond the initial candidate, a maximum of four feedback updates will be performed. The step size can be half the length of the corresponding parameter range. The above values ​​are only used to disclose one complete implementation. In actual deployment, different fixed values ​​can be written before generation based on processing resources and quality requirements, but the stopping rule cannot be arbitrarily changed in the same feedback loop.

[0047] Example 2: In addition to the content of the above examples, it also includes a controllable video generation device for off-road driving, combined with... Figure 1 The controllable video generation device for off-road driving includes an image and condition processing module, an adaptive parameter initialization module, a first-frame closed-loop generation module, a video closed-loop generation module, and a filtering and output module. The device can consist of at least one processor, a memory, an image input / output interface, and a computer program stored in the memory and executed by the processor. Alternatively, it can consist of multiple communicating software services or computing nodes, and be deployed on a cloud server, edge computing device, driving data processing platform, or model training platform.

[0048] The image and condition processing module executes steps S101 and S102: acquiring reference images via files, databases, or camera interfaces; receiving target control conditions via user interfaces, configuration files, or program interfaces; invoking condition classification, monocular depth estimation, edge detection, drivable region segmentation, and vanishing point detection programs; and outputting... ; The adaptive parameter initialization module outputs according to equations (2) to (8). The upper and lower limits of parameters, step size, threshold and maximum number of feedback updates are written into the current task status. The first frame closed loop generation module includes an image encoding unit, a text encoding unit, a structure control unit, an initial frame generation unit and a first frame evaluation unit; the initial frame evaluation unit outputs C, S and their sub-scores, the parameter update unit updates the parameters according to equations (12) to (15) and returns the updated values ​​to the initial frame generation unit.

[0049] The video closed-loop generation module includes an initial frame encoding unit, a video control information encoding unit, a timing generation unit, a video decoding unit, an optical flow and camera motion extraction unit, and a video evaluation unit. The video evaluation unit outputs M and T according to equations (16) to (20), and the video parameter update unit outputs the updated values ​​according to equation (21). and The process is then returned to the timing generation unit. When the maximum number of feedback updates is reached, the task status unit stops the corresponding loop and generates a failure status.

[0050] The screening and output module performs quality screening and near-duplicate processing according to equations (22) to (24), and stores the retained samples in association with condition labels, source identifiers, generation parameters, feedback history, scores, timestamps, and output status. The modules pass predefined data and parameters to each other, and it is not required to use a specific neural network, as long as the same input, output, and feedback relationship can be achieved.

[0051] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection. Equivalent substitutions or modifications made by those skilled in the art without departing from the concept of the present invention should fall within the scope of protection of the present invention.

Claims

1. A controllable video generation method for off-road driving, characterized in that, Includes the following steps: S1, obtain real off-road driving reference images and target control conditions; S2, obtain the reference condition confidence vector, extract the normalized depth map and binary edge map, and calculate the target condition variability U and scene structure complexity R; S3, construct image migration control information and video control information based on the target control conditions, and initialize migration intensity α and depth control intensity according to U and R. and edge control strength ; S4, calculate the target condition consistency score C and the structure preservation score S, and update α based on insufficient scores. and If both C and S reach the threshold, the initial frame of the target condition is determined; S5. Extract the drivable area mask and road vanishing point from the initial frame of the target condition to construct the forward motion direction template. Combine this frame with video control information to initialize the forward motion control intensity. and timing consistency control strength Generate candidate videos; S6, calculate the motion rationality score M and temporal stability score T of the candidate videos, and update the scores if necessary. and If the standard is met, the target off-road driving video will be determined; S7, filter and remove duplicates from at least one of the target condition initial frame and the target off-road driving video; S8 records qualified samples and constructs a long-tail off-road driving sample set.

2. The controllable video generation method for off-road driving according to claim 1, characterized in that, In step S1, the real off-road driving reference image is subjected to size and color normalization processing, and the target control conditions include at least one of terrain, weather, road surface condition and lighting condition; In step S2, the target condition change degree U is calculated based on the difference between the reference condition confidence vector and the target condition state vector, and the scene structure complexity R is calculated based on the grayscale distribution, binary edge map and normalized depth map of the reference image, where U and R ∈ [0, 1]. In step S3, α and α are initialized within their respective preset upper and lower limits. and ; In step S4, candidate initial frames are generated based on image migration control information, normalized depth map, binary edge map and the above intensity, and the target condition consistency score C and structure preservation score S are calculated. In step S7, the deduplication process includes comprehensive quality screening and near-duplicate detection within the same batch; In step S8, the target control condition label, reference image identifier, generation parameters, quality score, and timestamp of the qualified sample are recorded.

3. The controllable video generation method for off-road driving according to claim 2, characterized in that, The specified target condition categories form a non-empty set A of active conditions; for the k-th condition in A, let... Let X be the candidate label probability vector output by the conditional classifier for image X, and let X be the reference conditional confidence vector. Target conditional state vector Let U be the one-hot vector corresponding to the target label, and let U be the degree of change of the target condition. ;in, The image represents the actual off-road driving reference image, X represents the image to be classified, and k represents the active condition index. This represents the number of conditions contained in the active condition set A. Describing the L1 norm, Indicates the first The class condition classifier applies to the real off-road driving reference image. Output target label The probability of an unspecified condition; conditions not specified are not added to the active condition set A and are not included in the calculation.

4. The controllable video generation method for off-road driving according to claim 2, characterized in that, The real off-road driving reference image is divided into L grayscale intervals, where L≥2, and the pixel ratio of the l-th grayscale interval is... The image height H and width W are both not less than 2, and the number of edge pixels in the binary edge map is... Normalized depth map is Then the normalized gray entropy Edge density Normalized depth changes The scene structure complexity R is as follows: ， ， ， ； in, =0 is the rule ;in, and These are the pixel row index and pixel column index, respectively. To normalize the depth map at pixel positions Depth value at that location, It represents the natural logarithm.

5. The controllable video generation method for off-road driving according to claim 2, characterized in that, Let intermediate variables The initial values ​​for the migration intensity, depth control intensity, and edge control intensity are as follows: , ， ， Among them, each minimum value is less than the corresponding maximum value; unenabled deep control branches or edge control branches do not have corresponding control strengths set and do not participate in subsequent updates.

6. The controllable video generation method for off-road driving according to claim 5, characterized in that, For the initial frame of the candidate target generated for the nth time... The scores for consistency of objective conditions and preservation of structure are as follows: ， ， ; in, for Normalized cosine similarity within the same semantic feature space as the target condition positive description. For reference image and Depth-related scores, The intersection ratio of their edge regions. The intersection ratio of their drivable areas. The inverse score is the normalized distance between the vanishing points of the two regions; when the union of the two regions to be compared is empty, the intersection-union ratio is set to 1. Initial frame for candidate target conditions The corresponding minimum target label confidence, For the first A conditional classifier conditions the initial frame of candidate targets. Output target label The probability of; and, , , , Among them, threshold and All are greater than 0 and not greater than 1. This represents the larger value of z compared to 0; , , ; in, Each step size is a positive number; only when and Video generation begins at that time; and These are the scores for the depth structure and the edge structure, respectively. , , and These are respectively: insufficient consistency of target conditions, insufficient structure preservation, insufficient deep structure, and insufficient edge structure. and These are the target condition consistency threshold and the structure preservation threshold, respectively; , and These are the migration intensity update step size, depth control intensity update step size, and edge control intensity update step size, respectively.

7. The controllable video generation method for off-road driving according to claim 2, characterized in that, Let the vanishing point of the road in the initial frame of the target condition be... , >0, the template for the fixed forward motion direction at position p is: ， The candidate off-road driving videos include Frame and ≥3, The actual optical flow from frame t to frame t+1. For the corresponding road vanishing point, For the optical flow evaluation region within the drivable area, excluding the neighborhood of the vanishing point, the frame-by-frame forward motion direction template is: When any When M is empty, let M = 0. When neither is empty, calculate each component of motion using the following formula: ， ， ； in, For the first The region-normalized average optical flow amplitude corresponding to the frame. The normalized average optical flow amplitude over time is the average value of the region. To preset the minimum motion amplitude, Scoring exists for each sport; The image diagonal >0; Forward direction score From actual optical flow and The consistency of the nonnegative weighted cosines between them is determined; ； in, and These are the global translation magnitude and rotation angle magnitude, normalized to [0,1], respectively. For speed variation error, To score points for consistent speed; ， ； ； The optical flow compensation residual, the change in adjacent optical flow, and the difference in the color histogram of adjacent frames were normalized to: ; ， ， ； ； ； in, For camera motion variation error, To ensure the camera scores a stable score, and These are insufficient exercise and insufficient timing, respectively. and These are the motion rationality threshold and the temporal stability threshold, respectively. and These are the forward motion control intensity update step size and the temporal consistency control intensity update step size, respectively; threshold. and All are greater than 0 and not greater than 1. >0, >0; , This represents the larger value of z compared to 0; ； Only when M and T respectively reach and Output the target off-road driving video in real time; in, The normalized average optical flow amplitude for the region. The normalized average optical flow amplitude over time is the average value of the region. To preset the minimum motion amplitude, Scoring exists for each sport; and These are the normalized global translation amplitude and the normalized global rotation angle amplitude, respectively.

8. The controllable video generation method for off-road driving according to claim 2, characterized in that, In S1, the real off-road driving reference image is cropped according to a preset aspect ratio, and the drivable road surface area, road boundary, road center direction and distant vanishing point are preserved in the cropping process; The image migration control information includes positive and negative prompts. The positive prompts include reference scene, target terrain, target weather, target road surface condition, target lighting condition, forward off-road driving perspective, and realistic description. The negative prompts include descriptions for suppressing incorrect weather, incorrect terrain, urban road features, vehicle interior components, non-target traffic participants, blur, overexposure, color shift, duplicate textures, road structure distortion, and generated artifacts. The video control information also includes scene structure constraint information for maintaining road position, road extension direction and vanishing point direction, as well as motion timing constraint information for maintaining forward motion and inter-frame consistency and suppressing stillness, backward movement, shaking, speed change, viewpoint jump, color change and background flicker.

9. The controllable video generation method for off-road driving according to claim 2, characterized in that, The target condition consistency score (C), structure preservation score (S), sharpness score (F), exposure and color score (E), and sample variability score (D) are all normalized to [0,1]. For the target off-road driving video, C and S are taken from the initial frame score at the visual anchor point, F and E are taken from the arithmetic mean of the corresponding scores for each frame, and D is calculated based on the features of the entire video segment. The initial frame comprehensive quality score... Overall video quality score Both the overall threshold and the individual threshold are located in [0,1]. When the overall quality score reaches the overall threshold and all applicable individual scores except D reach the individual threshold, the generated sample passes the quality screening. For samples that pass the test within the same batch and belong to the same sample type and have the same target control condition labels, they are arranged in descending order of comprehensive quality score, ascending order of timestamp, and ascending order of sample identifier; in the... In the round-robin repetition process, let the set of samples to be processed be... Select The sample ranked first in the middle is used as the representative sample. and form near-repeating sets. : ; Retain representative samples And remove near-duplicate sets from the current samples to be processed. Repeat the above process until there are no more samples to be processed. in, This indicates the sample currently awaiting processing. Indicates the current sample to be processed. Representative sample Normalized sample similarity between them This indicates the preset near-repetition threshold.

10. A controllable video generation device for off-road driving, characterized in that, The controllable video generation device for off-road driving includes: The image and condition processing module is used to acquire and preprocess real off-road driving reference images and target control conditions, form a reference condition confidence vector, and extract normalized depth map, binary edge map, drivable area mask and road vanishing point; The adaptive parameter initialization module is used to calculate the target condition variability U and the scene structure complexity R, and initialize the migration strength α and depth control strength within preset upper and lower limits. and edge control strength ; The first-frame closed-loop generation module is used to generate the initial frame of candidate target conditions, calculate the target condition consistency score C, structure preservation score S, depth structure component score, and edge structure component score, and update α according to the corresponding deficiencies. and ; The video closed-loop generation module is used to generate candidate off-road driving videos using the initial frame that meets the target condition threshold as the visual anchor point, calculate the motion rationality score M and the temporal stability score T, and update the forward motion control strength based on the insufficiency of the two scores. and timing consistency control strength ; The filtering and output module is used to output the target off-road driving video through video feedback, perform quality filtering, near duplication detection and metadata recording on the generated samples, or output the corresponding generation failure flag; Furthermore, the controllable video generation device for off-road driving includes at least one processor, a memory, and an image input / output interface; when the computer program in the memory is executed by the processor, it controls each module to perform the method described in any one of claims 1 to 8, and records the control intensity, evaluation score, number of times, output status, and sample identifier of each feedback.

Images