Video generation method, electronic equipment and storage medium

Abstract

The invention provides a video generation method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a reference video of a vehicle, extracting a first reference image and a second reference image from the reference video, and enabling the reference video to be obtained through photographing a foreground object around the vehicle; generating first input data based on a first instance image corresponding to the foreground object in the first reference image and a second instance image corresponding to the foreground object in the second reference image; the first input data indicates size information and position information of the foreground object; generating a first feature by using an encoder in a prediction model based on the first input data; based on the first feature, performing weighting operation on the first reference image and the second reference image by using an attention mechanism to obtain a fusion feature of the reference video; and based on the fused features, generating a predicted video of the vehicle driving process by using a decoder in the prediction model. The accuracy of vehicle video generation can be improved.

Description

Technical Field

[0001] This application relates to the field of automatic parking technology, and more particularly to a video generation method, electronic device, and storage medium. Background Technology

[0002] With the increasing number of motor vehicles, the application environment for autonomous driving technology is becoming increasingly complex. To improve the performance of autonomous driving systems, pre-generating high-fidelity, precisely controllable driving scenario videos is crucial for testing and training these systems. To enhance the accuracy of autonomous driving decision-making logic, training videos are typically generated based on pre-acquired image or video information collected during vehicle operation. However, this method of generating videos during vehicle operation cannot adjust the appearance, position, and dynamic characteristics of foreground objects around the vehicle, resulting in blurred textures or motion distortion in the generated videos, leading to low accuracy in the generated training videos. Summary of the Invention

[0003] In view of the above, it is necessary to propose a video generation method, electronic device and storage medium to solve the technical problem of low accuracy in generating videos of vehicle driving process.

[0004] This application provides a video generation method, the method comprising: acquiring a reference video of a vehicle; extracting a first reference image and a second reference image from the reference video, the reference video being used to capture foreground objects surrounding the vehicle; pasting the first instance image into the second reference image based on the size data of a first instance image corresponding to the foreground object in the first reference image and a second instance image corresponding to the foreground object in the second reference image to generate first input data; the first input data indicating the size information of the foreground object and the position information of the foreground object in the area where the vehicle is located; generating a first feature based on the first input data using an encoder in a preset prediction model; performing a weighted operation on the first reference image and the second reference image using an attention mechanism based on the first feature to obtain a fusion feature of the reference video; and generating a predicted video of the vehicle's driving process using a decoder in the prediction model based on the fusion feature.

[0005] In some embodiments, the step of pasting the first instance image into the second reference image to generate first input data based on the size data of the first instance image corresponding to the foreground object in the first reference image and the second instance image corresponding to the foreground object in the second reference image includes: adjusting the size of the first instance image based on the size data of the second instance image to obtain a copied image; replacing the second instance image in the second reference image based on the copied image to obtain an input image; and determining the first input data based on the input image.

[0006] In some embodiments, generating a first feature based on the first input data using an encoder in a preset prediction model includes: determining a first input feature based on the first input data using the encoder; generating control parameters based on pre-stored control information using the encoder; wherein the control information indicates the need to adjust the foreground object in the first input data; and determining the first feature based on the control parameters and the first input feature.

[0007] In some embodiments, the step of weighting the first reference image and the second reference image based on the first feature to obtain the fusion feature of the reference video includes: encoding the first reference image and the second reference image to obtain encoded features; determining the mask features of the foreground object based on the first instance image and the second instance image; and adjusting the weights of information in different dimensions of the encoded features according to the first feature and the mask features to obtain the fusion feature of the reference video.

[0008] In some embodiments, adjusting the weights of information in different dimensions of the encoded features based on the first feature and the mask feature to obtain the fusion features of the reference video includes: determining the similarity between the first feature and the encoded features; and determining the fusion features of the reference video based on the product of the similarity and the mask feature.

[0009] In some embodiments, the method further includes training the prediction model, comprising: acquiring a first video of the vehicle; extracting a first sample image and a second sample image from the first video; generating first training data based on a first instance image corresponding to a specified object in the first sample image and a second instance image corresponding to the specified object in the second sample image; generating a first training feature based on the first training data using a preset initial model; performing a weighted operation on the first sample image and the second sample image based on the first training feature to obtain a sample fusion feature of the training video; predicting a second video of the vehicle's driving process using the initial model based on the sample fusion feature; determining a reward value of the initial model based on the semantic similarity between the first video and the second video; determining a loss value of the initial model based on the reward value; updating the initial model based on the loss value until the loss value meets a preset convergence condition, stopping the updating of the initial model and obtaining a prediction model trained to a convergent state.

[0010] In some embodiments, determining the reward value of the initial model based on the semantic similarity between the first video and the second video includes: determining a second video frame corresponding to the first video frame from the second video based on the timestamp of the first video frame in the first video; the timestamp corresponding to the second video frame is the same as the timestamp corresponding to the first video frame; determining the semantic similarity between the first video and the second video based on the similarity between the first video frame and the second video frame; and determining the semantic similarity as the reward value of the initial model.

[0011] In some embodiments, determining the loss value of the initial model based on the reward value includes: determining a first loss value of the initial model based on the product of the reward value and a preset learning rate; the learning rate indicating the convergence speed of the initial model; determining a second loss value of the initial model based on the mean squared error of the first video and the second video; and determining the loss value of the initial model based on the first loss value and the second loss value.

[0012] This application also provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing the instructions stored in the memory to implement the video generation method.

[0013] This application also provides a vehicle, which includes the electronic device.

[0014] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the video generation method described above.

[0015] As can be seen from the above technical solutions, the embodiments of this application convert the features of the foreground object into quantized data through the first input data generated by the first instance image and the second instance image. This avoids the prediction model parsing the semantics of a large number of pixels, eliminates the need to distinguish between the foreground object and background noise, reduces the load on the prediction model, and minimizes prediction bias caused by semantic ambiguity. The position of the foreground object is marked in multiple consecutive second reference images based on the first instance image and the second instance image, eliminating the need for complex temporal or pose annotations, thus improving the quality of the foreground object in the generated prediction video. The first reference image and the second reference image are weighted using an attention mechanism based on the first feature, ensuring that the fused features preferentially retain key information related to the motion of the foreground object. The attention mechanism also identifies regions in the second reference image that are more important for predicting the motion of the foreground object, filtering redundant information in the background and weakening the noise impact of blurred edges between the foreground object and background information, ensuring the stability of the fused features. Furthermore, it ensures that the decoder in the prediction model accurately learns the motion trend of the foreground object, avoids error accumulation caused by multi-module splicing, ensures a direct mapping from foreground information to the prediction video, and improves the accuracy of motion trajectory and scene evolution. Attached Figure Description

[0016] Figure 1 This is an application scenario diagram of a video generation method provided in an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of a framework for generating predictive videos provided in an embodiment of this application.

[0018] Figure 3 This is a flowchart of a video generation method provided in an embodiment of this application.

[0019] Figure 4 This is a flowchart of a method for generating first input data provided in an embodiment of this application.

[0020] Figure 5 This is a flowchart of a method for determining a first feature provided in an embodiment of this application.

[0021] Figure 6 This is a flowchart of a method for determining fusion features provided in an embodiment of this application.

[0022] Figure 7 This is a flowchart of a method for determining fusion features provided in an embodiment of this application.

[0023] Figure 8 This is a flowchart of a method for training a prediction model provided in an embodiment of this application.

[0024] Figure 9This is a flowchart of a method for determining a reward value provided in an embodiment of this application.

[0025] Figure 10 This is a flowchart of a method for determining a loss value provided in an embodiment of this application.

[0026] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.

[0028] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0030] This application provides a video generation method that can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0031] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0032] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0033] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0034] like Figure 1 The diagram illustrates an application scenario of a video generation method provided in an embodiment of this application. This video generation method can be applied to an electronic device 100. The electronic device 100 is disposed in a vehicle 200, and the electronic device 100 is communicatively connected to a vehicle component 300 within the vehicle 200.

[0035] In this application, electronic device 100 acquires a reference video of vehicle 200 and extracts a first reference image and a second reference image from the reference video. The reference video is obtained by vehicle component 300 capturing foreground objects surrounding vehicle 200. For example, vehicle component 300 may be an onboard camera of vehicle 200. Electronic device 100 generates first input data based on a first instance image corresponding to the foreground object in the first reference image and a second instance image corresponding to the foreground object in the second reference image. The first input data indicates the size information of the foreground object and the position information of the foreground object in the area where the vehicle is located.

[0036] In this application, the electronic device 100 generates a first feature based on the first input data using an encoder in a preset prediction model. Based on the first feature, an attention mechanism is used to weight the first reference image and the second reference image to obtain fused features of the reference video. The electronic device 100 also generates a predicted video of the vehicle's driving process based on the fused features using a decoder in the prediction model.

[0037] like Figure 2The diagram illustrates the process of using the prediction model 110 and attention module 120 in the electronic device 100 to process the first reference image 131 and the second reference image 132 in the reference video 130 corresponding to the vehicle 200, generating a framework for the prediction video 140. First, the first instance image 1311 in the first reference image 131 is pasted to a preset position in the second reference image 132, resulting in first input data 150. The first input data 150 indicates the size information of the foreground object and its position within the area occupied by the vehicle 200. The preset position can be the position of the second instance image 1321 corresponding to the foreground object within the second reference image 132.

[0038] The first reference image 131 and the second reference image 132 are encoded based on the preset CLIP encoder 160 to obtain the encoded feature 133; the first input data 150 and the pre-stored control information 151 are encoded using the encoder 111 in the prediction model 110 to obtain the first feature 152 output by the encoder 111.

[0039] Based on the first feature 152, the first instance image 1311, and the second instance image 1321, the mask feature 153 of the foreground object is determined.

[0040] The attention module 120 is used to adjust the attention weights of the encoded feature 133 and the mask feature 153 to obtain the fused feature 170 of the reference video.

[0041] The decoder 112 of the prediction model 110 is used to decode the fused features 170 to generate a predicted video 140 of the vehicle 200 during its driving process.

[0042] like Figure 3 The diagram shown is a flowchart of a video generation method according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The video generation method provided in this embodiment includes the following steps.

[0043] S20, acquire a reference video of the vehicle, and extract a first reference image and a second reference image from the reference video. The reference video is used to capture foreground objects around the vehicle. The foreground objects include at least pedestrians and / or other vehicles around the vehicle.

[0044] In one embodiment of this application, the vehicle's autonomous driving strategy may exhibit decision-making lag in dynamic environments. When the real-time performance of the autonomous driving strategy is low, it only responds instantly based on the vehicle's real-time perception data. In scenarios such as highways and complex intersections, insufficient decision-making time can easily lead to driving safety risks. To adjust the vehicle's autonomous driving strategy, reference videos of the vehicle can be acquired in advance, and a first reference image and a second reference image can be extracted from the reference videos.

[0045] The reference video is used to capture foreground objects around the vehicle. The information in the reference video indicates the visual characteristics of the foreground objects around the vehicle, as well as the relative positions of the foreground objects and the vehicle.

[0046] The first reference image can be the initial video frame in the reference video, where the initial video frame is the video frame with the earliest timestamp in the reference video. The second reference image can be all video frames in the reference video except for the initial video frame.

[0047] The image information corresponding to the foreground object in the first reference image can be used to indicate the visual information of the foreground object when it first appears in the reference video. For example, the image information corresponding to the foreground object in the first reference image can indicate the texture features, color features, size features, and initial relative position information of the foreground object and the vehicle when the foreground object appears in the reference video.

[0048] Based on the visual information of the foreground object indicated by the first reference image when it first appears in the reference video, and the continuously changing state of the foreground object indicated by the second reference image in the reference video, a predicted video of the vehicle at any time in the future can be generated.

[0049] Predictive videos provide global information, encompassing both temporal and spatial dimensions, for autonomous driving decision-making and planning. Essentially, a predictive video is a sequence of dynamic evolutions of the vehicle's surrounding environment over a future period. Specifically, it can include the trajectories of dynamic targets around the vehicle, such as whether a vehicle in front is changing lanes, whether a pedestrian is crossing the road, or whether a cyclist is traveling against traffic. It can also include changes in the surrounding environment, such as traffic light changes, sudden obstacles at intersections, and changes in road surface friction coefficients. Furthermore, it can include the interaction relationships between multiple targets around the vehicle, such as the avoidance intentions of two vehicles meeting, and the priority of pedestrians and vehicles. The information in the predictive video enables the vehicle's decision-making and planning module to perform spatiotemporal global optimization. For example, if the predictive video indicates that a red light will illuminate at the intersection ahead in 5 seconds, and a pedestrian is crossing laterally, the vehicle speed can be adjusted in advance, for example, smoothly decelerating from 60 km / h to 30 km / h, rather than abruptly braking at the intersection. This ensures both safety and improved comfort, avoiding frequent acceleration / deceleration or untimely risk avoidance caused by locally optimal decisions in autonomous driving strategies.

[0050] Predictive videos can support closed-loop training of end-to-end autonomous driving strategies. Specifically, predictive videos can serve as data augmentation tools to generate diverse future scenarios. For example, predictive videos of the same intersection with different traffic flows and different weather conditions can address the problems of scarce real-world data and difficulty in collecting data for extreme scenarios. Predictive videos can also serve as carriers of supervisory signals, using the optimal actions of human drivers in reference videos as labels. This allows end-to-end predictive models to learn how to adjust decisions based on future scenarios, rather than just learning immediate reactions in the current scenario, thereby significantly improving the model's generalization ability.

[0051] Predictive videos can also identify potential risks in the vehicle's environment in advance through scene prediction, thus allowing sufficient time for avoidance. In real traffic scenarios, many risks are hidden. For example, pedestrians obscured by large vehicles, or a stationary vehicle suddenly starting at an intersection. Traditional autonomous driving strategies only identify risks after they are exposed, resulting in insufficient avoidance time. Predictive videos can predict changes in the vehicle's surrounding environment in advance based on the scene context of a reference video, such as the vehicle's trajectory and intersection traffic patterns, ensuring that the vehicle avoids risks in advance and thus reducing the probability of accidents.

[0052] Predictive videos are typically generated by combining multi-sensor reference data (cameras, LiDAR, millimeter-wave radar), and their prediction results can be cross-validated with real-time perception data: if the real-time perception module misses a target (such as an obstacle ahead) due to severe weather (such as heavy rain or fog), but the predicted video can still predict its existence based on historical reference data (such as the position and movement trend of the obstacle before the rain), a safety redundancy strategy (such as slowing down or turning on emergency lighting) can be triggered; if the predicted video differs significantly from the real-time perception data (such as predicting the presence of a pedestrian but not detecting it in real time), the system can enter a "cautious mode" (reducing vehicle speed and increasing perception frequency) to avoid risks caused by misjudgment by a single module.

[0053] The predictive video not only includes environmental information but can also be combined with the vehicle's own status (such as battery charge and braking system health) to predict the "vehicle's operating status in the future time period." For example, based on the vehicle's energy consumption data in the reference video, if it is predicted that "the battery charge will be insufficient in 10 minutes and will not be able to support high-speed driving," the system can plan the route to the nearest charging station in advance to avoid breaking down halfway. If it is predicted that "the braking system may experience a minor malfunction in the next 3 seconds," the vehicle speed can be reduced in advance to reduce braking demand and reserve buffer space for fault handling.

[0054] In one embodiment of this application, since the engineering implementation of autonomous driving technology faces problems such as long development cycle, high testing cost and great deployment difficulty, the deployment efficiency of engineering autonomous driving technology can be improved by using a mechanism of virtual scene generation and real data reuse based on predictive video.

[0055] S21, based on the size and position of the first instance image corresponding to the foreground object in the first reference image and the second instance image corresponding to the foreground object in the second reference image, the first instance image is pasted into the second reference image to generate first input data; the first input data indicates the size information of the foreground object and the position information of the foreground object in the area where the vehicle is located.

[0056] In one embodiment of this application, in order to improve the quality of visual information corresponding to foreground objects in a predicted video, first input data can be generated based on a first instance image corresponding to the foreground object in a first reference image and a second instance image corresponding to the foreground object in a second reference image, and a predicted video can be generated based on the first input data.

[0057] Specifically, the bounding box of the foreground object in the second reference image can be determined based on the second instance image, thereby determining the position of the foreground object at any future time. The size of the first instance image in the first reference image is adjusted based on the size of the bounding box corresponding to the second instance image, and the first instance image is pasted into the bounding box corresponding to the second instance image in the second reference image to obtain the input image. Then, first input data is generated based on the input image. For example, the first input data can be a video composed of multiple images with dimensions (B×3×T×H×W). Here, B represents the batch of images; T represents the timestamp corresponding to the image; and H×W represents the image size, used to indicate the video resolution. This ensures that the motion continuity of the foreground object remains consistent throughout the temporal evolution process, avoiding prediction logic confusion caused by information overload.

[0058] Specifically, the first instance image of the foreground object in the initial video frame of the reference video is directly reused. By pasting the first instance image to the corresponding position in subsequent frames, the visual features and morphological benchmarks of the foreground object are locked. This ensures that the prediction model does not need to relearn the generation and motion rules of the foreground object, but only needs to focus on the mechanism of the background's temporal evolution as the foreground position changes, such as the background viewpoint shift and illumination gradation when the vehicle moves. When training autonomous driving functions based on the prediction video, it can ensure the consistency of the foreground object's identity and shape. For example, it can avoid predicting the target vehicle as another object or exhibiting shape distortion. This simplifies the complexity of temporal modeling in the prediction model and improves the efficiency and stability of generating prediction videos.

[0059] Specifically, due to feature differences and noise interference between multiple video frames, foreground objects may experience positional shifts or morphological distortions. Therefore, reusing the first instance image of the foreground object in the initial video frame of the reference video allows for coordinate matching and pasting into subsequent frames, thus providing a fixed identity anchor point for the foreground object. The prediction model can fine-tune the fusion details of the foreground and background, such as adjusting the edge lighting and shadow adaptation of the foreground object, without reconstructing the image information of the foreground object itself, thereby improving the consistency and reliability of foreground objects in the predicted video.

[0060] Because autonomous driving scenarios require millisecond-level response times for predicted video generation, traditional temporal generation models suffer from high computational complexity due to the need to handle numerous inter-frame dependencies, making it difficult to meet real-time requirements. This application achieves synchronization of foreground information in both temporal and spatial dimensions based on object detection, coordinate mapping, and pixel duplication, reducing the computational complexity of the model. This ensures that the prediction model does not need to generate the foreground, but only needs to encode and decode the background for reconstruction, thus adapting to the real-time prediction needs of autonomous driving.

[0061] Furthermore, reference videos of real traffic scenes are easily affected by interference. Traditional models are prone to generating abrupt background changes and logical inconsistencies under noisy input. This application provides a stable spatial reference for the prediction model by fixing the foreground object, ensuring that the prediction model can reasonably predict the temporal changes of the background based on the relative positional relationship between the foreground and the background, avoiding logical conflicts caused by the independent evolution of the background. At the same time, reusing the clear foreground image of the first frame can resist slight noise interference in subsequent frames, improving the model's robustness to input perturbations.

[0062] In one embodiment of this application, the specific method for generating the first input data based on the first instance image corresponding to the foreground object in the first reference image and the second instance image corresponding to the foreground object in the second reference image is described in the following text. Figure 4 The corresponding detailed explanation.

[0063] S22, based on the first input data, a first feature is generated using a preset prediction model; the first feature indicates the conditional latent variable output by the prediction model, which is used to indicate the position information of the foreground object.

[0064] In one embodiment of this application, the first input data includes black areas without filled natural pixels, meaning the image information corresponding to the non-foreground object portion of the first input data is null, thus avoiding interference from background information in the process of generating the predicted video. Furthermore, each video frame is different due to spatiotemporal copying and pasting, possessing more complex visual information compared to a single or repeated video frame. This ensures that the position of the foreground object is strictly aligned with the predicted video, enhancing the accuracy of the foreground object's positional information and appearance features.

[0065] In one embodiment of this application, the prediction model can be an autoencoder model. Generating a prediction video based on an autoencoder model enables denoising of visual information in a reference video. This is achieved by encoding and compressing the first input data to extract generalized features from the reference video, and then decoding and reconstructing these generalized features using a decoder in the prediction model, thereby filtering noise from the first input data. The noise in the first input data can include blurring, jitter, pixel interference, and other information from the reference video, while also preserving the structural and temporal consistency between the foreground object and the background information.

[0066] The encoder in the prediction model can be used to compress the dimensionality of the first input data and extract noise-free features from it. Specifically, the first input data contains two types of information: useful signals (the shape of the foreground object, the structure of the background, and the temporal correlation between frames) and noise (such as pixel shifts caused by camera shake, random pixels caused by sudden changes in illumination, and image compression distortion).

[0067] When extracting features from the first input data using an encoder, dimensionality reduction and feature refinement can be achieved. Specifically, the spatial dimension of the first input data is progressively compressed through convolutional and pooling layers in the encoder. During compression, noise, due to its lack of spatial consistency and structural correlation, is weakened or suppressed. For example, a first input data with a dimension of 256×256 can be compressed into a 128-dimensional latent vector. Only core features with strong structural correlation are retained, such as the contours of foreground objects and the relative positional relationships of foreground objects between different video frames. Because core features have strong noise resistance, noise variations will not affect the expression of core features. For example, random pixel shifts caused by jitter in the reference video will not be reflected in the contour features of foreground objects and the structural features of background information after dimensionality reduction by the encoder. In this way, the data compression mechanism of the encoder can achieve preliminary noise filtering, resulting in first features with less noise information.

[0068] For a specific method of determining the first feature in one embodiment of this application, please refer to [link to relevant documentation]. Figure 5 The corresponding detailed explanation.

[0069] S23, based on the first feature, the encoded features corresponding to the first instance image and the second instance image are weighted using an attention mechanism to obtain the fusion feature of the reference video; the fusion feature indicates the texture and color features of the foreground object.

[0070] In one embodiment of this application, feature extraction and encoding operations can be performed on the first reference image and the second reference image to obtain the encoded features corresponding to the reference video. Specifically, the first instance image in the first reference image and the second instance image in the second reference image can be input into a preset multimodal and trained encoder (Contrastive Language-Image Pre-training, CLIP) to obtain encoded features that can represent the deep information in the reference video.

[0071] In the process of generating predicted videos using a prediction model, a pre-defined CLIP encoder can be used to determine the semantics of the reference video. Specifically, the first input features corresponding to the first input data can be used as the semantics of the reference video, indicating the visual information contained in the reference video. The encoded vector corresponding to the reference video is used as a key and value, and cross-attention calculation is performed with the first input features. This allows the process of generating the predicted video to prioritize regions related to the first input features, thereby ensuring that the visual information of the foreground objects in the generated predicted video has high quality. By maximizing the similarity of the semantic information between the generated predicted video and the reference video, the noise level in the predicted video generation process is gradually adjusted, ultimately generating a predicted video that conforms to the text semantics. In this way, the CLIP encoder can guide the generation process of the predicted video based on the visual information of the foreground objects. Based on cross-modal semantic alignment and contrastive learning, it maps the global visual information in the reference video and the visual information corresponding to the foreground objects to the same feature space, ensuring that the semantic information of the two can be directly compared and associated. This makes the semantic information of the reference video and the visual information corresponding to the foreground objects close in this feature space, while the features corresponding to irrelevant visual information are far away from the visual information corresponding to the foreground objects.

[0072] Specifically, the CLIP encoder comprises a dual-encoder architecture. The CLIP encoder includes an image encoder and a semantic encoder. The semantic encoder, which can be a transfer learning-based encoder, is used to encode the semantics of multimodal data into encoded vectors. For example, when the multimodal data is text, the semantic encoder can encode and segment the text, generating corresponding encoded vectors. The image encoder, which can be a residual network or a transfer learning network—a neural network with data encoding and feature extraction capabilities—is used to convert the input image into a fixed-dimensional feature vector.

[0073] In one embodiment of this application, a fusion feature of the reference video can be obtained by weighting the first reference image and the second reference image based on a first feature using an attention mechanism. Specifically, the first input feature corresponding to the first input data can be used as the query instruction of the attention mechanism, and the encoded feature can be used as the key and value. The first input feature and the encoded feature are fused through attention calculation to obtain the fusion feature of the reference video. The attention mechanism satisfies the following formula: ; in, The first feature representing the encoder output, and The encoding features representing the reference video, The matrix dimension representing the encoded features, A spatial mask representing the foreground object is used to limit the scope of the attention mechanism. Aligning the semantic features of foreground information and the reference video based on the attention mechanism ensures that the visual attributes of the foreground object in the generated predicted video remain consistent with those in the reference video. For example, the visual attributes of the foreground object can be visual features such as texture or color features, and this application does not limit this.

[0074] For a specific method of determining the fusion features in one embodiment of this application, please refer to [link to relevant documentation]. Figure 6 and Figure 7 The corresponding detailed explanation.

[0075] S24, Based on the fusion features, generate a predicted video of the vehicle's driving process using the prediction model.

[0076] In one embodiment of this application, fused features can be input into the decoder of the prediction model, and based on the fused features, the decoder in the prediction model can be used to generate a predicted video of the vehicle driving process.

[0077] The decoder of the prediction model is used to reconstruct multiple video frames with low noise levels and high accuracy of foreground information based on fused features, and the predicted video is obtained by combining multiple video frames.

[0078] Based on the denoised core features of the encoder output received by the decoder, the spatial dimension of the video frame is gradually restored through transposed convolution (such as Transposed Conv) and upsampling, and finally the output frame (predicted video frame) with the same dimension as the input is reconstructed.

[0079] The denoising mechanism in the decoder stage includes: reconstruction based solely on fusion features, without relying on noise information in the first input data. Therefore, the video frames reconstructed by the decoder naturally eliminate the original noise information from the reference video. The decoder can enhance the consistency of spatial details between different video frames during video frame reconstruction, such as smoothing the edges of foreground objects and ensuring the continuity of background textures, further suppressing noise-induced local pixel abrupt changes. For example, isolated, high-brightness pixels in the reference video that appear due to lighting noise will be restored to pixels with the same brightness level as surrounding pixels during decoding and reconstruction due to the lack of fusion feature support, preventing noise from propagating in the predicted video.

[0080] In one embodiment of this application, to improve the accuracy of the prediction model in generating prediction videos based on reference videos, the prediction model can also be trained based on a pre-acquired first video. The first video refers to visual information about the vehicle's surrounding environment collected during the vehicle's movement.

[0081] For example, the prediction model can be trained on a publicly available dataset. This dataset could be the nuScenes dataset. This dataset includes 700 training videos and 150 validation videos, each video consisting of 20 seconds of footage captured by six surround-view cameras. The training videos can be input into the prediction model and iterated 200,000 times, with one sample per batch, processing eight consecutive frames (256×448 resolution) from six cameras, based on a learning rate of 3×10⁻⁶. -5 The optimizer modulates the gradient of the loss value of the prediction model.

[0082] Specifically, the performance of a prediction model can be evaluated based on multiple dimensions. These dimensions can include generation quality; for example, the performance of a prediction model can be evaluated based on the quality of a single video frame generated by the prediction model or the temporal consistency of multiple video frames. Multiple dimensions can also include the controllability of the video frames generated by the prediction model; specifically, the target detection performance of a prediction model can be evaluated based on the average accuracy of the video frames generated by the prediction model.

[0083] In one embodiment of this application, a similarity reward feedback of foreground information can be introduced during the training phase of the prediction model. The visual consistency between foreground objects in the predicted video and reference objects in the real video can be improved through the feedback during the training phase.

[0084] Specifically, image information of foreground objects can be extracted from the prediction video generated by the prediction model and the first video. Only foreground objects present in the initial frame of the first video are retained, and these are input into the CLIP encoder to obtain the feature vectors corresponding to the prediction video. This characterizes the i-th foreground object in the t-th frame of the predicted video and obtains the feature vector corresponding to the first video. , representing the foreground object corresponding to the first video.

[0085] For example, the cosine similarity between the feature vector corresponding to the predicted video and the feature vector corresponding to the first video can be calculated as semantic similarity to obtain the reward value of the prediction model. A higher reward value indicates a higher similarity between the predicted video output by the prediction model and the first video. The reward value is determined according to the following relationship: ; in This represents the feature vector of the i-th pixel in the t-th frame of the predicted video. Let i represent the feature vector of the i-th pixel in the t-th frame of the first video.

[0086] In one embodiment of this application, a loss function can be constructed based on the reward value, and the prediction model can be updated based on the loss function. The loss value of the loss function is used to maximize the similarity between the foreground information in the predicted video and the foreground information in the real video. Specifically, the loss function satisfies the following relationship: ; The loss function may also include: .

[0087] For details on how to train the prediction model, please refer to [link / reference]. Figure 8 , Figure 9 and Figure 10 The corresponding detailed explanation.

[0088] In one embodiment of this application, the metrics for evaluating the prediction model may include generation quality, controllability, and data augmentation effect.

[0089] The generation quality can include single frame quality (Frechet Inception Distance, FID) and temporal consistency (Frechet Video Distance, FVD). Controllability can include nuScenes detection score (NDS), average accuracy (mAP), and can also be evaluated using a video-aware state-of-the-art model for BEV 3D object detection performance in generated videos. In determining the data augmentation effect, multi-view images matching the size of the nuScenes training set can be synthesized to verify their improvement effect on StreamPETR training.

[0090] Table 1 shows the quality and retrograde comparison data of the predicted video and the control group based on the nuScenes validation set. A smaller FID value indicates higher single-frame quality of the video generated by the prediction model; a smaller FVD value indicates higher temporal consistency of the video generated by the prediction model.

[0091] Table 1 As shown in Table 1, InstanceDrive indicates the method in this application that guides the video generation process based on instance images. This method has the lowest FID value compared to the other methods in the control group, and its FVD value is also the lowest compared to the other methods. Therefore, the method in this application that guides the video generation process based on instance images can improve the quality of the generated video.

[0092] In one embodiment of this application, the quality of the predicted video output by the prediction model can also be evaluated based on the instance location and appearance in the real data. The closer the instance location and appearance are to the real data, the higher the quality of the predicted video.

[0093] Table 2 shows a comparison between NDS scores based on real data and NDS scores from predicted videos. The NDS score is a comprehensive metric used in the nuScenes dataset to evaluate 3D object detection. It indicates the localization / attribute error and characterizes the detection accuracy and robustness of the prediction model in multimodal perception for autonomous driving. The NDS score ranges from 0 to 100%, with higher values ​​being better.

[0094] Table 2 As shown in Table 2, the NDS score of the real data is 46.9, the NDS score of the video generated using the Panacea method is 32.1, and the NDS score of the predicted video obtained using the instance image-guided video generation process method provided in this application is 38.1. Compared to the Panacea method, the difference between the predicted video obtained using the instance image-guided video generation process method provided in this application and the real data is smaller, indicating that the predicted video obtained using the instance image-guided video generation process method in this application is more accurate in determining the position and appearance of the foreground object.

[0095] As can be seen from the above technical solutions, the embodiments of this application convert the features of the foreground object into quantized data through the first input data generated by the first instance image and the second instance image. This avoids the prediction model parsing the semantics of a large number of pixels, eliminates the need to distinguish between the foreground object and background noise, reduces the load on the prediction model, and minimizes prediction bias caused by semantic ambiguity. The position of the foreground object is marked in multiple consecutive second reference images based on the first instance image and the second instance image, eliminating the need for complex temporal or pose annotations, thus improving the quality of the foreground object in the generated prediction video. The first reference image and the second reference image are weighted using an attention mechanism based on the first feature, ensuring that the fused features preferentially retain key information related to the motion of the foreground object. The attention mechanism also identifies regions in the second reference image that are more important for predicting the motion of the foreground object, filtering redundant information in the background and weakening the noise impact of blurred edges between the foreground object and background information, ensuring the stability of the fused features. Furthermore, it ensures that the decoder in the prediction model accurately learns the motion trend of the foreground object, avoids error accumulation caused by multi-module splicing, ensures a direct mapping from foreground information to the prediction video, and improves the accuracy of motion trajectory and scene evolution.

[0096] like Figure 4The diagram shown is a flowchart of a method for generating first input data according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for generating first input data provided in this embodiment includes the following steps.

[0097] S30, adjust the size of the first instance image based on the size data of the second instance image to obtain a copied image.

[0098] In one embodiment of this application, the bounding box of a foreground object in a second reference image can be determined based on the second instance image, thereby determining the position of the foreground object at any future time. The size of the first instance image in the first reference image is adjusted based on the size of the bounding box corresponding to the second instance image to obtain a copied image.

[0099] S31, based on the copied image, the second instance image in the second reference image is replaced to obtain the input image.

[0100] In one embodiment of this application, the first instance image can be pasted into the bounding box corresponding to the second instance image in the second reference image to obtain the input image.

[0101] S32, Based on the input image, determine the first input data.

[0102] In one embodiment of this application, first input data can be generated based on an input image. For example, the first input data can be a video composed of multiple images with dimensions (B×3×T×H×W). Here, B represents the batch of images; T represents the timestamp corresponding to the image; and H×W represents the image dimensions, used to indicate the video resolution. This ensures that the motion continuity of the foreground object remains consistent throughout the temporal evolution, avoiding prediction logic confusion caused by information overload.

[0103] like Figure 5 The diagram shown is a flowchart of a method for determining a first feature according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining a first feature provided in this embodiment includes the following steps.

[0104] S40, based on the first input data, the encoder is used to determine the first input feature.

[0105] In one embodiment of this application, the first input data includes black areas without filled natural pixels, meaning the image information corresponding to the non-foreground object portion of the first input data is null, thus avoiding interference from background information in the process of generating the predicted video. Furthermore, each video frame is different due to spatiotemporal copying and pasting, possessing more complex visual information compared to a single or repeated video frame. This ensures that the position of the foreground object is strictly aligned with the predicted video, enhancing the accuracy of the foreground object's positional information and appearance features.

[0106] When extracting features from the first input data based on the encoder, dimensionality reduction and feature purification can be achieved. Specifically, the spatial dimension of the first input data is gradually compressed through convolutional and pooling layers in the encoder. During the compression process, noise is weakened or suppressed due to the lack of spatial consistency and structural correlation.

[0107] S41, Based on the pre-stored control information, control parameters are generated using the encoder; wherein the control information indicates the need to adjust the foreground object in the first input data.

[0108] In one embodiment of this application, an encoder can be used to encode pre-stored control information to obtain control parameters. The control information indicates the need to adjust the foreground object in the first input data. For example, the control information may indicate a need to adjust the size of the foreground object; the control information may also indicate a need to adjust the shape of the foreground object.

[0109] S42, Based on the control parameters and the first input feature, determine the first feature.

[0110] In one embodiment of this application, control parameters and a first input feature can be input into an encoder, and the first input feature can be encoded again based on the control parameters to obtain a first feature.

[0111] like Figure 6 The diagram shown is a flowchart of a method for determining fusion features according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining fusion features provided in this embodiment includes the following steps.

[0112] S50, the first reference image and the second reference image are encoded to obtain encoded features.

[0113] In one embodiment of this application, a first instance image in a first reference image and a second instance image in a second reference image can be input into a preset multimodal and training encoder (Contrastive Language-Image Pre-training, CLIP) to obtain encoded features that can characterize deep information in the reference video.

[0114] S51, based on the first instance image and the second instance image, determine the mask features of the foreground object.

[0115] In one embodiment of this application, the mask features of a foreground object can be determined based on the position of a first instance image in a first reference image and the position of a second instance image in a second reference image. The mask features can be used to indicate the positional information of the foreground object.

[0116] S52, based on the first feature and the mask feature, adjust the weights of information in different dimensions of the encoded feature to obtain the fusion feature of the reference video.

[0117] In one embodiment of this application, the first reference image and the second reference image can be weighted using an attention mechanism based on the first feature to obtain the fused features of the reference video.

[0118] like Figure 7 The diagram shown is a flowchart of a method for determining fusion features according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining fusion features provided in this embodiment includes the following steps.

[0119] S60, determine the similarity between the first feature and the encoded feature.

[0120] In one embodiment of this application, the similarity between the first feature and the encoded feature can be determined based on the dot product of the first feature and the encoded feature.

[0121] S61, Based on the product of the similarity and the mask features, determine the fusion features of the reference video.

[0122] In one embodiment of this application, the first input feature corresponding to the first input data can be used as the query instruction of the attention mechanism, and the encoded feature can be used as the key and value. The first input feature and the encoded feature are fused through attention calculation to obtain the fused feature of the reference video. The attention mechanism satisfies the following formula: ; in, The first feature representing the encoder output, and The encoding features representing the reference video, The matrix dimension representing the encoded features, A spatial mask representing the foreground object is used to limit the scope of the attention mechanism. Aligning the semantic features of foreground information and the reference video based on the attention mechanism ensures that the visual attributes of the foreground object in the generated predicted video remain consistent with those in the reference video. For example, the visual attributes of the foreground object can be visual features such as texture or color features, and this application does not limit this.

[0123] like Figure 8 The diagram shown is a flowchart of a method for training a prediction model according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for training a prediction model provided in this embodiment includes the following steps.

[0124] S70: Acquire a first video of the vehicle, and extract a first sample image and a second sample image from the first video.

[0125] In one embodiment of this application, to improve the accuracy of the prediction model in generating prediction videos based on reference videos, the prediction model can also be trained based on a pre-acquired first video. The first video refers to visual information about the vehicle's surrounding environment collected during the vehicle's movement.

[0126] The first sample image can be the initial video frame of the first video, and the second sample image can be other video frames in the first video besides the initial video frame.

[0127] S71, First training data is generated based on the first instance image corresponding to the specified object in the first sample image and the second instance image corresponding to the specified object in the second sample image.

[0128] In one embodiment of this application, feature extraction and encoding operations can be performed on a first instance image in a first sample image and a second instance image in a second sample image to obtain encoded features corresponding to a first video. Specifically, the first instance image in the first sample image and the second instance image in the second sample image can be input into a preset multimodal and trained encoder (Contrastive Language-Image Pre-training, CLIP) to obtain encoded features that can characterize deep information in the first video.

[0129] S72, Based on the first training data, generate the first training feature using a preset initial model.

[0130] In one embodiment of this application, the encoded features can be used as the semantics of the first video to indicate the visual information contained in the first video; the encoded vector corresponding to the reference video can be used as the key and value, and cross-attention calculation can be performed with the first input features, so that the process of generating the prediction video can give priority to the region related to the first input features, thereby ensuring that the visual information of the foreground object in the generated prediction video has high quality.

[0131] S73, perform a weighted operation on the first sample image and the second sample image based on the first training features to obtain the sample fusion features of the training video.

[0132] In one embodiment of this application, a fusion feature of the reference video can be obtained by weighting the first reference image and the second reference image based on a first feature using an attention mechanism. Specifically, the first input feature corresponding to the first input data can be used as the query instruction of the attention mechanism, and the encoded feature can be used as the key and value. The first input feature and the encoded feature are fused through attention calculation to obtain the fusion feature of the reference video. The attention mechanism satisfies the following formula: ; in, The first feature representing the encoder output, and The encoding features representing the reference video, The matrix dimension representing the encoded features, A spatial mask representing the foreground object is used to limit the scope of the attention mechanism. Aligning the semantic features of foreground information and the reference video based on the attention mechanism ensures that the visual attributes of the foreground object in the generated predicted video remain consistent with those in the reference video. For example, the visual attributes of the foreground object can be visual features such as texture or color features, and this application does not limit this.

[0133] S74, Based on the sample fusion features, use the initial model to predict the second video of the vehicle's driving process.

[0134] In one embodiment of this application, fused features can be input into the decoder of the prediction model, and based on the fused features, the decoder in the prediction model can be used to generate a predicted video of the vehicle driving process.

[0135] The decoder of the prediction model is used to reconstruct multiple video frames with low noise levels and high accuracy of foreground information based on fused features, and the predicted video is obtained by combining multiple video frames.

[0136] S75, Based on the semantic similarity between the first video and the second video, determine the reward value of the initial model.

[0137] In one embodiment of this application, a similarity reward feedback of foreground information can be introduced during the training phase of the prediction model. The visual consistency between foreground objects in the predicted video and reference objects in the real video can be improved through the feedback during the training phase.

[0138] S76, Based on the reward value, determine the loss value of the initial model.

[0139] In one embodiment of this application, a loss function can be constructed based on the reward value, and the prediction model can be updated based on the loss function. The loss value of the loss function is used to maximize the similarity between the foreground information in the predicted video and the foreground information in the real video.

[0140] S77, Update the initial model based on the loss value until the loss value meets the preset convergence condition, stop updating the initial model and obtain the prediction model trained to the convergence state.

[0141] In one embodiment of this application, when the loss value does not meet the preset convergence condition, it indicates that the difference between the first video and the second video is large, and the initial model can be continuously updated; when the loss value meets the preset convergence condition, it indicates that the difference between the first video and the second video is small, and the initial model can be stopped from being updated, thus obtaining a prediction model trained to a convergent state.

[0142] like Figure 9 The diagram shown is a flowchart of a method for determining a reward value according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different needs. The method for determining a reward value provided in this embodiment of the application includes the following steps.

[0143] S80, based on the timestamp of the first video frame in the first video, determine the second video frame corresponding to the first video frame from the second video; the timestamp corresponding to the second video frame is the same as the timestamp corresponding to the first video frame.

[0144] In one embodiment of this application, in order to assess the similarity between a first video and a second video, a second video frame corresponding to the first video frame can be determined from the second video based on the timestamp of the first video frame in the first video. The timestamp corresponding to the second video frame is the same as the timestamp corresponding to the first video frame.

[0145] S81, based on the similarity between the first video frame and the second video frame, determine the semantic similarity between the first video and the second video.

[0146] In one embodiment of this application, image information of foreground objects can be extracted from a second video and a first video generated by a prediction model. Only foreground objects present in the initial frame of the first video are retained, and these are input into a CLIP encoder to obtain feature vectors corresponding to the predicted video. The first feature vector is obtained by representing the i-th foreground object in the t-th frame of the second video and obtaining the corresponding feature vector of the first video. , representing the foreground object corresponding to the first video.

[0147] For example, the cosine similarity between the feature vector corresponding to the second video and the feature vector corresponding to the first video can be calculated as semantic similarity to obtain the reward value of the prediction model. A higher reward value indicates a higher similarity between the second video and the first video output by the prediction model. The reward value is determined according to the following relationship: ; in Let represent the feature vector of the i-th pixel in the t-th frame of the second video. Let i represent the feature vector of the i-th pixel in the t-th frame of the first video.

[0148] S82, determine the semantic similarity as the reward value of the initial model.

[0149] like Figure 10 The diagram shown is a flowchart of a method for determining a loss value according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining a loss value provided in this embodiment of the application includes the following steps.

[0150] S90, based on the product of the reward value and the preset learning rate, determine the first loss value of the initial model; the learning rate is used to indicate the convergence speed of the initial model.

[0151] In one embodiment of this application, a loss function can be constructed based on the reward value, and the prediction model can be updated based on the loss function. The loss value of the loss function is used to maximize the similarity between the foreground information in the predicted video and the foreground information in the real video. Specifically, the loss function satisfies the following relationship: ; in, This represents the preset learning rate. Specifically, a higher learning rate indicates a faster convergence speed of the initial model, fewer updates are needed, and lower accuracy of the parameters in the resulting prediction model; conversely, a lower learning rate indicates a slower convergence speed of the initial model, more updates are needed, and higher accuracy of the parameters in the resulting prediction model.

[0152] S91, based on the mean square error of the first video and the second video, determine the second loss value of the initial model.

[0153] S92, based on the first loss value and the second loss value, determine the loss value of the initial model.

[0154] In one embodiment of this application, the loss value of the initial model can be determined based on the first loss value and the second loss value. For example, the sum of the first loss value and the second loss value can be determined as the loss value of the initial model; alternatively, a weighted summation operation can be performed on the first loss value and the second loss value to obtain the loss value of the initial model.

[0155] Please see Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement a video generation method as described in any of the above embodiments.

[0156] In one embodiment of this application, the electronic device 100 further includes a bus and a computer program, such as a video generation program, stored in the memory 12 and executable on the processor 13.

[0157] Figure 11 Only an electronic device 100 with memory 12 and processor 13 is shown; those skilled in the art will understand that... Figure 11 The structure shown does not constitute a limitation on the electronic device 100, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0158] Combination Figure 3The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement the video generation method, and the processor 13 can execute the plurality of instructions to achieve: acquiring a reference video of the vehicle; extracting a first reference image and a second reference image from the reference video, wherein the reference video is obtained by shooting foreground objects around the vehicle; generating first input data based on a first instance image corresponding to the foreground object in the first reference image and a second instance image corresponding to the foreground object in the second reference image; the first input data indicating the size information of the foreground object and the position information of the foreground object in the area where the vehicle is located; generating a first feature based on the first input data using an encoder in a preset prediction model; performing a weighted operation on the first reference image and the second reference image using an attention mechanism based on the first feature to obtain a fusion feature of the reference video; and generating a predicted video of the vehicle driving process using a decoder in the prediction model based on the fusion feature.

[0159] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0160] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, the electronic device 100 may also include input / output devices, network access devices, etc.

[0161] It should be noted that electronic device 100 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0162] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as the portable hard drive of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 100. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 100, such as the code of a video generation program, but also to temporarily store data that has been output or will be output.

[0163] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 100, connecting to various components of the electronic device 100 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing video generation programs) and calls data stored in the memory 12 to perform various functions and process data of the electronic device 100.

[0164] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in the various video generation method embodiments described above, for example... Figure 3 The steps are shown.

[0165] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 100.

[0166] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the video generation method described in the various embodiments of this application.

[0167] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0168] The computer program includes computer program code, which may be in the form of source code, foreground object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.

[0169] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0170] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 11 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0171] This application also provides a computer-readable storage medium (not shown), which stores computer-readable instructions that are executed by a processor in an electronic device to implement the video generation method described in any of the above embodiments.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0174] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A video generation method, characterized in that, The method includes: A reference video of the vehicle is acquired, and a first reference image and a second reference image are extracted from the reference video. The reference video includes foreground objects around the vehicle. The foreground objects include at least pedestrians and / or other vehicles around the vehicle. Based on the size and position of the first instance image corresponding to the foreground object in the first reference image and the second instance image corresponding to the foreground object in the second reference image, the first instance image is pasted into the second reference image to generate first input data; the first input data indicates the size information of the foreground object and the position information of the foreground object in the area where the vehicle is located; Based on the first input data, a first feature is generated using a preset prediction model; the first feature indicates the conditional latent variable output by the prediction model, which is used to indicate the location information of the foreground object. Based on the first feature, the encoded features corresponding to the first instance image and the second instance image are weighted using an attention mechanism to obtain the fusion feature of the reference video; the fusion feature indicates the texture and color features of the foreground object; Based on the fusion features, the prediction model is used to generate a predicted video of the vehicle's driving process.

2. The video generation method according to claim 1, characterized in that, The step of pasting the first instance image into the second reference image based on the size and position of the first instance image corresponding to the foreground object in the first reference image and the second instance image corresponding to the foreground object in the second reference image to generate first input data includes: The size of the first instance image is adjusted based on the size data of the second instance image to obtain a copied image; Based on the copied image, the second instance image in the second reference image is replaced to obtain the input image; Based on the input image, the first input data is determined.

3. The video generation method according to claim 1, characterized in that, The step of generating a first feature based on the first input data using a preset prediction model includes: Based on the first input data, the first input feature is determined using the encoder in the prediction model; Based on pre-stored control information, the encoder is used to generate control parameters; wherein the control information indicates the need to adjust the foreground object in the first input data; The first feature is determined based on the control parameters and the first input feature.

4. The video generation method according to claim 1, characterized in that, The step of weighting the encoded features corresponding to the first instance image and the second instance image using an attention mechanism based on the first feature to obtain the fused features of the reference video includes: Encode the first instance image and the second instance image to obtain encoded features; Based on the first feature, the mask feature of the foreground object is determined; Based on the mask features, the weights of information in different dimensions of the encoding features are adjusted to obtain the fusion features of the reference video.

5. The video generation method according to claim 4, characterized in that, The step of adjusting the weights of different dimensions of information in the encoded features based on the mask features to obtain the fusion features of the reference video includes: Determine the similarity between the mask features and the encoded features; The fusion features of the reference video are determined based on the product of the similarity and the mask features.

6. The video generation method according to claim 1, characterized in that, The method further includes training the prediction model, including: Acquire a first video of the vehicle, and extract a first sample image and a second sample image from the first video; First training data is generated based on the first instance image corresponding to the specified object in the first sample image and the second instance image corresponding to the specified object in the second sample image; Based on the first training data, a first training feature is generated using a preset initial model; Based on the first training feature, a weighted operation is performed on the first sample image and the second sample image to obtain the sample fusion feature of the training video; Based on the sample fusion features, the initial model is used to predict a second video of the vehicle's driving process. The reward value of the initial model is determined based on the semantic similarity between the first video and the second video; Based on the reward value, determine the loss value of the initial model; The initial model is updated based on the loss value until the loss value meets the preset convergence condition. Then, the initial model is stopped from being updated and a prediction model trained to a convergent state is obtained.

7. The video generation method according to claim 6, characterized in that, Determining the reward value of the initial model based on the semantic similarity between the first video and the second video includes: Based on the timestamp of the first video frame in the first video, the second video frame corresponding to the first video frame is determined from the second video; the timestamp corresponding to the second video frame is the same as the timestamp corresponding to the first video frame. Based on the similarity between the first video frame and the second video frame, the semantic similarity between the first video and the second video is determined; The semantic similarity is determined as the reward value of the initial model.

8. The video generation method according to claim 6, characterized in that, Determining the loss value of the initial model based on the reward value includes: The first loss value of the initial model is determined based on the product of the reward value and the preset learning rate; the learning rate indicates the convergence speed of the initial model. Based on the mean squared error of the first video and the second video, the second loss value of the initial model is determined; Based on the first loss value and the second loss value, the loss value of the initial model is determined.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the video generation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the video generation method as described in any one of claims 1 to 8.

Images