Time sequence simulation image sequence generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a time sequence simulation image sequence generation method, device and equipment and a medium, and the method comprises the steps: obtaining a target electric power field operation video image, and coding the target electric power field operation video image into a starting token sequence through a video Tokenizer model, the video Tokenizer model is obtained by adopting the constructed training data set to train the Tokenizer model; inputting the initial token sequence into an autoregression time sequence planner, and generating a complete token sequence in an autoregression mode; reconstructing the complete token sequence into a coarse-grained image sequence through a video Tokenizer model; and rendering the coarse-grained image sequence by using the frame-level rendering model to obtain a final time sequence simulation image sequence. According to the invention, the generation quality of the time sequence simulation image sequence is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for generating time-series simulation image sequences. Background Technology

[0002] Power field operations are characterized by strong temporal sequence and high safety risks. In recent years, although there has been research and application of deep learning technology for the automatic identification of abnormalities or violations in surveillance videos, in real-world safety supervision, violations often exhibit a typical "low-frequency, long-tail" distribution. Because it is difficult to acquire labeled violation sequence data on a large scale, model training faces challenges of sample scarcity and class imbalance, thus affecting the cross-scenario generalization ability and stable alarm performance of the identification model.

[0003] To alleviate the problem of insufficient data, generative artificial intelligence methods have begun to be explored for synthesizing training samples. Among them, diffusion models combined with spatial condition control (such as edge maps, segmentation maps, and pose keypoints) have improved the structural controllability of single-frame generated content to some extent. However, for the task of generating time-series simulation image sequences for the dynamic process of "evolution of violations," existing methods are still generally constrained by problems such as cross-frame flickering, subject identity drift, and difficulty in maintaining spatiotemporal consistency among multiple targets. In complex power operation scenarios with multiple interactions and frequent occlusion, it is particularly difficult to simultaneously satisfy the "temporal realism and continuity" and "content controllability and compliance" of the generated sequence. Therefore, the quality of existing time-series simulation image sequence generation is relatively low. Summary of the Invention

[0004] This invention provides a method, apparatus, computer device, and medium for generating time-series simulation image sequences, in order to solve the technical problem of low quality in existing time-series simulation image sequence generation.

[0005] Firstly, a method for generating time-series simulation image sequences is provided, including: Acquire target power field operation video images, and encode the target power field operation video images into a starting token sequence using a video tokenizer model, wherein the video tokenizer model is obtained by training the tokenizer model using a constructed training dataset; The initial token sequence is input into the autoregressive time series planner to generate a complete token sequence in an autoregressive manner. The autoregressive time series planner is trained using the training dataset and a discrete token grid encoded by the video tokenizer model. The complete token sequence is reconstructed into a coarse-grained image sequence using the video tokenizer model; The coarse-grained image sequence is rendered using a frame-level rendering model to obtain the final time-series simulation image sequence.

[0006] Secondly, a time-series simulation image sequence generation device is provided, comprising: An encoding unit is used to acquire video images of the target power field operation and encode the video images of the target power field operation into a sequence of starting tokens using a video tokenizer model. The video tokenizer model is trained using a constructed training dataset. An input generation unit is used to input the initial token sequence into an autoregressive time series planner to generate a complete token sequence in an autoregressive manner. The autoregressive time series planner is trained using the training dataset and a discrete token grid encoded by the video tokenizer model. The reconstruction unit is used to reconstruct the complete token sequence into a coarse-grained image sequence through the video tokenizer model; The rendering unit is used to render the coarse-grained image sequence using a frame-level rendering model to obtain the final time-series simulation image sequence.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described time-series simulation image sequence generation method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described time-series simulation image sequence generation method.

[0009] The scheme implemented by the above-mentioned method, apparatus, computer equipment, and storage medium for generating time-series simulation image sequences can acquire target power field operation video images and encode the target power field operation video images into a starting token sequence using a video tokenizer model. The video tokenizer model is trained using a constructed training dataset. The starting token sequence is input into an autoregressive time-series planner to generate a complete token sequence in an autoregressive manner. The autoregressive time-series planner is trained using the training dataset and a discrete token grid encoded by the video tokenizer model. The complete token sequence is reconstructed into a coarse-grained image sequence using the video tokenizer model. The coarse-grained image sequence is rendered using a frame-level rendering model to obtain the final time-series simulation image sequence. In this invention, the target power field operation video image is first encoded into a structured initial token sequence using a video tokenizer model; then, a complete token sequence is generated based on the initial token sequence using an autoregressive time-series planner; finally, the final time-series simulation image sequence is obtained based on the complete token sequence using a video tokenizer model and a frame-level rendering model. This effectively eliminates the problems of cross-frame flickering, identity drift, and detail blurring in existing methods, thereby improving the quality of time-series simulation image sequence generation. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating a method for generating time-series simulation image sequences according to an embodiment of the present invention; Figure 2 yes Figure 1 A schematic diagram of a specific implementation of step S120; Figure 3 This is a schematic block diagram of a timing simulation image sequence generation device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The time-series simulation image sequence generation method provided in this invention can be applied to a client or server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. Currently, in the field of artificial intelligence technology, the quality of existing time-series simulation image sequence generation is relatively low. To address the above problems, this invention proposes a time-series simulation image sequence generation method. This method first encodes the target power field operation video image into a structured initial token sequence using a video tokenizer model; then, based on the initial token sequence, it generates a complete token sequence using an autoregressive time-series planner; finally, based on the complete token sequence, it obtains the final time-series simulation image sequence using a video tokenizer model and a frame-level rendering model. This effectively eliminates the problems of cross-frame flickering, identity drift, and detail blurring in existing methods, thereby improving the quality of time-series simulation image sequence generation. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 1 As shown, Figure 1 A flowchart of a time-series simulation image sequence generation method provided in an embodiment of the present invention includes the following steps: S110-S140.

[0015] S110. Acquire target power field operation video images, and encode the target power field operation video images into a starting token sequence using a video tokenizer model, wherein the video tokenizer model is obtained by training the tokenizer model using a constructed training dataset.

[0016] In this embodiment, the video tokenizer model includes an encoder, a codebook, and a decoder. The encoder maps the target power field operation video image into a continuous feature representation. The codebook performs nearest neighbor matching on the continuous feature identifiers using its own embedding vector to obtain the discrete token grid and the quantized embedding vector corresponding to the discrete token grid. The decoder reconstructs the video sequence from the quantized embedding vector. It should be noted that the encoder includes spatiotemporal downsampling and attention residual blocks. The spatiotemporal downsampling includes multiple 3D convolutions, and the attention residual blocks include layer normalization and axial self-attention mechanisms. The decoder is the inverse process of the encoder and includes attention residual blocks and 3D transposed convolutional layers.

[0017] The specific steps for encoding the acquired target power field operation video images into a starting token sequence using the video tokenizer model are as follows: The process involves acquiring a target power field operation video image, typically a single frame, which is then cropped or scaled to a uniform resolution (e.g., H×W = 256×256 pixels). This cropped or scaled image is then input into the encoder of a video tokenizer model, which includes spatiotemporal downsampling and attention residual blocks. Specifically, spatiotemporal downsampling, comprised of multiple 3D convolutional layers (Conv3D), simultaneously downsamples the input cropped or scaled video image in both temporal and spatial dimensions, extracting local spatiotemporal texture and motion features. Subsequently, the attention residual blocks (containing LayerNorm layers and an axial self-attention mechanism, AxialAttention) further process the downsampled features to model long-range spatiotemporal dependencies while reducing computational complexity through axial decomposition. The final output of the encoder is a continuous feature representation. ; where (T′, H′, W′) is the spatiotemporal grid size after downsampling.

[0018] The continuous feature representation H obtained in the previous step is fed into the quantization module. The core of this module is a pre-trained codebook. The codebook contains K d-dimensional embedding vectors. For each feature vector h at a spatial-temporal location in H, a nearest neighbor search is performed to find the embedding vector e with the closest Euclidean distance in the codebook. k ,Right now

[0019] The index k at this position is a discrete token. Collecting the token indices of all positions yields a discrete token grid Z∈{1,…,K}. T′×H′×W′ Simultaneously, the corresponding vector e is retrieved from the codebook based on index k. k The quantized embedding vector q(h) = e at that position is obtained. k The quantized embedding vectors at all positions constitute the quantized feature representation q(H).

[0020] During the sequence generation stage, tokens corresponding to the cropped or scaled target power field operation video images are extracted from the complete discrete token grid Z to form the initial token sequence.

[0021] It should be noted that the step of training the Tokenizer model using the constructed training dataset to obtain the video Tokenizer model includes: acquiring unlabeled power field operation video data, and preprocessing and quality screening the power field operation video data to construct the training dataset; inputting the training dataset into the Tokenizer model, and training the Tokenizer model using a composite loss function to obtain the video Tokenizer model. It should also be noted that the step of acquiring unlabeled power field operation video data, and preprocessing and quality screening the power field operation video data to construct the training dataset includes: sequentially sampling the power field operation video data at a fixed frame rate, cropping or scaling it to a uniform resolution, and extracting fixed-length video segments using a sliding window or random window method to obtain a preliminary dataset; removing low-quality video segments with large areas of black screen, overexposure, severe blur, or long periods of static repeated frames from the preliminary dataset to obtain the training dataset.

[0022] Specifically, constructing a high-quality unlabeled training dataset involves: First, collecting a large amount of raw video data from the power operation site monitoring system to obtain power operation video data, which constitutes the initial unlabeled dataset D. u To construct a high-quality training dataset D suitable for model training. u ∗ Preprocessing and quality screening procedures are required: Preprocessing: For D uEach long video in the dataset undergoes uniform processing to ensure data format consistency. Specifically, this includes: Fixed frame rate sampling: The long video is resampled at a preset fixed frame rate f (e.g., 15fps or 30fps) to unify the information density in the temporal dimension and avoid inconsistencies in temporal information caused by differences in the original frame rate. Resolution unification: Each frame is cropped or scaled to a uniform resolution H×W (e.g., 256×256 or 512×512 pixels) to ensure spatial consistency and facilitate model processing. Segment extraction: Based on the processed video, a large number of short video segments of fixed length T (e.g., 16 frames or 32 frames) are extracted using a sliding window (step size can be smaller than the window length to increase data volume) or a random window. Each segment can be represented as... ,in Thus, a preliminary dataset was obtained.

[0023] Quality Screening: Video clips in the initial dataset undergo multi-dimensional quality assessment and screening to remove low-quality clips that may interfere with model training and affect learning outcomes. Screening criteria include, but are not limited to, the following four types of problems: Large areas of black screen: More than a certain percentage (e.g., 30%) of the image is pure black, possibly due to equipment malfunction or occlusion causing information loss. Overexposure: Loss of image detail due to excessive lighting or improper exposure parameters; highlight areas lose texture information. Severe motion blur: Blurred image due to rapid movement or equipment shake, making it impossible to identify the subject's outline. Long periods of static repeated frames: Scenes remain unchanged for extended periods, or multiple consecutive frames contain identical content. After removing these low-quality clips from the initial dataset, a high-quality, high-fidelity training dataset D is obtained. u ∗ .

[0024] Tokenizer model training: Train a video tokenizer model that can map the video to a discrete token space and reconstruct it. Its parameters are {E,C,D}, which represent the encoder, codebook and decoder, respectively.

[0025] Model forward pass: The training dataset D... u ∗ Video clips X are batch-input into the Tokenizer model; Encoding and quantization: The encoder maps X into continuous feature representations. Where (T′, H′, W′) are the spatiotemporal grid dimensions after downsampling, and d is the feature dimension. Subsequently, the codebook... (Containing K d-dimensional embedding vectors) Perform nearest neighbor quantization on the feature vector at each spatial-temporal location in H, that is, find the vector e with the closest Euclidean distance in the codebook. k This yields a discrete token grid Z∈{1,…,K} T′×H′×W′And its corresponding quantized embedding vector q(H). Decoding and reconstruction: The decoder (as the inverse process of the encoder, consisting of attention residual blocks and 3D transposed convolutional layers) reconstructs q(H) into the video. Its dimension is the same as the original input X.

[0026] Loss calculation and parameter optimization: using a composite loss function The model is trained end-to-end. It should be noted that the composite loss function includes three key terms: reconstruction loss... Reconstructing video using L1 norm constraints To ensure reconstruction quality, the image is pixel-level close to the original video X. (Codebook loss) By stopping the gradient operator sg[·] to fix the encoder output, the codebook vector is made to move closer to the encoder output, and the codebook parameters are updated.

[0027] Promised loss The quantization result is fixed, constraining the encoder output to be close to its selected codebook vector, ensuring the encoder learns meaningful representations. Here, λ and β are hyperparameters used to balance the weights of different loss terms. This composite loss is minimized using the backpropagation algorithm, and the model parameters are iteratively updated using optimizers such as Adam or SGD, ultimately training a convergent and stable video tokenizer model. This model can efficiently encode any video segment into a discrete token sequence.

[0028] S120. Input the initial token sequence into the autoregressive time series planner to generate a complete token sequence in an autoregressive manner. The autoregressive time series planner is trained using the training dataset and a discrete token grid encoded by the video tokenizer model.

[0029] In this embodiment, as Figure 2 As shown, step S120 includes steps S121-S122: S121, obtaining the spatiotemporal location code based on the initial token sequence, and obtaining the appearance anchor point through an image encoder based on the first frame image in the target power field operation video image; S122, obtaining the stage control quantity generated by a predefined script, and inputting the spatiotemporal location code, the appearance anchor point, and the stage control quantity as conditional information into the autoregressive time series planner to generate the complete token sequence in an autoregressive manner. Specifically, based on the position index i of each token in the initial token sequence in the flattened one-dimensional sequence, the corresponding spatiotemporal location code p is obtained by looking up a table from the learnable parameters of the autoregressive time series planner. i, the spatio-temporal position encoding is used to distinguish the spatio-temporal attribution of tokens. Meanwhile, based on the first frame image I0 in the target power field operation video image, its high-level visual features are extracted by an independent image encoder E1 to obtain an appearance anchor c = E1(I0). This appearance anchor is used to lock the appearance features of the scene, personnel and equipment during the generation process to maintain the subject consistency. In addition, a sequence of stage control quantities {π1, π2,..., π T} is obtained, which is used to express the evolution intensity of the violation behavior from mild to obvious. Through a predefined mapping function: ; the position index i in the one-dimensional sequence is mapped to its corresponding frame index t, so that the stage control quantity π at the frame level t is associated with the corresponding token, obtaining π t(i) . Then, the starting token sequence z<i, its corresponding spatio-temporal position encoding p<i, appearance anchor c and stage control quantity π t(i) are jointly used as conditional information and input into the trained autoregressive temporal planner P θ . Based on the causal masking mechanism, the autoregressive temporal planner predicts the conditional probability distribution i of the next position token z with the prefix history information as the condition. Sampling is carried out according to this probability distribution to obtain a new token z i and append it to the end of the sequence, thereby updating the historical prefix. Repeat this causal autoregressive process of "prediction-sampling-append" to iteratively generate each subsequent token until a complete token sequence of the specified length N is generated.

[0030] It should be noted that the step of training the time-series planner to obtain the autoregressive time-series planner using the training dataset and the discrete token grid obtained by the video tokenizer model includes: flattening the discrete token grid obtained by the video tokenizer model into a one-dimensional token sequence in a time-first-space scanning order, and superimposing a learnable spatiotemporal position code on each token in the one-dimensional token sequence; obtaining appearance anchors through an image encoder based on the first frame image of the first training sample in the training dataset; obtaining the stage control variable generated by a predefined script, and inputting the spatiotemporal position code, the appearance anchors, and the stage control variable as conditional information into the time-series planner to obtain the conditional probability distribution of the current token; calculating the negative log-likelihood loss based on the conditional probability distribution, and optimizing the model parameters of the time-series planner based on the negative log-likelihood loss to obtain the autoregressive time-series planner. Specifically, step one: sequence construction and conditional information preparation. First, the training dataset D... u ∗ Each training sample (i.e., a short video clip X) is input into a pre-trained video tokenizer model, and encoded into a discrete token grid Z∈{1,…,K} by its encoder and codebook. T′×H′×W′ Subsequently, the three-dimensional token grid Z is flattened into a one-dimensional token sequence according to a fixed scanning order of "time first, space second". The total sequence length N = T′ × H′ × W′. This order ensures that the tokens at the beginning of the sequence correspond to tokens in earlier frames, and the tokens at the end correspond to tokens in later frames, giving the model a natural temporal generation rhythm. To avoid confusion between tokens at different spatiotemporal positions in the sequence, a learnable spatiotemporal position encoding p is superimposed on each position index i in the sequence. i This encoding, as part of the model parameters, enables the model to distinguish the spatiotemporal affiliation of tokens.

[0031] Simultaneously, prepare the necessary conditional information for training: the appearance anchor point c and the stage control variable π. t Appearance anchor point c and stage control quantity π t The acquisition process is as described above and will not be repeated here for simplicity. Step 2: Conditional probability modeling and loss calculation, using the prepared one-dimensional token sequence z, spatiotemporal location code p, appearance anchor point c, and stage control variable π. t(i)As input, it is used to train a temporal planner (usually a Decoder-only Transformer in structure). During training, the Teacher Forcing strategy (the "teacher forcing" strategy or the "teacher guidance" strategy, that is, using the true prefix of historical tokens to predict the true next item) is adopted, that is, the true historical token prefix z and its corresponding conditional information (p, c, π t(i) ) are used as the conditional information input, and the temporal planner is used to predict the conditional probability distribution of the token z i at the current position. This conditional probability distribution is obtained by normalizing the predicted value output by the temporal planner through the softmax function, that is: ; Based on this conditional probability distribution, calculate the negative log-likelihood loss of the temporal planner at all N positions: . This loss function measures the difference between the probability distribution predicted by the temporal planner and the true token sequence. Step 3: Model parameter optimization. Through the backpropagation algorithm, calculate the gradient of the negative log-likelihood loss LAR with respect to the model parameters of the temporal planner. Use optimizers such as Adam or SGD, and iteratively update the parameters according to the calculated gradient to minimize the loss LAR. This optimization process enables the temporal planner to gradually learn how to accurately predict the probability distribution of the next token according to the historical token prefix under various conditional constraints. When the model training converges, an autoregressive temporal planner available for inference generation is obtained. Understandably, this planner has the ability to generate a future frame token sequence that conforms to spatio-temporal consistency and behavioral evolution logic in the discrete token space based on the starting conditions and control signals.

[0032] S130. Reconstruct the complete token sequence into a coarse-grained image sequence through the video Tokenizer model.

[0033] Specifically, reshape the complete token sequence back to a three-dimensional discrete token grid through a reshape operation . Subsequently, input this token grid into the decoder of the trained video Tokenizer model. The decoder maps the discrete token indices back to the pixel space through its attention residual block and 3D transposed convolutional layer, and reconstructs the corresponding coarse-grained temporal image sequence , where . This sequence already has consistency in structure and main outline, but the texture details need to be further refined.

[0034] S140. Render the coarse-grained image sequence using a frame-level rendering model to obtain the final time-series simulation image sequence.

[0035] Specifically, the coarse-grained image sequence The input is fed into a pre-trained frame-level rendering model, which acts as a powerful image editor to perform high-fidelity rendering of key visual elements in power field operation scenarios, such as texture details, fine structures of personal protective equipment, hand-tool-equipment interaction boundaries, and complex occlusion fusion. This rendering process effectively compensates for the shortcomings in visual realism of the results generated by the autoregressive temporal planner and tokenizer decoder, ultimately outputting a final temporal simulation image sequence that combines high visual quality and rich detail with spatiotemporal consistency, thus meeting the stringent data fidelity requirements of downstream model training and validation.

[0036] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0037] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0038] In one embodiment, a time-series simulation image sequence generation apparatus 200 is provided, which corresponds one-to-one with the time-series simulation image sequence generation method in the above embodiments. For example... Figure 3 As shown, the time-series simulation image sequence generation device 200 includes an acquisition and encoding unit 201, an input generation unit 202, a reconstruction unit 203, and a rendering unit 204. Detailed descriptions of each functional module are as follows: The encoding unit 201 is used to acquire a target power field operation video image and encode the target power field operation video image into a starting token sequence through a video tokenizer model. The video tokenizer model is obtained by training the tokenizer model using a constructed training dataset. The input generation unit 202 is used to input the initial token sequence into the autoregressive time series planner to generate a complete token sequence in an autoregressive manner. The autoregressive time series planner is trained using the training dataset and a discrete token grid encoded by the video tokenizer model. Reconstruction unit 203 is used to reconstruct the complete token sequence into a coarse-grained image sequence through the video tokenizer model; The rendering unit 204 is used to render the coarse-grained image sequence using a frame-level rendering model to obtain the final time-series simulation image sequence.

[0039] In one embodiment, the encoding unit 201 is specifically used for: Unlabeled power field operation video data is acquired, and the power field operation video data is preprocessed and quality filtered to construct the training dataset; The training dataset is input into the Tokenizer model, and the Tokenizer model is trained using a composite loss function to obtain the video Tokenizer model.

[0040] In one embodiment, the encoding unit 201 is further configured to: The power field operation video data is sampled sequentially at a fixed frame rate, cropped or scaled to a uniform resolution, and video segments of fixed length are extracted using a sliding window or random window method to obtain a preliminary dataset. The training dataset is obtained by removing low-quality video clips with large areas of black screen, overexposure, severe blur, or long periods of static repeated frames from the preliminary dataset.

[0041] In one embodiment, the encoding unit 201 is further configured to: The video tokenizer model includes an encoder, a codebook, and a decoder. The encoder maps the target power field operation video image into a continuous feature representation. The codebook performs nearest neighbor matching on the continuous feature identifiers using its own embedding vector to obtain the discrete token grid and the quantized embedding vector corresponding to the discrete token grid. The decoder reconstructs the video sequence from the quantized embedding vector.

[0042] In one embodiment, the encoding unit 201 is further configured to: The encoder includes spatiotemporal downsampling and attention residual blocks, wherein the spatiotemporal downsampling includes multi-layer 3D convolution, and the attention residual blocks include layer normalization and axial self-attention mechanisms; the decoder is the inverse process of the encoder, and includes attention residual blocks and 3D transposed convolutional layers.

[0043] In one embodiment, the input generation unit 202 is specifically used for: The spatiotemporal location code is obtained based on the starting token sequence, and the appearance anchor point is obtained through an image encoder based on the first frame image in the target power field operation video image; The stage control variable generated by the predefined script is obtained, and the spatiotemporal position code, the appearance anchor point, and the stage control variable are input into the autoregressive time series planner as conditional information to generate the complete token sequence in an autoregressive manner.

[0044] In one embodiment, the input generation unit 202 is further configured to: The discrete token grid obtained by encoding the video tokenizer model is flattened into a one-dimensional token sequence in a scanning order of first time and then space, and a learnable spatiotemporal position code is superimposed on each token in the one-dimensional token sequence; Based on the first frame image of the first training sample in the training dataset, the appearance anchor point is obtained by an image encoder. Obtain the stage control quantity generated by the predefined script, and input the spatiotemporal position code, the appearance anchor point and the stage control quantity as conditional information into the time series planner to obtain the conditional probability distribution of the current token; The negative log-likelihood loss is calculated based on the conditional probability distribution, and the model parameters of the time series planner are optimized according to the negative log-likelihood loss to obtain the autoregressive time series planner.

[0045] The time-series simulation image sequence generation device of this invention first encodes the target power field operation video image into a structured starting token sequence using a video tokenizer model; then, it generates a complete token sequence based on the starting token sequence using an autoregressive time-series planner; finally, it obtains the final time-series simulation image sequence based on the complete token sequence using a video tokenizer model and a frame-level rendering model. This effectively eliminates the problems of cross-frame flickering, identity drift, and detail blurring in existing methods, thereby improving the quality of time-series simulation image sequence generation.

[0046] Specific limitations regarding the timing simulation image sequence generation device can be found in the limitations of the timing simulation image sequence generation method described above, and will not be repeated here. Each unit in the aforementioned timing simulation image sequence generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0047] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a time-series simulation image sequence generation method on the server side.

[0048] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a time-series simulation image sequence generation method.

[0049] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described time-series simulation image sequence generation method.

[0050] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described time-series simulation image sequence generation method.

[0051] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0052] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0053] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0054] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for generating time-series simulation image sequences, characterized in that, include: Acquire target power field operation video images, and encode the target power field operation video images into a starting token sequence using a video tokenizer model, wherein the video tokenizer model is obtained by training the tokenizer model using a constructed training dataset; The initial token sequence is input into the autoregressive time series planner to generate a complete token sequence in an autoregressive manner. The autoregressive time series planner is trained using the training dataset and a discrete token grid encoded by the video tokenizer model. The complete token sequence is reconstructed into a coarse-grained image sequence using the video tokenizer model; The coarse-grained image sequence is rendered using a frame-level rendering model to obtain the final time-series simulation image sequence.

2. The method for generating time-series simulation image sequences as described in claim 1, characterized in that, The steps of training the Tokenizer model using the constructed training dataset to obtain the video Tokenizer model include: Unlabeled power field operation video data is acquired, and the power field operation video data is preprocessed and quality filtered to construct the training dataset; The training dataset is input into the Tokenizer model, and the Tokenizer model is trained using a composite loss function to obtain the video Tokenizer model.

3. The method for generating time-series simulation image sequences as described in claim 2, characterized in that, The steps of acquiring unlabeled power field operation video data and preprocessing and quality screening the power field operation video data to construct the training dataset include: The power field operation video data is sampled sequentially at a fixed frame rate, cropped or scaled to a uniform resolution, and video segments of fixed length are extracted using a sliding window or random window method to obtain a preliminary dataset. The training dataset is obtained by removing low-quality video clips with large areas of black screen, overexposure, severe blur, or long periods of static repeated frames from the preliminary dataset.

4. The method for generating time-series simulation image sequences as described in claim 1, characterized in that, The video tokenizer model includes an encoder, a codebook, and a decoder. The encoder maps the target power field operation video image into a continuous feature representation. The codebook performs nearest neighbor matching on the continuous feature identifiers using its own embedding vector to obtain the discrete token grid and the quantized embedding vector corresponding to the discrete token grid. The decoder reconstructs the video sequence from the quantized embedding vector.

5. The method for generating time-series simulation image sequences as described in claim 4, characterized in that, The encoder includes spatiotemporal downsampling and attention residual blocks, wherein the spatiotemporal downsampling includes multi-layer 3D convolution, and the attention residual blocks include layer normalization and axial self-attention mechanisms; the decoder is the inverse process of the encoder, and includes attention residual blocks and 3D transposed convolutional layers.

6. The method for generating time-series simulation image sequences as described in claim 4, characterized in that, The step of inputting the initial token sequence into an autoregressive time-series planner to generate a complete token sequence in an autoregressive manner includes: The spatiotemporal location code is obtained based on the starting token sequence, and the appearance anchor point is obtained through an image encoder based on the first frame image in the target power field operation video image; The stage control variable generated by the predefined script is obtained, and the spatiotemporal position code, the appearance anchor point, and the stage control variable are input into the autoregressive time series planner as conditional information to generate the complete token sequence in an autoregressive manner.

7. The method for generating time-series simulation image sequences as described in claim 1, characterized in that, The steps of training the time series planner using the training dataset and the discrete token grid encoded by the video tokenizer model to obtain the autoregressive time series planner include: The discrete token grid obtained by encoding the video tokenizer model is flattened into a one-dimensional token sequence in a scanning order of first time and then space, and a learnable spatiotemporal position code is superimposed on each token in the one-dimensional token sequence; Based on the first frame image of the first training sample in the training dataset, the appearance anchor point is obtained by an image encoder. Obtain the stage control quantity generated by the predefined script, and input the spatiotemporal position code, the appearance anchor point and the stage control quantity as conditional information into the time series planner to obtain the conditional probability distribution of the current token; The negative log-likelihood loss is calculated based on the conditional probability distribution, and the model parameters of the time series planner are optimized according to the negative log-likelihood loss to obtain the autoregressive time series planner.

8. A time-series simulation image sequence generation device, characterized in that, include: An encoding unit is used to acquire video images of the target power field operation and encode the video images of the target power field operation into a sequence of starting tokens using a video tokenizer model. The video tokenizer model is trained using a constructed training dataset. An input generation unit is used to input the initial token sequence into an autoregressive time series planner to generate a complete token sequence in an autoregressive manner. The autoregressive time series planner is trained using the training dataset and a discrete token grid encoded by the video tokenizer model. The reconstruction unit is used to reconstruct the complete token sequence into a coarse-grained image sequence through the video tokenizer model; The rendering unit is used to render the coarse-grained image sequence using a frame-level rendering model to obtain the final time-series simulation image sequence.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the time-series simulation image sequence generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the time-series simulation image sequence generation method as described in any one of claims 1 to 7.

Images