Method and apparatus for processing video stream, method for generating video stream

Abstract

This application discloses a video stream processing method and apparatus, and a video stream generation method. The method includes: receiving an input video stream; generating a target feature vector based on multi-dimensional descriptive information of the input video stream, wherein the multi-dimensional descriptive information describes the video content recorded in the input video stream from different dimensions; retrieving candidate memory entries in a first memory bank based on the target feature vector; determining the reconstruction cost generated by reconstructing the candidate memory entries using each encoding method; determining a target encoding method from multiple encoding methods based on the multiple reconstruction costs corresponding to multiple encoding methods; extracting information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream, and performing the target encoding method on the information to be encoded to obtain an encoding result; and outputting the encoding result to a decoding end, wherein the encoding result is used to retrieve memory entries at the decoding end.

Description

Technical Field

[0001] This application relates to the field of video compression and encoding technology, and more specifically, to a video stream processing method and apparatus, and a video stream generation method. Background Technology

[0002] The video compression techniques employed in related technologies are mainly based on redundancy removal in the pixel domain or transform domain, compressing video data through motion compensation, inter-frame prediction, and entropy coding. This method lacks a long-term reuse mechanism for stable semantic structures, leading to the repeated transmission of the same information during video compression and resulting in long compression times.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a video stream processing method and apparatus, and a video stream generation method, to at least solve the technical problem of long processing time and low efficiency caused by the lack of a reuse mechanism for stable semantic structures in the video processing methods used in related technologies.

[0005] According to one aspect of the embodiments of this application, a video stream processing method is provided, comprising: receiving an input video stream; generating a target feature vector based on multi-dimensional description information of the input video stream, wherein the multi-dimensional description information is used to describe the video content recorded in the input video stream from different dimensions; retrieving candidate memory entries in a first memory bank based on the target feature vector, wherein the first memory bank is a database used by an encoder to store multiple memory entries, and the memory entries include video stream reconstruction templates; determining the reconstruction cost generated by performing a reconstruction operation on the candidate memory entries using each encoding method, wherein the reconstruction operation indicates reproducing the input video stream using the candidate memory entries, and the reconstruction cost is determined based on the computing resources consumed by the reconstruction operation and the reconstruction result corresponding to the reconstruction operation; determining a target encoding method from multiple encoding methods based on multiple reconstruction costs corresponding to multiple encoding methods; extracting information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream, and performing the target encoding method on the information to be encoded to obtain an encoding result; and outputting the encoding result to a decoder, wherein the encoding result is used to retrieve memory entries at the decoder.

[0006] Optionally, generating a target feature vector based on the multi-dimensional descriptive information of the input video stream includes: extracting multi-dimensional descriptive information, wherein the multi-dimensional descriptive information includes: structural descriptive information of the target object, appearance descriptive information of the target object, dynamic change descriptive information of the target object, and supplementary descriptive information. The supplementary descriptive information is information in the video content that cannot be covered by structural descriptive information, appearance descriptive information, and dynamic change descriptive information. The target object is an object appearing in the video content. Normalizing each type of multi-dimensional descriptive information to obtain multiple initial vectors; projecting the multiple initial vectors onto the semantic space to obtain multiple projection results, and generating a target feature vector based on the multiple projection results, wherein vectors recording the same semantic content have the same distribution characteristics in the semantic space.

[0007] Optionally, retrieving candidate memory entries from the first memory bank based on the target feature vector includes: for each memory entry, determining the semantic similarity between the query key recorded in the memory entry and the target feature vector, wherein the query key is the feature vector of the video stream matching the memory entry; generating a preliminary candidate set based on memory entries with semantic similarity greater than a preset similarity threshold; and determining candidate memory entries in the preliminary candidate set based on the tag information of the input video stream, wherein the tag information is used to describe the encoding environment and encoding requirements of the input video stream.

[0008] Optionally, candidate memory entries are determined in the preliminary candidate set based on the tag information of the input video stream, including: for each memory entry to be screened in the preliminary candidate set, a consistency check is performed on the metadata recorded in the memory entry to be screened and the tag information of the input video stream, wherein the metadata is used to record the encoding requirements and encoding environment applicable to the memory entry; and the memory entries to be screened that pass the consistency check are determined as candidate memory entries.

[0009] Optionally, determining the reconstruction cost generated by reconstructing candidate memory entries using each encoding method includes: for each encoding method, determining the distortion cost based on the similarity between the reconstructed feature vector generated using candidate memory entries and the target feature vector, wherein the reconstructed feature vector is the feature vector of the reconstructed video stream generated based on the feature information recorded in the candidate memory entries; determining the computational resources consumed in generating the reconstructed feature vector, and determining the computational cost based on the computational resources; determining the transmission resources consumed in transmitting target information during the generation of the reconstructed feature vector, and determining the bitrate cost based on the transmission resources, wherein the target information is determined according to the encoding method; determining the cumulative error generated by reusing candidate memory entries, and determining the drift risk cost based on the cumulative error; determining the storage cost based on the computational resources consumed in updating the first memory bank, wherein updating the first memory bank includes: deleting candidate memory entries; determining the reconstruction cost corresponding to the encoding method based on the label information of the input video stream, the distortion cost, the computational cost, the bitrate cost, the drift risk cost, and the storage cost, wherein the label information of the input video stream is used to determine the weights of each influencing factor of the reconstruction cost.

[0010] Optionally, the target encoding method is determined based on multiple reconstruction costs corresponding to multiple encoding methods, including: finding the minimum reconstruction cost among multiple reconstruction costs; and determining the encoding method corresponding to the minimum reconstruction cost as the target encoding method.

[0011] Optionally, information to be encoded corresponding to the target encoding method is extracted from candidate memory entries and the input video stream, including: when the target encoding method is an index substitution method, extracting the memory index of the candidate memory entry as information to be encoded, wherein the memory index is used to record the version information and video content summary of the candidate memory entry; when the target encoding method is an index plus incremental method, using the memory index and incremental information extracted from the input video stream as information to be encoded, wherein the incremental information is determined based on the error between the target feature vector and the reconstructed feature vector, and the reconstructed feature vector is generated based on the candidate memory entry; when the target encoding method is a full transmission method, extracting the target feature vector of the input video stream as information to be encoded; when the target encoding method is an anchor refresh method, using the keyframe feature vector extracted from the input video stream as information to be encoded, wherein the keyframe feature vector is generated based on the multi-dimensional description information of the keyframe, and the keyframe is the frame whose generation time is closest to the time of determining the target encoding method among the multiple frames contained in the input video stream.

[0012] Optionally, after determining the target encoding method among multiple encoding methods based on the reconstruction cost, the method further includes: when the target encoding method is an index-plus-increment method, verifying the target feature vector using write conditions; if the verification passes, updating candidate memory entries based on the target feature vector, wherein the write conditions include: reconstruction cost verification and storage capacity verification; when the target encoding method is anchor refresh, generating new memory entries based on keyframe feature vectors and label information of the input video stream; verifying the new memory entries using write conditions; if the verification passes, writing the new memory entries into the first memory bank.

[0013] According to another aspect of the embodiments of this application, a method for generating a video stream is also provided, comprising: receiving an encoding result from an encoding end, wherein the encoding result is obtained according to the above-described video stream processing method; parsing the encoding result to obtain information to be encoded; retrieving a target memory entry in a second memory bank according to the information to be encoded; and generating a target video stream according to the target memory entry or the information to be encoded, wherein the target video stream is consistent with the video content recorded in the input video stream received by the encoding end.

[0014] Optionally, generating a target video stream based on a target memory entry or encoding result includes: if the information to be encoded contains a memory index, retrieving a target memory entry based on the memory index; if a target memory entry is retrieved, generating a target video stream based on the target memory entry, wherein the memory index is used to record version information and video content summaries of candidate memory entries, and the candidate memory entries are memory entries used to generate the information to be encoded, and the memory entries include a video stream reconstruction template; if the information to be encoded does not contain a memory index, generating a target video stream based on a target feature vector or keyframe feature vector in the information to be encoded, wherein the target feature vector is generated based on multi-dimensional description information of the input video stream, and the keyframe feature vector is generated based on keyframes of the input video stream, where the keyframe is the frame whose generation time is closest to the time of determining the target encoding method among multiple frames contained in the input video stream, and the target encoding method is the encoding method used to generate the encoding result.

[0015] Optionally, if no target memory entry is found, a backoff mechanism is executed, wherein the backoff mechanism instructs the generation of a target video stream based on incremental information in the information to be encoded, or to pause processing until a new encoding result is received, and to generate a target video stream based on the new encoding result, wherein the incremental information is determined based on the error between the target feature vector and the reconstructed feature vector, and the reconstructed feature vector is generated based on the candidate memory entry.

[0016] Optionally, after obtaining the information to be encoded, the method further includes: when the information to be encoded contains both memory index and incremental information, verifying the incremental information using write conditions; if the verification passes, updating the target memory entry based on the incremental information, wherein the write conditions include: reconstruction cost verification and storage capacity verification; when the information to be encoded only contains keyframe feature vectors, generating a new memory entry based on the keyframe feature vectors and the tag information of the input video stream; verifying the new memory entry using write conditions; if the verification passes, writing the new memory entry into the second memory bank.

[0017] According to another aspect of the embodiments of this application, a video stream processing apparatus is also provided, comprising: a receiving module, configured to receive an input video stream and generate a target feature vector based on multi-dimensional description information of the input video stream, wherein the multi-dimensional description information is used to describe the video content recorded in the input video stream from different dimensions; a retrieval module, configured to retrieve candidate memory entries in a first memory bank based on the target feature vector, wherein the first memory bank is a database used by the encoding end to store multiple memory entries, the memory entries including a video stream reconstruction template, and the memory entries recording the feature vector of the video stream; a first determining module, configured to determine the reconstruction cost generated by reconstructing the candidate memory entries using each encoding method, wherein the reconstruction operation indicates reproducing the input video stream using the candidate memory entries, and the reconstruction cost is determined based on the computing resources consumed by the reconstruction operation and the reconstruction result corresponding to the reconstruction operation; a second determining module, configured to determine a target encoding method from multiple encoding methods based on multiple reconstruction costs corresponding to multiple encoding methods, extract information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream, and execute the target encoding method on the information to be encoded to obtain an encoding result; and an output module, configured to output the encoding result to a decoding end, wherein the encoding result is used to retrieve memory entries at the decoding end.

[0018] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, which stores a computer program, wherein the device where the non-volatile storage medium is located executes the above-described video stream processing method and video stream generation method by running the computer program.

[0019] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to execute the above-described video stream processing method and video stream generation method through the computer program.

[0020] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the steps of the video stream processing method described above, and the steps of the video stream generation method.

[0021] In this embodiment, an input video stream is received, and a target feature vector is generated based on the multi-dimensional description information of the input video stream. The multi-dimensional description information describes the video content recorded in the input video stream from different dimensions. Candidate memory entries are retrieved from a first memory bank based on the target feature vector. The first memory bank is a database used by the encoder to store multiple memory entries, and each memory entry includes a video stream reconstruction template. The reconstruction cost generated by reconstructing the candidate memory entries using each encoding method is determined. The reconstruction operation indicates the reproduction of the input video stream using the candidate memory entries, and the reconstruction cost is determined based on the computational resources consumed by the reconstruction operation and the corresponding reconstruction result. A target encoding method is determined from multiple encoding methods based on the multiple reconstruction costs corresponding to various encoding methods, and the target encoding method is then retrieved from the candidate memory bank. The method involves extracting the information to be encoded corresponding to the target encoding method from the entries and input video stream, executing the target encoding method on the information to be encoded, and obtaining the encoding result. The encoding result is then output to the decoding end. By constructing a memory bank to maintain memory entries that record stable semantic structures, the reusability of memory entries is evaluated based on reconstruction cost during video stream processing. Memory entries with higher reusability are used to complete the encoding of the input video stream, achieving the goal of efficient reuse of stable semantic structures during video stream processing. This reduces redundant transmission during video stream processing, improves video stream processing efficiency, and reduces video stream processing time. In turn, it solves the technical problem of long video processing time and low efficiency caused by the lack of a mechanism for reusing stable semantic structures in the video processing methods used in related technologies. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a video stream processing method according to an embodiment of this application;

[0024] Figure 2 This is a flowchart of a video stream processing method according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the structure of a memory entry according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the selection process of an encoding method according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an update memory according to an embodiment of this application;

[0028] Figure 6 This is a flowchart of a video stream generation method according to an embodiment of this application;

[0029] Figure 7 This is a structural diagram of a video stream processing apparatus according to an embodiment of this application;

[0030] Figure 8 This is an architecture diagram of a system for performing a video stream processing method according to an embodiment of this application;

[0031] Figure 9 This is a schematic diagram illustrating information interaction between an encoding end and a decoding end according to an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0035] Reconstruction cost: In the embodiments of this application, reconstruction cost refers to the resource consumption generated during the process of video reconstruction using candidate memory entries at the encoding end.

[0036] In related technologies, video compression is achieved by removing redundancy in the pixel domain or transform domain, and then compressing video data through motion compensation, inter-frame prediction, and entropy coding. However, even when processing videos containing the same content, this method requires reprocessing and transmitting previously processed video content, leading to the problem of repeated transmission of identical semantic information and limiting the potential for bitrate reduction. Furthermore, generative video compression methods in related technologies reduce bitrate by reconstructing the video from a generative model at the decoding end, based on keyframes or latent representations. However, due to the lack of a mechanism for reusing stable semantic structures, the video reconstructed by the generative model at the decoding end based on keyframes or latent representations often suffers from temporal drift (the content generated by the decoding end deviates from the original content of the input video stream, exhibiting structural offsets, texture distortion, or motion trajectory misalignment). To address this issue, this application provides related solutions, which are detailed below.

[0037] According to an embodiment of this application, a method embodiment for processing a video stream is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing a video stream processing method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0039] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video stream processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned video stream processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0042] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0043] This application provides a video stream processing method that can run under the above-described operating environment. Figure 2 This is a flowchart of the video stream processing method provided in the embodiments of this application, as follows: Figure 2 As shown, the method includes the following steps:

[0044] Step S202: Receive the input video stream and generate a target feature vector based on the multi-dimensional description information of the input video stream, wherein the multi-dimensional description information is used to describe the video content recorded in the input video stream from different dimensions.

[0045] This application provides a method for video reconstruction based on semantic-level (rather than pixel-level) information. In step S202, after receiving the input video stream, the encoding end processes the input video stream frame by frame or segment to extract semantic-level features. In this embodiment, the encoding end uses a neural encoder to extract a high-dimensional latent representation of the input video stream and generates a feature vector (i.e., a target feature vector) based on the high-dimensional latent representation. The high-dimensional latent representation of the input video stream carries multi-dimensional descriptive information of the video content recorded in the input video stream, such as structural information, appearance information, motion information, semantic information, and residual information. This high-dimensional latent representation (i.e., the target feature vector) can be further quantized and entropy-encoded before being transmitted in the bitstream, or it can be used as a conditional input for the decoding end to reconstruct the video stream.

[0046] Optionally, generating a target feature vector based on the multi-dimensional descriptive information of the input video stream includes: extracting multi-dimensional descriptive information, wherein the multi-dimensional descriptive information includes: structural descriptive information of the target object, appearance descriptive information of the target object, dynamic change descriptive information of the target object, and supplementary descriptive information. The supplementary descriptive information is information in the video content that cannot be covered by structural descriptive information, appearance descriptive information, and dynamic change descriptive information. The target object is an object appearing in the video content. Normalizing each type of multi-dimensional descriptive information to obtain multiple initial vectors; projecting the multiple initial vectors onto the semantic space to obtain multiple projection results, and generating a target feature vector based on the multiple projection results, wherein vectors recording the same semantic content have the same distribution characteristics in the semantic space.

[0047] Since the method provided in this application embodiment realizes video reproduction based on semantic-level information, in this embodiment, the multi-dimensional descriptive information extracted by the encoding end from the input video stream for generating the latent representation (i.e., the target feature vector) is semantic information that describes the video content in the input video stream set from different dimensions. Specifically, the encoding end uses a neural network to decouple the video content of the input video stream into four semantic dimensions: structure, appearance, motion, and residual. In this embodiment, the multi-dimensional descriptive information used to generate the potential representation (i.e., target feature vector) of the input video stream includes: structural description information, appearance description information, dynamic change description information, and residual information (i.e., supplementary description information). The structural description information includes geometric information such as background outlines, geometric boundaries, and spatial layout in the video content; the appearance description information includes visual information such as the main color, texture, clothing, and style in the video content; the dynamic change description information is used to describe inter-frame displacement, dynamic trends, and motion changes; and the residual information (i.e., supplementary description information) is used to describe local compensation details that cannot be covered by the three types of information: structural description information, appearance description information, and dynamic change description information. The objects described by this multi-dimensional description information (i.e., target objects) are various objects such as objects and people appearing in the video content recorded in the input video stream.

[0048] The method provided in this application can be processed frame-by-frame, group of images (GOP), or video segment-by-segment. For each unit to be encoded (which can be a single video frame, a group of images, or a video segment), the encoder uses a neural encoder to extract a latent representation (i.e., a target feature vector). This latent representation can be a single vector, a sequence of tokens, or it can be split into different types of latent components. For example, it can be split according to the description dimensions of the multi-dimensional description information used to generate the latent representation (i.e., the target feature vector). If the latent representation of the input video stream is a single vector, after extracting the multi-dimensional description information, each type of multi-dimensional description information is normalized to obtain multiple feature vectors (i.e., initial vectors) for each type of dimension description information, where each dimension of description information is a type of dimension description information. Finally, the multiple initial vectors output by the normalization process are projected onto a shared semantic space. In this semantic space, multiple initial vectors with the same semantic content (such as fixed background, same character appearance, similar action patterns) will converge to a similar region even if they come from different frames. That is to say, vectors that record the same semantic content have the same distribution characteristics in the semantic space. Based on this characteristic of the semantic space, the projection results of multiple initial vectors can be aligned and fused in the semantic space based on semantics to obtain the target feature vector. Therefore, in this embodiment, the target feature vector has semantic consistency across frames and scenes.

[0049] The aforementioned normalization and alignment fusion can specifically employ the following methods: amplitude normalization, temporal alignment, feature whitening, structure / appearance decoupling, semantic projection insensitive to minute pixel perturbations, and robust mapping for slight camera shake, illumination changes, and compression noise. The normalization process in this embodiment aims to improve the comparability and reusability of the target feature vector in encoding decisions. The purpose of normalization is to enable the target feature energy generated after normalization to replace the input video stream in video reconstruction, thereby reducing transmission overhead.

[0050] The method provided in this embodiment robustly normalizes and semantically aligns descriptive information of different semantic dimensions, so that vectors recording the same semantic content are distributed consistently in the feature space. This significantly improves the reusability and retrieval accuracy of semantic feature vectors (target feature vectors) across frames and scenes, and can effectively reduce the bitrate.

[0051] Step S204: Retrieve candidate memory entries in the first memory bank based on the target feature vector. The first memory bank is a database used by the encoding end to store multiple memory entries, and the memory entries include video stream reconstruction templates.

[0052] In step S204, the encoder uses the target feature vector obtained by the method provided in step S202 as the retrieval key to perform multi-level retrieval of matching memory entries in the encoder's hierarchical memory bank (i.e., the first memory bank). The memory entry in the first memory bank that best matches the target feature vector is determined as a candidate memory entry. In this embodiment, each memory entry contains a video stream reconstruction template that can be directly input into the generative decoder. This video stream reconstruction template contains multi-dimensional descriptive information of a type of video content (including: structural condition code, appearance template code, motion prior code, and optional local residual anchor points). When retrieving matching candidate memory entries, the scene label of the input video stream, the version compatibility of each candidate entry, and the reuse risk are also considered for filtering.

[0053] Optionally, retrieving candidate memory entries from the first memory bank based on the target feature vector includes: for each memory entry, determining the semantic similarity between the query key recorded in the memory entry and the target feature vector, wherein the query key is the feature vector of the video stream matching the memory entry; generating a preliminary candidate set based on memory entries with semantic similarity greater than a preset similarity threshold; and determining candidate memory entries in the preliminary candidate set based on the tag information of the input video stream, wherein the tag information is used to describe the encoding environment and encoding requirements of the input video stream.

[0054] Figure 3 This is a structural diagram of memory entries, such as... Figure 3As shown, each memory entry consists of three parts: a query key, feature information (Value), and metadata (Meta). The key is used to retrieve candidate entries and can be a normalized semantic vector of the multi-dimensional semantic information (multi-dimensional descriptive information) of a video stream, a hash signature, a cluster center identifier, or other compact retrieval codes. The value directly serves as the conditional input to the generative decoder used for reconstruction cost prediction. The feature information includes: a structural condition code (structural feature constraint) to constrain the background structure and spatial layout of the generated results; an appearance template code (appearance feature constraint) to constrain the main texture and style; a motion prior code (dynamic constraint) to indicate temporal change trends; optional local residual anchors for more stable detail recovery in key regions; and optional masks or region constraints to limit the applicability of the memory entry. Metadata is used for memory entry management and synchronization. Metadata includes: identification information for each memory entry (such as entry ID), version information for each memory entry (such as entry version number), write time of each memory entry to the memory, most recent hit time of each memory entry (the time when it was most recently identified as a candidate memory entry), hit count of each memory entry (the total number of times it has been identified as a candidate memory entry since it was written to the memory), confidence statistics for each memory entry (the degree of confidence that the memory entry has been successfully reused in historical use and generated high-quality reconstruction results), drift risk statistics for each memory entry (the trend of reconstruction error accumulated by the memory entry during continuous reuse), scenario tag for each memory entry, verification information for each memory entry (such as checksum, hash digest, used to record the content summary and version information of the memory entry), and permission tag for each memory entry. The permission tag is used to record the scope of use of the memory entry (such as usage scenario and access control policy).

[0055] Among the information (Key, Value, and Meta) contained in the memory entries, the entry ID, version number, and verification information form the basis for the synchronization of the two memory banks (the first memory bank and the second memory bank). When sending the memory index, the encoding end must not only indicate "which memory entry to reference," but also let the decoding end know "which version of the memory entry is available, and how to verify its validity." In this embodiment, the preliminary retrieval of candidate memory entries is achieved by matching the target feature vector with the Key contained in the memory entry. Specifically, for each memory entry stored in the shared memory bank (i.e., the first memory bank), the semantic similarity (e.g., cosine similarity) between the Key of each memory entry and the target feature vector is determined, and memory entries with semantic similarity exceeding a preset similarity threshold (e.g., 0.8) are combined into a preliminary candidate set. In the method provided in this application embodiment, the screening of candidate memory entries adopts a two-level screening mechanism of initial screening based on semantic similarity and screening based on scene label constraints. Therefore, after obtaining the initial candidate set, the initial candidate set is further filtered by combining the tag information of the input video stream. Only memory entries that are consistent with the scene semantics of the input video stream, have the permissions, and are not marked as high drift risk are retained as candidate memory entries. Among them, the tag information of the input video stream contains at least two types of information: encoding environment and encoding requirements. The encoding environment is used to describe the context state of the video content and system operation constraints, including: scene type, memory entry version, drift risk, and permission information. The encoding requirements are used to describe the areas in the input video stream that need to be encoded.

[0056] The method provided in this embodiment uses a two-level screening mechanism that combines initial screening based on semantic similarity with final screening based on scene tag constraints to select memory entries in the memory bank that are semantically matched to the video content of the input video stream and are suitable for the current encoding requirements. This avoids introducing irrelevant memory entries into the video reconstruction process, improves the accuracy of memory entry retrieval, and avoids the reuse of irrelevant semantic structures that could lead to reconstruction distortion and bitrate waste.

[0057] According to some optional embodiments of this application, candidate memory entries are determined in a preliminary candidate set based on the tag information of the input video stream, including: for each memory entry to be screened in the preliminary candidate set, performing a consistency check on the metadata recorded in the memory entry to be screened and the tag information of the input video stream, wherein the metadata is used to record the encoding requirements and encoding environment applicable to the memory entry; and determining the memory entries to be screened that pass the consistency check as candidate memory entries.

[0058] As mentioned in the above embodiments, after obtaining a preliminary candidate set based on the semantic similarity between the target feature vector and the Key contained in the memory entry, the tag information of the input video stream is used to filter candidate memory entries in the preliminary candidate set. Specifically, in this embodiment, the tag information of the input video stream records scene tags (used to record the scene of the video content, such as fixed monitoring, video conferencing, remote desktop, etc.), timestamps, privacy permissions, and current reuse risks, etc., encoding environment information. The information recorded in the tag information is used as additional constraints to filter the preliminary candidate set, and only entries that are semantically consistent with the scene of the video stream, have the permissions, and are not marked as high drift risk are retained as final candidate memory entries. Specifically, when using the information recorded in the tag information as additional constraints to filter the preliminary candidate set, a consistency check is performed on the metadata (Meta) in the tag information and each memory entry (i.e., the memory entry to be filtered) in the preliminary candidate set. The consistency check process is as follows: comparing whether the applicable scenario of the memory entry recorded in the Meta is consistent with the scenario recorded in the scene tag in the tag information; comparing whether the version information recorded in the Meta is compatible with the local state (version information of the memory entry maintained by the encoding end) recorded in the tag information; comparing the permissions and access control policies recorded in the Meta with the permission information recorded in the tag information; and comparing whether the encoding area recorded in the Meta covers the encoding area indicated by the encoding requirements recorded in the tag information; and, during the consistency check process, the drift risk recorded in the Meta is also checked; the memory entries that pass the consistency check are determined to be candidate memory entries that are consistent with the scene semantics of the input video stream, have the permissions, and are not marked as high drift risk.

[0059] The method provided in the above embodiments does not employ a simple "similarity threshold filtering" strategy when matching reusable semantic structures (memory entries). Instead, it designs a joint decision mechanism for compressed coding. After similarity threshold filtering, it performs state consistency verification on the metadata of the memory entries and the tags of the video stream to accurately filter out memory entries that both conform to the current coding environment and meet coding requirements. This avoids version mismatch or scene misuse when matching memory entries, which helps improve the accuracy of video reconstruction results.

[0060] Step S206: Determine the reconstruction cost generated by reconstructing the candidate memory entries using each encoding method, wherein the reconstruction operation indicates reproducing the input video stream using the candidate memory entries, and the reconstruction cost is determined based on the computing resources consumed by the reconstruction operation and the reconstruction result corresponding to the reconstruction operation.

[0061] After candidate memory entries are determined using the method provided in step S204, step S206 performs a reuse risk assessment on the candidate memory entries. Specifically, the encoding end simulates and executes multiple encoding methods on the candidate memory entries. The reconstruction cost of each encoding method is jointly determined based on the computational resources consumed in executing each encoding method and the reconstruction result obtained from each encoding method. Specifically, the reconstruction result is used in conjunction with the target feature vector to determine the distortion cost. This distortion cost is a quantitative representation of the error between the reconstruction result and the target feature vector. Each reconstruction result corresponds to a distortion cost, and each encoding method corresponds to a reconstruction result; therefore, each encoding method has a corresponding distortion cost. Further, the reconstruction cost of each encoding method is jointly determined based on the computational resources consumed in executing each encoding method and the distortion cost corresponding to that encoding method. That is, the reconstruction cost B generated by performing a reconstruction operation on a candidate memory entry using each encoding method A is determined jointly based on the distortion cost and computational resources consumed by encoding method A.

[0062] In this embodiment, the encoding method is used to realize video stream reconstruction (also known as video stream reproduction). The result obtained by processing the candidate memory entries according to the encoding method is sent to the decoding end. The encoding result can guide the decoding end to reproduce the input video stream. Therefore, in this embodiment, the operation of processing the candidate memory entries using the encoding method is called reconstruction operation.

[0063] It should be noted that when the above-mentioned encoder simulates the execution of various encoding methods for candidate memory entries, it does not actually perform a complete generative decoding and reconstruction process. Instead, without generating complete pixel frames, it uses the information recorded in the candidate memory entries as conditions to drive the generative model to perform partial forward inference and generate a predictive representation in the latent space. This predictive representation is the reconstruction result. The above-mentioned generative model is a generative decoder built into the encoder with the same structure as the decoder. It is used to simulate the decoder reconstructing the video stream based on the encoding result and is only applied in the process of reconstruction cost evaluation.

[0064] Optionally, determining the reconstruction cost generated by reconstructing candidate memory entries using each encoding method includes: for each encoding method, determining the distortion cost based on the similarity between the reconstructed feature vector generated using candidate memory entries and the target feature vector, wherein the reconstructed feature vector is the feature vector of the reconstructed video stream generated based on the feature information recorded in the candidate memory entries; determining the computational resources consumed in generating the reconstructed feature vector, and determining the computational cost based on the computational resources; determining the transmission resources consumed in transmitting target information during the generation of the reconstructed feature vector, and determining the bitrate cost based on the transmission resources, wherein the target information is determined according to the encoding method; determining the cumulative error generated by reusing candidate memory entries, and determining the drift risk cost based on the cumulative error; determining the storage cost based on the computational resources consumed in updating the first memory bank, wherein updating the first memory bank includes: deleting candidate memory entries; determining the reconstruction cost corresponding to the encoding method based on the label information of the input video stream, the distortion cost, the computational cost, the bitrate cost, the drift risk cost, and the storage cost, wherein the label information of the input video stream is used to determine the weights of each influencing factor of the reconstruction cost.

[0065] Figure 4 This is a flowchart illustrating the process of selecting an encoding method, such as... Figure 4 As shown, after the encoding end obtains the query key (here referring to the target feature vector) for matching and retrieving candidate memory entries through latent representation extraction, normalization, and other operations, it uses the target feature vector to filter candidate memory entries in the shared memory bank (i.e., the first memory bank) of the encoding end. After retrieving a candidate entry, it is not immediately used for replacement, but it needs to be determined whether it has the ability to replace the target feature vector. That is, in the method provided in this embodiment, after obtaining the candidate memory entries, the substitutability evaluation of each encoding method is performed based on the candidate memory entries. The substitutability evaluation of each method is to evaluate the reconstruction cost generated by the complete reconstruction operation of encoding the candidate memory entries with this encoding method, obtaining the encoding result, and completing the video reconstruction based on the encoding result. The reconstruction cost is jointly evaluated by the computing resources consumed by the reconstruction operation and the similarity between the reconstruction result and the target feature vector. Therefore, in this embodiment, the process of evaluating the reconstruction cost can also be called joint cost evaluation.

[0066] Specifically, the methods provided in this application evaluate four encoding methods: full transmission, index replacement, index plus incremental, and anchor refresh. Figure 4As shown, when evaluating each encoding method, the following costs are determined: distortion cost reflecting reconstruction error, bitrate cost reflecting the number of bits required for transmission, computational cost reflecting the computational resource consumption for retrieving, predicting, and generating reconstruction results, storage cost reflecting the computational resource consumption for introducing new memory entries and maintaining the memory bank, and drift risk cost reflecting the risk of error accumulation caused by continuously reusing the same memory entry. A joint cost evaluation is then performed based on these multiple costs. Specifically, during the joint cost evaluation, weights are assigned to each cost (i.e., each influencing factor) according to the tag information of the input video stream (e.g., the scenario is "low-bandwidth monitoring" or "high-latency conference"). The reconstruction cost of the encoding method is evaluated by dynamically weighting these five types of costs.

[0067] The costs for each of the above categories are determined using the following method: When determining the distortion cost, the reconstruction result (i.e., the reconstructed feature vector) obtained by using this encoding method is first predicted using the Value of the candidate memory entry. Then, the distortion cost is determined based on the similarity between the target feature vector and the reconstructed feature vector. For example, the L2 (Euclidean norm distance) or perceptual similarity (such as LPIPS) between the target feature vector and the reconstructed feature vector can be used as the distortion cost. The computational cost is determined by the computational resources consumed in the process of generating the reconstructed feature vector using the candidate memory entries. For example, the number of decoder inference layers and sampling steps required to generate the reconstructed feature vector can be used as the computational cost. The bitrate cost for each encoding method is determined based on the number of bitstream bits (i.e., transmission resources) consumed when transmitting information (i.e., target information) to the decoder using the selected encoding method. When using different encoding methods to reconstruct the video, the objects from which the information to be encoded is extracted are different, resulting in different encoding results (i.e., target information). The drift risk cost is determined based on the number of times the candidate memory entry has been continuously reused in history and the error present in each reuse. The drift risk cost is determined by quantifying the cumulative error trend of the candidate memory entries. The storage cost is incurred when a (first) memory update operation occurs after video reconstruction is performed using the encoding method. In the case of updating the (first) memory, the cost is determined based on the memory bandwidth consumed by updating the memory and the computing resources consumed by index management.

[0068] The method provided in this embodiment comprehensively evaluates each encoding method from multiple dimensions, including reconstruction quality, transmission overhead, computational load, memory drift, and storage resources, and performs joint optimization based on dynamic weighting according to scene labels. This enables the matching of the optimal encoding method to different business needs. For example, in monitoring scenarios, more emphasis can be placed on bitrate and stability, while in video conferencing scenarios, more emphasis can be placed on latency and appearance continuity, thus achieving the technical effect of adapting to different business needs.

[0069] Step S208: Determine the target encoding method from multiple encoding methods based on the multiple reconstruction costs corresponding to multiple encoding methods, extract the information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream, and execute the target encoding method on the information to be encoded to obtain the encoding result.

[0070] In step S208, the optimal encoding method (i.e., the target encoding method) for reconstructing the input video stream is determined based on the reconstruction cost corresponding to each encoding method. This optimal encoding method (i.e., the target encoding method) is then used to extract the information to be transmitted to the decoding end (i.e., the information to be encoded) from the input video stream and / or candidate memory entries. This information to be encoded is then encapsulated (i.e., encoded) into a structured bitstream (i.e., the encoding result) using the target encoding method. Whether the information to be encoded is extracted from the input video stream, from candidate memory entries, or from both, is determined by the target encoding method, and the information to be encoded differs for each encoding method.

[0071] Optionally, the target encoding method is determined based on multiple reconstruction costs corresponding to multiple encoding methods, including: finding the minimum reconstruction cost among multiple reconstruction costs; and determining the encoding method corresponding to the minimum reconstruction cost as the target encoding method.

[0072] Still Figure 4 As shown in the embodiments of this application, the joint evaluation is based on minimizing cost and substitutability. Substitutability can be evaluated using distortion cost and drift risk cost. For multiple encoding methods, when the drift risk cost they generate is lower than the preset drift risk cost, the encoding method with the lowest reconstruction cost among these multiple encoding methods is determined as the encoding method used for the reconstruction operation of the input video stream (i.e., the target encoding method). Otherwise, if there is an encoding method with a drift risk cost greater than the preset drift risk cost, even if its reconstruction cost is the lowest, it cannot be determined as the target encoding method.

[0073] The method provided in this embodiment selects the optimal coding strategy through multi-dimensional reconstruction cost evaluation, achieving a multi-dimensional dynamic balance of bitrate, quality, computing power and stability, and significantly improving the efficiency of generative video coding in complex scenarios.

[0074] According to some optional embodiments of this application, extracting information to be encoded corresponding to the target encoding method from candidate memory entries and the input video stream includes: when the target encoding method is an index substitution method, extracting the memory index of the candidate memory entry as the information to be encoded, wherein the memory index is used to record the version information and video content summary of the candidate memory entry; when the target encoding method is an index plus incremental method, using the memory index and incremental information extracted from the input video stream as the information to be encoded, wherein the incremental information is determined based on the error between the target feature vector and the reconstructed feature vector, and the reconstructed feature vector is generated based on the candidate memory entry; when the target encoding method is a full transmission method, extracting the target feature vector of the input video stream as the information to be encoded; when the target encoding method is an anchor refresh method, extracting keyframes from the input video stream, and using the keyframe feature vector as the information to be encoded, wherein the keyframe feature vector is generated based on the multi-dimensional description information of the keyframe, and the keyframe is the frame whose generation time is closest to the time of determining the target encoding method among the multiple frames contained in the input video stream.

[0075] As mentioned in the above embodiments, the objects to be encoded differ when using different encoding methods to reconstruct the video. Therefore, in this embodiment, the objects to be encoded include candidate memory entries and the input video stream. When the target encoding method is index substitution, the encoding end only sends the memory index of the candidate entries. At this time, the encoding end extracts the memory index of the candidate memory entries as the information to be encoded. The memory index records the ID, version number, and verification summary of the candidate memory entries. That is, the memory index records the version information of the memory entries and the video content summary. The memory index can be used by the decoding end to accurately locate and verify the local memory entries.

[0076] When the index plus increment method is used as the target encoding method, the encoder not only sends the memory index mentioned above, but also extracts and quantizes the residual difference in structure, appearance or motion components between the target feature vector and the reconstruction result (i.e. the reconstructed feature vector). In other words, when the target encoding method is the index plus increment method, the objects to be encoded are candidate memory entries and input video stream. Candidate memory entries are used to provide the memory index, and the input video stream is used to jointly determine the increment information with the candidate memory entries.

[0077] When the target encoding method is full transmission, the encoder only sends the complete latent representation of the input video stream (i.e., the target feature vector) to the decoder. In this case, the object providing the information to be encoded is the input video stream. The encoder directly encodes the normalized complete latent representation of the current frame without relying on any memory content.

[0078] When the target encoding method is anchor refresh, the encoder sends the complete latent representation of the keyframe (i.e., the keyframe feature vector) to the decoder to refresh the state. Here, the encoder determines the frame with the closest temporal distance between the generation time and the decision time from the multiple frames contained in the input video stream as the keyframe, and extracts multi-dimensional descriptive information of the keyframe (including structure, appearance, dynamic information, etc.) to generate the keyframe feature vector. The keyframe feature vector is sent to the decoder as the anchor point for state reset, ensuring that the decoder can recover and reconstruct from a high-quality starting point without accumulated errors. The aforementioned decision time is the moment when the encoder determines the optimal encoding method (i.e., the target encoding method) among multiple encoding methods.

[0079] The method provided in this embodiment extracts the necessary information that matches different encoding methods as the bitstream content, realizing "on-demand encoding and on-demand transmission". This method can maximize the compression of bit rate while ensuring that the video can be reconstructed at the decoding end.

[0080] According to some alternative embodiments of this application, after determining the target encoding method among multiple encoding methods based on the reconstruction cost, the method further includes: when the target encoding method is an index-plus-increment method, verifying the target feature vector using write conditions; if the verification passes, updating the candidate memory entry based on the target feature vector, wherein the write conditions include: reconstruction cost verification and storage capacity verification; when the target encoding method is anchor refresh, generating a new memory entry based on the keyframe feature vector and the tag information of the input video stream; verifying the new memory entry using write conditions; if the verification passes, writing the new memory entry into the first memory bank.

[0081] Figure 5 This is a diagram illustrating the process of updating the memory bank, such as... Figure 5As shown, both the encoding end's memory bank (i.e., the first memory bank) and the decoding end's memory bank (i.e., the second memory bank) adopt a layered structure and are updated according to the same rules. When the first memory bank is updated, after the video reproduction is completed, the decoding end verifies the updated content of the first memory bank. If the verification passes, the updated content of the first memory bank is written into the second memory bank, completing the update of the second memory bank. To achieve a balance between retrieval hit rate, retrieval latency, and storage overhead, the memory bank (first memory bank and second memory bank) in this embodiment adopts a layered structure, including: an instantaneous multiplexing layer, a scene structure layer, and a priori constraint layer. The instantaneous multiplexing layer is used to cache potential memory entries that occur frequently in several frames or several GOPs with the generation time close to the encoding and encapsulation time, serving the fast reuse between adjacent frames. This layer has a small capacity, high update frequency, and high retrieval priority, making it suitable for processing short-term stable regions, slight movements, and repetitive textures. The scene structure layer stores relatively stable background structures, main appearance templates, spatial layouts, and medium- to long-term reusable priors within a scene segment. Its lifecycle is longer than that of the transient reuse layer, making it the primary source of memory entries used when the encoding end employs an index-based encoding method instead of transmission. In scenarios such as fixed-position monitoring, conference room footage, and remote office desktop interfaces, the memory entries stored in this layer typically have high reuse value. As described above, unlike caches that are simply divided by time, the hierarchical memory library in this embodiment considers both the reuse time domain and update strategies.

[0082] In the method provided in this application embodiment, not all video reconstruction operations are updated in the memory bank. Only when other information besides memory entries is introduced during the video reconstruction operation will it be determined whether to update the memory bank. In the above-mentioned encoding method, when the target encoding method only includes the mechanism of sending memory indexes, the memory bank will not be updated after video reconstruction is completed. When the target encoding method is index plus increment, after completing the reconstruction cost evaluation, the encoding end will use write conditions to perform write verification on the target feature vector. Write verification verifies whether the reconstruction cost brought by the candidate memory entries used this time is lower than a preset threshold (i.e., the current reconstruction is stable and the distortion is controllable, avoiding the solidification of high-error content into memory); it also verifies whether the storage capacity of the first memory bank still has spare capacity, and whether the memory layer to which the candidate memory entries belong (such as the scene structure layer) has not exceeded its capacity quota. Only when both verifications pass is it allowed to use the target feature vector to update the candidate memory entries. Simultaneously with updating the candidate memory entries, the index management information of the memory bank is also updated. For example, when a memory entry is replaced by a new memory entry, undergoes significant modification, or its applicable area changes, the entry version number recorded in the index management information will be incremented.

[0083] When the target encoding method is "anchor refresh", the memory is updated using the following method: The encoding end generates new memory entries based on the keyframe feature vector and the tag information of the input video stream (such as scene type, timestamp, and device permissions), and performs the write verification as described above. Only if the new memory entry passes the write verification is it allowed to be written to the scene structure layer or prior constraint layer; otherwise, it is discarded. In addition, the memory can determine whether to delete stored memory entries based on factors such as hit frequency, recent hit time, bitrate gain, drift risk, and capacity budget. For example, memory entries that have not been hit for a long time, have become obviously invalid in the current scene, or have continuously caused reconstruction drift should be downgraded, frozen, or deleted in a timely manner.

[0084] The method provided in this embodiment first uses write conditions to verify the update information when updating the memory. Only the update information that passes the verification is used to update the memory, ensuring that only high-value, low-risk, and reusable content can be written to the shared memory. This achieves the technical effect of preventing error accumulation and improving reconstruction quality, and enables the sustainable maintenance of the memory.

[0085] Step S210: Output the encoding result to the decoding end, wherein the encoding result is used to retrieve the memory entry at the decoding end.

[0086] In step S210, the encoding result output in step S208 is output to the decoding end. The decoding end will complete the reproduction (i.e. reconstruction) of the input video stream under the guidance of the encoding result. The operation performed by the decoding end when reproducing the input video stream varies depending on the information to be encoded encapsulated in the encoding result. If the encoding result encapsulates the index information of the memory entry (i.e., the memory index), the decoding end will retrieve the memory entry in the memory bank (i.e., the second memory bank) of the decoding segment according to the memory index. The index information of the memory entry (i.e., the memory index) records the version information of the memory entry.

[0087] The method provided in this application, when reusing memory entries, sends more than just an entry number. To ensure that the decoding end can correctly interpret the received memory index and perform rollback in case of anomalies, the bitstream (i.e., the encoding result) may include the following fields: frame type information: such as keyframe, ordinary prediction frame, index reuse frame, anchor refresh frame, etc.; memory index ID, used to locate memory entries in shared memory; memory entry version number, used to confirm the memory entry status; memory layer identifier, used to indicate which layer of the memory bank the memory entry originates from; confidence quantification value or control parameter; local region identifier, used to indicate the applicable region of the memory entry; incremental token or local residual; timestamp, frame sequence number or synchronization sequence number; verification field, used to verify the consistency of shared memory; rollback flag, used to indicate what recovery method the decoding end adopts in case of anomalies. This bitstream organization method makes the memory index no longer an ordinary retrieval result number at the decoding end, but a compressed symbol that can truly participate in the encoding and decoding link control.

[0088] Through the above steps, a stable semantic structure reuse mechanism is provided for video reconstruction methods. A database (memory bank) is used to maintain memory entries that record stable semantic results. The optimal memory entry is selected through reconstruction cost evaluation. Video reconstruction is realized based on the optimal memory entry, which avoids the problem of slow video reconstruction speed and low efficiency caused by transmitting duplicate information during the video reconstruction process, and improves the accuracy of video reconstruction results.

[0089] Figure 6 This is a flowchart of a video stream generation method provided according to an embodiment of this application, such as... Figure 6 As shown, the method for generating a video stream includes the following steps:

[0090] Step S602: Receive the encoding result from the encoding end, wherein the encoding result is obtained according to the video stream processing method described above.

[0091] In step S602, the decoding end receives the encoding result sent by the encoding end, which is used to guide the decoding segment in reconstructing the input video stream.

[0092] Step S604: Parse the encoding result to obtain the information to be encoded, retrieve the target memory entry in the second memory bank according to the information to be encoded, and generate the target video stream according to the target memory entry or the information to be encoded. The target video stream is consistent with the video content recorded in the input video stream received by the encoding end.

[0093] In step S604, after receiving the encoding result, the decoding end parses the encoding result to obtain the information it records (i.e., the information to be encoded). This information to be encoded varies depending on the encoding method used to generate the encoding result. The information to be encoded may include the following: memory index, version number, checksum, incremental token, or anchor feature vector (keyframe feature vector), etc. Further, if the information to be encoded contains a memory index, the decoding segment will use the memory index to retrieve a memory entry (i.e., a target memory entry) that matches the information recorded in the memory index in the decoding end's memory bank (i.e., the second memory bank). The target memory entry is used to generate the target video stream. The video content of the target video stream is consistent with the video content of the input video stream in terms of structure, appearance, and timing, but the bitrate may be different. Therefore, even at extremely low bitrates, a stable background and main body shape can still be reproduced, that is, the target video stream and the input video stream are visually and semantically consistent. If the target encoding method corresponding to the encoding result is anchor refresh, the keyframe feature vector in the encoding result is directly used to initialize the reconstruction state.

[0094] According to some optional embodiments of this application, generating a target video stream based on a target memory entry or encoding result includes: when the information to be encoded contains a memory index, retrieving a target memory entry based on the memory index; and when a target memory entry is retrieved, generating a target video stream based on the target memory entry, wherein the memory index is used to record version information and video content summaries of candidate memory entries, and the candidate memory entries are memory entries used to generate the information to be encoded, and the memory entries include a video stream reconstruction template; and when the information to be encoded does not contain a memory index, generating a target video stream based on a target feature vector or keyframe feature vector in the information to be encoded, wherein the target feature vector is generated based on multi-dimensional description information of the input video stream, and the keyframe feature vector is generated based on keyframes of the input video stream, where the keyframe is the frame whose generation time is closest to the time of determining the target encoding method among multiple frames contained in the input video stream, and the target encoding method is the encoding method used to generate the encoding result.

[0095] As mentioned in the above embodiments, when the encoding result includes a memory index, the decoder can retrieve the target memory entry based on the memory index. For example, when the decoder parses the encoding result to obtain the memory index (including entry ID, version number, memory layer identifier, and checksum), it triggers the decoder's local memory to perform entry filtering and verification. Only when the memory entry retrieved by the decoder matches the version number of the candidate memory entry, the checksum matches, and the scene tag of the memory entry matches the scene of the input video stream, is it determined to be the target memory entry. The value of this memory entry (i.e., the video stream reconstruction template composed of structural condition code, appearance template code, motion prior code, etc.) is then input into the generative decoder of the decoder. The generative decoder of the decoder will combine the context to reconstruct the input video stream and obtain the target video stream.

[0096] When the information to be encoded does not contain a memory index (such as when the encoding method for generating the encoding result is full transmission or anchor point refresh), the target feature vector or keyframe feature vector in the encoding result is used as the generation condition to drive the decoder to complete the video reconstruction; in this case, there is no need to rely on the memory library.

[0097] If the target encoding method corresponding to the encoding result is an index-plus-increment method, the target memory entry will also be retrieved. This target memory entry will be fused with the incremental information in the encoding result, and the fused result will be used as the conditional input to the generative model. Based on the above input and combined with historical context, the generative decoder reconstructs the target video stream using generative architectures such as diffusion models. This ensures that the video content of the target video stream is consistent with the video content of the input video stream in terms of structure, appearance, and timing, although the bitrate can be different. Therefore, even at extremely low bitrates, a stable background and subject shape can still be reproduced.

[0098] The method provided in this embodiment dynamically switches the video reconstruction method based on whether a memory index exists in the encoding result. When the memory index is valid, a high-value reconstruction template is reused to save bandwidth. When the memory index is missing, it seamlessly reverts to the anchor point or performs a full reconstruction. This approach can ensure that the encoding and decoding ends maintain video content consistency and reconstruction continuity under any network fluctuations, achieving highly robust generative video restoration at low bitrates.

[0099] Optionally, if no target memory entry is found, a backoff mechanism is executed, wherein the backoff mechanism instructs the generation of a target video stream based on incremental information in the information to be encoded, or to pause processing until a new encoding result is received, and to generate a target video stream based on the new encoding result, wherein the incremental information is determined based on the error between the target feature vector and the reconstructed feature vector, and the reconstructed feature vector is generated based on the candidate memory entry.

[0100] In this embodiment, the decoding end triggers a rollback mechanism when any of the following situations occur: 1) The local memory bank (i.e., the second memory bank) of the decoding end does not contain a memory entry corresponding to the entry ID in the bitstream (i.e., the encoding result); 2) The version information of the memory entry retrieved in the second memory bank that matches the entry ID in the bitstream (i.e., the encoding result) is inconsistent with the version information recorded in the bitstream; 3) Other information verification of the memory entry fails; 4) A rapid quality assessment reveals that the current reconstruction result deviates significantly from expectations; 5) The drift risk cost of the memory entry retrieved according to the memory index is greater than the preset drift risk cost. All of these situations are collectively referred to as "no target memory entry found," meaning the target memory entry is a memory entry that satisfies the retrieval conditions recorded in the memory index and the reconstruction quality verification conditions (points 4 and 5 above).

[0101] The fallback mechanism provided in this application includes: 1) reconstructing the video based solely on the keyframe feature vector and incremental information to obtain the target video stream; 2) calling a low-complexity or low-fidelity emergency model for reconstruction; 3) pausing processing and waiting for the subsequent anchor refresh frame (i.e., the new encoding result) to re-establish synchronization; 4) temporarily prohibiting a certain memory entry from continuing to participate in reuse. Therefore, if the received encoding result carries incremental information (i.e., residual compensation information between the target feature vector and the reconstructed feature vector), the decoder uses this incremental information to locally correct the existing approximate template (such as the previous frame or the default background) to generate the target video stream; if there is no incremental information in the bitstream (such as the original encoding mode being only a pure index), the decoder pauses the reconstruction process of the current frame and waits for the encoding result of the next frame (such as the anchor frame or a new full representation) to arrive. After receiving reconstructable stable reference information, the synchronization state is re-established and continuous decoding is resumed.

[0102] The method provided in this embodiment allows the decoding end to prioritize local repair using the lightweight incremental residual transmitted with the bitstream when the shared memory fails. If this is not available, the decoding end actively pauses and waits for effective anchor point resynchronization, thus avoiding screen tearing or drifting caused by memory mismatch. This ensures the continuity and stability of reconstruction without relying on retransmission, and improves the stability of video reproduction in weak network and asynchronous environments.

[0103] According to some optional embodiments of this application, after obtaining the information to be encoded, the method further includes: when the information to be encoded contains both a memory index and incremental information, verifying the incremental information using write conditions; if the verification passes, updating the target memory entry based on the incremental information, wherein the write conditions include: reconstruction cost verification and storage capacity verification; when the information to be encoded only contains keyframe feature vectors, generating a new memory entry based on the keyframe feature vectors and the tag information of the input video stream; verifying the new memory entry using write conditions; if the verification passes, writing the new memory entry into the second memory bank.

[0104] Similar to the encoding end, when the decoding end updates the local memory bank (i.e., the second memory bank), it also needs to verify the updated content using write conditions. The second memory bank is only updated with the updated content if the verification passes. In this embodiment, if the information to be encoded in the generated encoding result contains both memory index and incremental information, the decoding end will further verify the incremental information after completing the reconstruction using the target memory entry. Specifically, firstly, a reconstruction cost verification is performed: assessing whether the reconstruction cost generated by video reconstruction based on the incremental information is lower than a preset threshold (i.e., the quality is stable after compensation and no abnormal distortion is introduced); secondly, checking whether the storage capacity of the memory entry corresponding to the memory index in the second memory bank allows for updating (e.g., the memory layer has not reached its capacity limit and the entry priority has not been frozen); only when both verifications pass will the incremental information be merged into the Value of the target memory entry, updating its appearance, motion, or residual components, and incrementing the version number to achieve dynamic evolution of the memory content.

[0105] If the encoding result contains only keyframe feature vectors (i.e., the encoding end has performed anchor point refresh), the decoding end generates a new memory entry based on the keyframe feature vectors and the tag information of the input video stream (such as scene type, device ID, and timestamp). The new memory entry is also verified using the write condition. If the write condition verification determines that the new memory entry has high stability, high reuse potential, no privacy-sensitive content, and is within the capacity allowed by the current memory layer, the new memory entry is written to the second memory bank to complete the update of the second memory bank; otherwise, the new memory entry is discarded.

[0106] The method provided in this embodiment ensures that the encoding and decoding ends follow the same rules for updating, thus guaranteeing the synchronization of the memory banks maintained by the encoding and decoding ends. This provides a reliable prerequisite for memory index replacement transmission and avoids decoding errors caused by memory mismatch.

[0107] Through the above steps, the decoding end analyzes the semantic encoding results and links with the local synchronous memory to perform conditional generation and reconstruction, realizing video reproduction (or video reconstruction) that "uses indexes instead of transmission and memory instead of redundancy". This ensures end-to-end video content consistency without relying on the complete pixel stream; while reducing the resources consumed by video reconstruction, it ensures semantic consistency between the reconstructed video and the original video.

[0108] Figure 7 This is a structural diagram of a video stream processing apparatus provided according to an embodiment of this application, such as... Figure 7As shown, the video stream processing device includes: a receiving module 70, used to receive an input video stream and generate a target feature vector based on multi-dimensional description information of the input video stream, wherein the multi-dimensional description information is used to describe the video content recorded in the input video stream from different dimensions; a retrieval module 72, used to retrieve candidate memory entries in a first memory bank based on the target feature vector, wherein the first memory bank is a database used by the encoding end to store multiple memory entries, the memory entries including video stream reconstruction templates, and the memory entries recording the feature vector of the video stream; a first determining module 74, used to determine the reconstruction cost generated by reconstructing the candidate memory entries using each encoding method, wherein the reconstruction operation indicates reproducing the input video stream using the candidate memory entries, and the reconstruction cost is determined based on the computing resources consumed by the reconstruction operation and the reconstruction result corresponding to the reconstruction operation; a second determining module 76, used to determine a target encoding method from multiple encoding methods based on multiple reconstruction costs corresponding to multiple encoding methods, extract the information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream, and execute the target encoding method on the information to be encoded to obtain the encoding result; and an output module 78, used to output the encoding result to the decoding end, wherein the encoding result is used to retrieve memory entries at the decoding end.

[0109] It should be noted that, Figure 7 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.

[0110] Figure 8 This is an architecture diagram of a system for executing a video stream processing method. The video stream processing method provided in this application embodiment can be performed by... Figure 8 The system execution shown is as follows: Figure 8 As shown, the system comprises three parts: an encoding end, a transmission end, and a decoding end. The encoding end analyzes the input video stream, extracts latent representations, retrieves shared memory, selects the encoding mode, and encapsulates the bitstream. Internally, it mainly includes a video preprocessing module, a neural latent encoder, a memory query module, a joint decision module, a quantization and entropy coding module, a bitstream encapsulation module, and an encoding end memory update module. The transmission end is responsible for transmitting keyframes, anchor representations, index information, incremental tokens, version information, verification information, timestamps, etc. Since this invention is applicable to real-time communication or weak network environments, the transmission end can adapt to different reliable transmission or low-latency transmission protocols according to the actual network conditions. The decoding end is responsible for parsing the bitstream, retrieving shared memory, calling the generative decoder for reconstruction, performing drift detection and rollback, and updating the local memory state according to unified rules. Internally, it mainly includes a bitstream parsing module, a shared memory library, a shared memory retrieval module, a generative decoder, a quality monitoring module, a rollback control module, and a decoding end memory update module. Figure 8In the system shown, both the encoder and decoder have a shared memory. However, the shared memory is not mirrored by continuously transmitting all content. Instead, it forms a logically consistent shared state at both ends by uniformly writing rules, version numbers, checksums, and entry eviction policies. Therefore, when the encoder sends an entry index, the decoder can find the generation conditions for the same interpretation locally.

[0111] Figure 9 This is a schematic diagram illustrating the information exchange between the encoding and decoding ends, such as... Figure 9 As shown, depending on the encoding method used at the encoding end, the information to be encoded in the bitstream transmitted to the decoding end also varies. Encoding methods that the encoding end can use include: index transmission (i.e., index substitution), index plus increment, and anchor refresh. When using the index transmission method, the information in the bitstream is a memory index, which records the entry ID, version number, and verification information. When using the index plus increment method, the bitstream contains the memory index and the increment token. When using the anchor refresh method, the bitstream contains the anchor refresh signal (i.e., the keyframe feature vector).

[0112] After receiving the bitstream, the decoder retrieves the corresponding memory entry from the local shared memory bank based on the entry ID and version number. If the retrieval is successful, the version matches, and the verification passes, the entry's value is used as the generation condition, combined with incremental information or keyframes for reconstruction. If the retrieval fails, the version mismatches, or the verification fails, a rollback mechanism is triggered: the current frame is reconstructed using keyframes, incremental tokens, or by waiting for anchor point refresh, and the local memory state is adjusted synchronously. During the reconstruction process, the decoder continuously monitors the stability of the video reconstruction. If it finds that continuous reuse of the same memory entry leads to quality degradation or drift accumulation, the memory entry is marked as high-risk, its usage priority is reduced, or it is frozen. Both the encoder and decoder maintain the shared memory bank according to unified writing rules, version control, verification mechanisms, and eviction policies to ensure logical consistency of the memory state at both ends, thereby guaranteeing the reliability and deployability of index replacement in the real communication link.

[0113] Furthermore, in long video sequences, even if candidate items are available at local moments, continuous reuse can still lead to error accumulation and temporal drift. To address this issue, embodiments of this application provide a drift suppression mechanism, which includes... Figure 9The system consists of two parts: drift detection and drift limit judgment, as follows: First, a continuous reuse limit can be set for each memory entry or each reuse mode. When the continuous reuse reaches the limit, even if a hit is still possible in the short term, anchor point insertion or incremental compensation can be requested. Second, the decoder can continuously monitor temporal consistency indicators, perceived quality indicators, or the degree of potential spatial offset. Once a continuous decline in reconstruction stability is detected, the current memory entry is marked as a high-drift-risk entry. Third, for key areas, such as faces, text, screen edges, or vehicle outlines, local residual compensation can be used to avoid obvious visible distortion caused by generation offset. In addition, for entries that have caused drift multiple times in the past, their retrieval priority can be reduced, or even temporarily frozen for use. Through the above drift suppression mechanism, it is possible to ensure that video reconstruction can meet the requirements of low bitrate while also taking into account the controllability of long-term reconstruction.

[0114] The method provided in this application can be applied to various scenarios, including fixed monitoring, video conferencing, remote desktop, and edge-cloud collaboration. Specifically, applying the method to video streams generated by fixed camera monitoring significantly reduces background repetition because the background in a fixed camera monitoring scenario remains constant over a long period, with only localized movement of people or vehicles. Therefore, the encoding end can write the background structure and spatial layout into the scene structure layer and the recently moving areas into the instantaneous multiplexing layer. For most frames, background-related structures and textures are no longer repeatedly sent; only the corresponding index is sent. For moving target areas, incremental tokens are sent. If there are significant changes in ambient lighting or scene switching, anchor point refresh is sent.

[0115] In video conferencing scenarios, since the camera position is fixed and the conference room background and speaker appearance are highly stable, the encoding end can write the background layout and speaker head and shoulder appearance template into shared memory. The speaker's posture and mouth movements are expressed through local incremental tokens, while the main templates for the background and main appearance are mainly reused through indexing. If the speaker is switched, someone stands up, the camera zooms, or the screen is shared, an entry update or anchor refresh is triggered.

[0116] For remote desktop scenarios, the interface layout, icons, window borders, and toolbars are highly repetitive in remote desktop or cloud office environments. The code side can write these stable user interface elements (UI elements) into the scene structure layer, sending incremental tokens for scrolling text, mouse pointer movements, and changes in local windows. This method avoids the continuous transmission of a large number of repetitive UI structures and is suitable for low-bandwidth office scenarios.

[0117] In edge-cloud collaborative scenarios, the terminal retains only a small-capacity instantaneous multiplexing layer, the edge nodes maintain the scenario structure layer, and the cloud maintains the general prior layer. The encoding end decides whether to access the edge-side memory layer based on the terminal's computing power and network status. This approach is suitable for layered deployments in environments with uneven computing power.

[0118] This application also provides a non-volatile storage medium storing a computer program, wherein the device containing the non-volatile storage medium executes the above-described video stream processing method and video stream generation method by running the computer program.

[0119] The aforementioned non-volatile storage medium is used to store a program that performs the following functions: receiving an input video stream; generating a target feature vector based on multi-dimensional description information of the input video stream, wherein the multi-dimensional description information is used to describe the video content recorded in the input video stream from different dimensions; retrieving candidate memory entries from a first memory bank based on the target feature vector, wherein the first memory bank is a database used by the encoding end to store multiple memory entries, and the memory entries include video stream reconstruction templates; determining the reconstruction cost generated by reconstructing the candidate memory entries using each encoding method, wherein the reconstruction operation indicates reproducing the input video stream using the candidate memory entries, and the reconstruction cost is determined based on the computational resources consumed by the reconstruction operation and the reconstruction result corresponding to the reconstruction operation; determining a target encoding method from multiple encoding methods based on multiple reconstruction costs corresponding to multiple encoding methods; extracting the information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream, and performing the target encoding method on the information to be encoded to obtain the encoding result; and outputting the encoding result to the decoding end, wherein the encoding result is used to retrieve memory entries at the decoding end.

[0120] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to execute the above-described video stream processing method and video stream generation method through the computer program.

[0121] The processor in the aforementioned electronic device is used to run a program that performs the following functions: receiving an input video stream; generating a target feature vector based on multi-dimensional description information of the input video stream, wherein the multi-dimensional description information is used to describe the video content recorded in the input video stream from different dimensions; retrieving candidate memory entries from a first memory bank based on the target feature vector, wherein the first memory bank is a database used by the encoding end to store multiple memory entries, and the memory entries include video stream reconstruction templates; determining the reconstruction cost generated by reconstructing the candidate memory entries using each encoding method, wherein the reconstruction operation indicates reproducing the input video stream using the candidate memory entries, and the reconstruction cost is determined based on the computing resources consumed by the reconstruction operation and the reconstruction result corresponding to the reconstruction operation; determining a target encoding method from multiple encoding methods based on multiple reconstruction costs corresponding to multiple encoding methods; extracting the information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream, and performing the target encoding method on the information to be encoded to obtain the encoding result; and outputting the encoding result to the decoding end, wherein the encoding result is used to retrieve memory entries at the decoding end.

[0122] This application also provides a computer program product, including computer instructions, steps for implementing the above-described video stream processing method when the computer instructions are executed by a processor, and steps for generating a video stream.

[0123] It should be noted that the modules in the aforementioned video stream processing device can be program modules (e.g., a set of program instructions that implement a specific function) or hardware modules. For the latter, they can take the following forms, but are not limited to them: each of the aforementioned modules is represented by a processor, or the functions of each of the aforementioned modules are implemented by a processor.

[0124] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0125] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0130] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing a video stream, characterized in that, include: Receive an input video stream and generate a target feature vector based on the multi-dimensional description information of the input video stream, wherein the multi-dimensional description information is used to describe the video content recorded in the input video stream from different dimensions; Candidate memory entries are retrieved from the first memory bank based on the target feature vector, wherein the first memory bank is a database used by the encoding end to store multiple memory entries, and the memory entries include video stream reconstruction templates; Determine the reconstruction cost generated by reconstructing the candidate memory entries using each encoding method, wherein the reconstruction operation indicates reproducing the input video stream using the candidate memory entries, and the reconstruction cost is determined based on the computing resources consumed by the reconstruction operation and the reconstruction result corresponding to the reconstruction operation; A target encoding method is determined from the multiple encoding methods based on the multiple reconstruction costs corresponding to the multiple encoding methods. Information to be encoded corresponding to the target encoding method is extracted from the candidate memory entries and the input video stream, and the target encoding method is executed on the information to be encoded to obtain the encoding result. The encoding result is output to the decoding end, wherein the encoding result is used to retrieve memory entries at the decoding end.

2. The method according to claim 1, characterized in that, Generate a target feature vector based on the multi-dimensional descriptive information of the input video stream, including: Extract the multi-dimensional description information, wherein the multi-dimensional description information includes: structural description information of the target object, appearance description information of the target object, dynamic change description information of the target object, and supplementary description information. The supplementary description information is information in the video content that cannot be covered by the structural description information, appearance description information, and dynamic change description information. The target object is an object that appears in the video content. The multi-dimensional descriptive information for each category is normalized to obtain multiple initial vectors; Multiple initial vectors are projected onto a semantic space to obtain multiple projection results, and the target feature vector is generated based on the multiple projection results, wherein vectors that record the same semantic content have the same distribution characteristics in the semantic space.

3. The method according to claim 1, characterized in that, Retrieving candidate memory entries from the first memory bank based on the target feature vector includes: For each memory entry, determine the semantic similarity between the query key recorded in the memory entry and the target feature vector, wherein the query key is the feature vector of the video stream that matches the memory entry; A preliminary candidate set is generated based on the memory entries with semantic similarity greater than a preset similarity threshold; The candidate memory entries are determined in the preliminary candidate set based on the tag information of the input video stream, wherein the tag information is used to describe the encoding environment and encoding requirements of the input video stream.

4. The method according to claim 3, characterized in that, Based on the tag information of the input video stream, the candidate memory entries are determined in the preliminary candidate set, including: For each memory entry to be screened in the preliminary candidate set, a consistency check is performed on the metadata recorded in the memory entry to be screened and the tag information of the input video stream, wherein the metadata is used to record the encoding requirements and encoding environment applicable to the memory entry; The memory entries that pass the consistency check are identified as candidate memory entries.

5. The method according to claim 1, characterized in that, Determine the reconstruction cost incurred by reconstructing the candidate memory entries using each encoding method, including: For each of the encoding methods, a distortion cost is determined based on the similarity between the reconstructed feature vector generated using the candidate memory entries and the target feature vector, wherein the reconstructed feature vector is a feature vector of the reconstructed video stream generated based on the feature information recorded in the candidate memory entries; Determine the computational resources consumed in generating the reconstructed feature vector, and determine the computational cost based on the computational resources; The transmission resources consumed in transmitting target information during the generation of the reconstructed feature vector are determined, and the code rate cost is determined based on the transmission resources, wherein the target information is determined according to the encoding method; Determine the cumulative error resulting from reusing the candidate memory entries, and determine the drift risk cost based on the cumulative error; The storage cost is determined based on the computing resources consumed in updating the first memory bank, wherein updating the first memory bank includes: deleting the candidate memory entries; The reconstruction cost corresponding to the encoding method is determined based on the label information of the input video stream, the distortion cost, the computation cost, the bitrate cost, the drift risk cost, and the storage cost, wherein the label information of the input video stream is used to determine the weight of each influencing factor of the reconstruction cost.

6. The method according to claim 5, characterized in that, Determining the target encoding method based on multiple reconstruction costs corresponding to various encoding methods includes: The minimum reconstruction cost among the multiple reconstruction costs; The encoding method corresponding to the minimum reconstruction cost is determined as the target encoding method.

7. The method according to claim 1, characterized in that, Extracting the information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream includes: When the target encoding method is an index substitution method, the memory index of the candidate memory entry is extracted as the information to be encoded, wherein the memory index is used to record the version information and video content summary of the candidate memory entry; When the target encoding method is an index plus incremental method, the memory index and the incremental information extracted from the input video stream are used as the information to be encoded, wherein the incremental information is determined based on the error between the target feature vector and the reconstructed feature vector, and the reconstructed feature vector is generated based on the candidate memory entries; When the target encoding method is the full transmission method, the target feature vector of the input video stream is extracted as the information to be encoded; When the target encoding method is anchor refresh, the keyframes extracted from the input video stream are used as the information to be encoded. The keyframe feature vectors are generated based on the multi-dimensional description information of the keyframes. The keyframe is the frame whose generation time is closest to the time of determining the target encoding method among the multiple frames contained in the input video stream.

8. The method according to claim 7, characterized in that, After determining the target encoding method among the various encoding methods based on the reconstruction cost, the method further includes: When the target encoding method is an index plus incremental method, the target feature vector is verified using write conditions. If the verification passes, the candidate memory entry is updated according to the target feature vector. The write conditions include: reconstruction cost verification and storage capacity verification. When the target encoding method is the anchor point refresh, a new memory entry is generated based on the keyframe feature vector and the tag information of the input video stream; the new memory entry is verified using the write conditions, and if the verification passes, the new memory entry is written into the first memory bank.

9. A method for generating a video stream, characterized in that, include: Receive encoding result from encoding end, wherein the encoding result is obtained according to the video stream processing method according to any one of claims 1 to 8; The encoding result is parsed to obtain the information to be encoded. A target memory entry is retrieved from the second memory bank based on the information to be encoded. A target video stream is generated based on the target memory entry or the information to be encoded. The target video stream is consistent with the video content recorded in the input video stream received by the encoding end.

10. The method according to claim 9, characterized in that, Generating a target video stream based on the target memory entry or the encoding result includes: If the information to be encoded contains a memory index, the target memory entry is retrieved according to the memory index. If the target memory entry is retrieved, the target video stream is generated according to the target memory entry. The memory index is used to record the version information and video content summary of the candidate memory entry. The candidate memory entry is a memory entry used to generate the information to be encoded. The memory entry includes a video stream reconstruction template. If the information to be encoded does not contain the memory index, the target video stream is generated based on the target feature vector or keyframe feature vector in the information to be encoded. The target feature vector is generated based on the multi-dimensional description information of the input video stream, and the keyframe feature vector is generated based on the keyframes of the input video stream. The keyframe is the frame whose generation time is closest to the time of determining the target encoding method among the multiple frames contained in the input video stream. The target encoding method is the encoding method used to generate the encoding result.

11. The method according to claim 10, characterized in that, The method further includes: If the target memory entry is not found, a rollback mechanism is executed, wherein the rollback mechanism instructs the generation of the target video stream based on incremental information in the information to be encoded, or to pause processing until a new encoding result is received, and to generate the target video stream based on the new encoding result, wherein the incremental information is determined based on the error between the target feature vector and the reconstructed feature vector, and the reconstructed feature vector is generated based on the candidate memory entry.

12. The method according to claim 11, characterized in that, After obtaining the information to be encoded, the method further includes: When the information to be encoded contains both the memory index and the incremental information, the incremental information is verified using write conditions. If the verification passes, the target memory entry is updated according to the incremental information. The write conditions include: reconstruction cost verification and storage capacity verification. When the information to be encoded contains only the keyframe feature vector, a new memory entry is generated based on the keyframe feature vector and the tag information of the input video stream; the new memory entry is verified using the writing conditions, and if the verification passes, the new memory entry is written into the second memory bank.

13. A video stream processing apparatus, characterized in that, include: A receiving module is used to receive an input video stream and generate a target feature vector based on the multi-dimensional description information of the input video stream, wherein the multi-dimensional description information is used to describe the video content recorded in the input video stream from different dimensions; The retrieval module is used to retrieve candidate memory entries in a first memory bank based on the target feature vector, wherein the first memory bank is a database used by the encoding end to store multiple memory entries, the memory entries include video stream reconstruction templates, and the memory entries record the feature vectors of the video stream; The first determining module is used to determine the reconstruction cost generated by reconstructing the candidate memory entry using each encoding method, wherein the reconstruction operation indicates reproducing the input video stream using the candidate memory entry, and the reconstruction cost is determined based on the computing resources consumed by the reconstruction operation and the reconstruction result corresponding to the reconstruction operation; The second determining module is used to determine a target encoding method from multiple encoding methods based on multiple reconstruction costs corresponding to multiple encoding methods, extract information to be encoded corresponding to the target encoding method from the candidate memory entries and the input video stream, and execute the target encoding method on the information to be encoded to obtain an encoding result; An output module is used to output the encoding result to a decoding end, wherein the encoding result is used to retrieve a memory entry at the decoding end.

14. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein the device containing the non-volatile storage medium executes the video stream processing method according to any one of claims 1 to 8, and the video stream generation method according to any one of claims 9 to 12, by running the computer program.

15. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute, through the computer program, the video stream processing method according to any one of claims 1 to 8, and the video stream generation method according to any one of claims 9 to 12.

16. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the video stream processing method according to any one of claims 1 to 8, and the steps of the video stream generation method according to any one of claims 9 to 12.

Images