Video transmission method, system, apparatus and non-transitory storage medium

Abstract

The application discloses a video transmission method, system and device and a nonvolatile storage medium. The method comprises the following steps: receiving semantic data set sent by a pushing end and a data magnitude level corresponding to the semantic data set; determining a generation mode according to the data magnitude level, and generating a first target video frame by calling a generation model of a generation end in the generation mode, wherein the generation mode comprises reconstructing the first target video frame according to the semantic data set by calling the generation model, and predicting the first target video frame by calling the generation model with the semantic data set as input; determining a stability evaluation result corresponding to the first target video frame, and adjusting generation parameters of the generation model according to the stability evaluation result; and generating video frames after the first target video frame by calling the generation model after the adjustment. The application solves the technical problem that video transmission may fail in a weak network environment due to the use of a complete video frame transmission mode in the related art.

Description

Technical Field

[0001] This application relates to the field of data transmission, and more specifically, to a video transmission method, system, apparatus, and non-volatile storage medium. Background Technology

[0002] In related technologies, video transmission scenarios such as live streaming often require the transmission of complete video frames. The problem with this approach is that when the network environment is poor, the excessive content contained in a single video frame can lead to poor live streaming quality, or even prevent the receiving end from receiving the live stream, resulting in video transmission failure.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a video transmission method, system, apparatus, and non-volatile storage medium to at least solve the technical problem that video transmission may fail in weak network environments due to the use of transmitting complete video frames in related technologies.

[0005] According to one aspect of the embodiments of this application, a video transmission method is provided, comprising: receiving a semantic data set sent by a push terminal, and a data volume level corresponding to the semantic data set; determining a generation method for a first target video frame based on the data volume level, and calling a generation model of a generation terminal to generate the first target video frame using the generation method, wherein the generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and using the semantic data set as input to call the generation model to predict the first target video frame; determining a stability evaluation result corresponding to the first target video frame, and adjusting the generation parameters of the generation model based on the stability evaluation result; and calling the adjusted generation model to generate video frames after the first target video frame.

[0006] Optionally, the data volume level includes a first data volume level and a second data volume level; determining the generation method of the first target video frame based on the data volume level includes: when the data volume level is the first data volume level, determining the generation method as calling the generation model to predict and generate the first target video frame based on the semantic data set; when the data volume level is the second data volume level, determining the generation method as calling the generation model to reconstruct the first target video frame based on the semantic data set.

[0007] Optionally, the first data volume level includes a first level, a second level, and a third level, wherein the first level indicates that the semantic data set does not contain semantic information, the second level indicates that the semantic data set contains first semantic difference information of key points in the second target video frame relative to corresponding key points in the reference video frame, and the third level indicates that the semantic data set contains second semantic difference information of the second target video frame relative to the reference video frame. Both the second target video frame and the reference video frame are video frames collected by the push terminal, and the collection time of the reference video frame is earlier than the collection time of the second target video frame. The second data volume level includes a fourth level, wherein the fourth level indicates that the semantic data set contains semantic information of the second target video frame.

[0008] Optionally, before calling the generation model on the generation end to generate the first target video frame, the method further includes: determining the identity lock weight of the target object, wherein the identity lock weight is used to impose a continuous identity constraint on the feature space of the generation model; and adding the identity lock weight to the generation model.

[0009] Optionally, calling the generation model on the generation end to generate the first target video frame includes: determining semantic optical flow, wherein the semantic optical flow is used to reflect the semantic position change information of key points in a preset semantic skeleton space; predicting the target semantic position of key points at the target time based on the semantic optical flow; determining the predicted semantic information based on the target semantic position of key points; and generating the first target video frame based on the predicted semantic information.

[0010] Optionally, generating a first target video frame based on predicted semantic information includes: determining the difference information between the predicted semantic information and reference semantic information, wherein the reference semantic information is semantic information in a historical semantic sequence; determining a deviation evaluation index corresponding to the predicted semantic information based on the difference information, wherein the deviation evaluation index is used to indicate the risk probability that the video frame generated based on the predicted semantic information is an abnormal frame, and the larger the deviation evaluation index, the greater the risk probability; and generating a first target video frame based on the predicted semantic information when the deviation evaluation index is not greater than a preset threshold.

[0011] Optionally, determining the stability evaluation result corresponding to the first target video frame includes: determining the motion intensity evaluation result corresponding to the first target video frame; determining the temporal smoothness evaluation result corresponding to the first target video frame; determining the identity consistency evaluation result of the first target video frame; and performing a weighted summation of the motion intensity evaluation result, the temporal smoothness evaluation result, and the identity consistency evaluation result to obtain the stability evaluation result.

[0012] Optionally, determining the identity consistency evaluation result of the first target video frame includes: extracting the identity embedding vector of the first target video frame; determining the similarity between the identity embedding vector and the reference identity embedding vector; and determining the identity consistency evaluation result based on the similarity.

[0013] Optionally, after receiving the semantic data set sent by the push terminal and the data volume level corresponding to the semantic data set, the method further includes: when the semantic data set is compressed into a data packet, determining the data packet template corresponding to the data packet based on the data volume level, wherein the data packet template includes a set of fields of the data packet; and decoding the data packet based on the data packet template to obtain the semantic data set.

[0014] According to another aspect of the embodiments of this application, a video transmission system is also provided, including a push end and a generation end. The push end device is used to determine the semantic data set of the acquired video frames and the data volume level corresponding to the semantic data set. The generation end device is used to receive the semantic data set sent by the push end and the data volume level corresponding to the semantic data set; determine the generation method of a first target video frame based on the data volume level, and call a generation model to generate the first target video frame using the generation method. The generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and calling the generation model to predict and generate the first target video frame based on the semantic data set; determine the stability evaluation result corresponding to the first target video frame, and adjust the generation parameters of the generation model based on the stability evaluation result; and call the adjusted generation model to generate video frames after the first target video frame.

[0015] According to another aspect of the embodiments of this application, a video transmission device is also provided, comprising: a first processing module, configured to receive a semantic data set sent by a push terminal, and a data volume level corresponding to the semantic data set; a second processing module, configured to determine a generation method for a first target video frame based on the data volume level, and call a generation model of a generation terminal to generate the first target video frame using the generation method, wherein the generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and using the semantic data set as input to call the generation model to predict and obtain the first target video frame; a third processing module, configured to determine a stability evaluation result corresponding to the first target video frame, and adjust the generation parameters of the generation model based on the stability evaluation result; and a fourth processing module, configured to call the adjusted generation model to generate video frames after the first target video frame.

[0016] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute a video transmission method when it runs.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of a video transmission method.

[0018] In this embodiment, a semantic data set sent by the receiving push terminal and a corresponding data volume level are used. The generation method of the first target video frame is determined based on the data volume level, and the generation model of the generation terminal is called to generate the first target video frame using the generation method. The generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and using the semantic data set as input to predict the first target video frame. The stability evaluation result corresponding to the first target video frame is determined, and the generation parameters of the generation model are adjusted based on the stability evaluation result. The adjusted generation model is then used to generate subsequent video frames. By receiving the semantic data set sent by the push terminal instead of complete video data, the amount of data that needs to be transmitted during video transmission is reduced, achieving the goal of reducing the amount of data transmitted during live streaming. This achieves the technical effect of data transmission even in weak network environments, thus solving the technical problem of potential transmission failure in weak network environments caused by transmitting complete video frames in related technologies. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 A schematic diagram of the structure of a computer terminal (or mobile device) provided according to an embodiment of this application;

[0021] Figure 2 This is a schematic flowchart of a video transmission method according to an embodiment of this application;

[0022] Figure 3 This is a flowchart illustrating a semantic feature extraction process according to an embodiment of this application;

[0023] Figure 4 This is a flowchart illustrating a semantic level determination process according to an embodiment of this application;

[0024] Figure 5 This is a flowchart illustrating a data packet encoding and decoding process according to an embodiment of this application;

[0025] Figure 6 This is a flowchart illustrating an identity locking and verification process provided according to an embodiment of this application;

[0026] Figure 7 This is a flowchart illustrating a semantic prediction process according to an embodiment of this application;

[0027] Figure 8 This is a flowchart illustrating a stability assessment feedback process provided according to an embodiment of this application;

[0028] Figure 9 This is a flowchart illustrating a computing power scheduling process according to an embodiment of this application;

[0029] Figure 10 This is a flowchart illustrating a video transmission process according to an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of the structure of a video transmission system according to an embodiment of this application;

[0031] Figure 12 This is a schematic diagram of the structure of a video transmission device according to an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] While live video streaming of individuals is now widely used in various scenarios, most related technologies employ pixel-level encoding and transmission schemes. However, pixel-level encoding and transmission schemes suffer from strong bandwidth dependence in weak network environments, are prone to image anomalies under high packet loss, and have uncontrollable degradation mechanisms. Furthermore, while generative live streaming schemes can reduce the amount of data transmitted, they face limitations such as difficulty in ensuring the long-term consistency of the individual's identity and the sensitivity of the generated effect to the completeness of the input. Simultaneously, the overall technology lacks system-level closed-loop control capabilities for transmission semantic judgment, transmission and generation linkage feedback, and unified quantitative evaluation, making it difficult to achieve stable and reliable live streaming of individuals even in extremely weak network conditions.

[0035] To address this issue, relevant solutions are provided in the embodiments of this application, which are described in detail below.

[0036] According to an embodiment of this application, a method embodiment for video transmission is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a video transmission method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video transmission method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned video transmission method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0041] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0042] Under the aforementioned operating environment, this application provides a video transmission method applicable to receiving or generating devices during video transmission, such as... Figure 2 As shown, the method includes the following steps:

[0043] Step S202: Receive the semantic data set sent by the push terminal, and the data volume level corresponding to the semantic data set;

[0044] In some embodiments of this application, such as Figure 3 As shown, the push terminal can use time-series video frames. As input, output a discretized, differenceable semantic state. It includes the following steps:

[0045] Step 1: Multidimensional semantic skeleton feature parsing;

[0046] For the current video frame (That is, the second target video frame) performs semantic parsing, extracts multi-dimensional skeleton features strongly correlated with character generation, and uniformly represents them as a semantic state vector:

[0047]

[0048] in:

[0049] : Facial geometric skeleton feature set (such as the two-dimensional or three-dimensional coordinates of 468 facial key points);

[0050] Human skeletal structure (such as the head, torso, and key joints of the limbs);

[0051] : Expression-driven vectors (such as eyebrow and eye opening and closing, mouth shape parameters, etc.);

[0052] Semantic-level auxiliary prompts (such as lip shape status, semantic labels, or generation constraint identifiers).

[0053] Optionally, since the generative model at the generation end does not require complete pixel information, but relies more on a combination of geometric structure, motion trends, and semantic constraints to generate images, it is possible to extract only the relatively important information from the video frames, such as facial geometric skeleton feature sets, human pose skeletons, expression driving vectors, and semantic-level auxiliary cues.

[0054] Step 2, calculate the semantic changes across frames;

[0055] Optionally, to determine which skeletal key points (i.e., the aforementioned key points) in the facial geometric skeleton, human pose skeleton, etc., have higher transmission value, for each skeletal key point... The variation relative to the reference video frame can be calculated using the following formula:

[0056]

[0057] in This represents the Euclidean distance or weighted geometric distance between two skeleton keypoints. In the formula above, Indicates the relationship between the skeleton keypoints in the reference video frame. The corresponding skeleton keypoints are the same point on the target object (including the human body) within the video frame. The aforementioned reference video frame can be an adjacent video frame to the current video frame (the second target video frame), or a video frame captured before the second target video frame that can serve as a reference object.

[0058] The variation amplitude calculated using the above formula can be used to measure whether the skeletal keypoints in an emotional video frame participated in effective action, and whether they belong to static or redundant regions, such as facial edge points that remain stationary for a long time. A larger variation amplitude indicates a higher probability that the skeletal keypoint participated in effective action. A smaller variation amplitude indicates a higher probability that the skeletal keypoint belongs to a static or redundant region.

[0059] Step 3, Dynamic attention weight modeling;

[0060] To address the dynamic importance of different keypoints in the current frame, an attention weight distribution based on the intensity of change can be constructed:

[0061]

[0062] Softmax is computed independently within the same semantic subspace (such as face or pose skeleton), ensuring that attention weights are comparable and normalized.

[0063] By using the above attention weight distribution, high dynamic areas (such as the mouth, eyes, and hands) can be automatically amplified; low dynamic and long-term unchanging points can be suppressed; and different anchors and different action intensities can be adapted without the need for manual rules.

[0064] Step 4: Attention threshold screening and Top-K key point selection;

[0065] To further reduce the transmission size, we can select to include a skeleton keypoint in the transmission candidate set only if the skeleton keypoint meets the attention threshold condition. The selection method can be expressed as the following formula:

[0066]

[0067] Or, in high dynamic scenarios, select directly:

[0068]

[0069] in:

[0070] For adaptive threshold, Typically, the size is 20 to 40, which is much smaller than the size of a complete skeleton. The settings can be determined by the push terminal based on factors such as the actual network communication status and the variation amplitude of the current frame, or they can be preset fixed values. Considering that transmitting all semantic information in a video frame is unsustainable in a weak network environment, by selecting transmittable skeleton key points, the transmitted data can be compressed on the one hand, and the key information can be guaranteed to be included in the transmitted data as much as possible on the other hand.

[0071] Step 5: Semantic difference calculation and adaptive quantization;

[0072] For the filtered set of key points, only their relative to the previous reference state can be transmitted. Difference components:

[0073]

[0074] To further reduce bandwidth consumption, adaptive quantization can also be performed on the differential values:

[0075]

[0076] Among them, quantization accuracy Related to attention weights:

[0077]

[0078] In other words: high attention points → high precision quantization; low attention points → low precision quantization. This approach ensures that the most visually sensitive areas are preserved with high fidelity even at extremely low bitrates.

[0079] It can be seen that the push terminal can stably map high-dimensional video frames into low-dimensional, controllable, and differential semantic skeleton states. By utilizing the attention weighting mechanism, cross-frame adaptive semantic sparsity is achieved, reducing bandwidth requirements from the source.

[0080] In some embodiments of this application, the data volume level includes a first data volume level and a second data volume level. The first data volume level includes a first level, a second level, and a third level, wherein the first level indicates that the semantic data set does not contain semantic information; the second level indicates that the semantic data set contains first semantic difference information between key points in the second target video frame and corresponding key points in a reference video frame; the third level indicates that the semantic data set contains second semantic difference information between the second target video frame and the reference video frame; both the second target video frame and the reference video frame are video frames captured by the push terminal, and the capture time of the reference video frame is earlier than the capture time of the second target video frame. The second data volume level includes a fourth level, wherein the fourth level indicates that the semantic data set contains semantic information of the second target video frame.

[0081] Optional, such as Figure 4 As shown, it can be applied to the current semantic state. Compared with the previous reference state Determine the semantic transmission level of the current semantic state. And the corresponding data packet control instructions, thereby determining whether to transmit the complete current semantic state, or only the semantic difference components (first semantic difference information) of the filtered skeleton key points, or the current semantic state. Compared with the previous reference state The semantic difference component (second semantic difference information). The semantic transmission level is the same as the data volume level mentioned above. It includes the following steps:

[0082] Step 1: Constructing a semantic distance metric;

[0083] To avoid directly relying on pixel or model-internal features, a cross-frame semantic distance function can be defined. This is used to quantify the strength of the change in the current semantic state relative to the reference state.

[0084]

[0085] in:

[0086] For facial semantic skeleton, For posture skeleton, : Facial expression vectors; These are configurable weights used to balance the importance of different semantic dimensions.

[0087] Considering that generative models have varying sensitivities to different semantic changes—for example, changes in mouth expressions have a much greater impact on perceptual quality than slight body swaying—the aforementioned distance function provides a more practical and adjustable quantization basis for subsequent state decisions.

[0088] Step 2: Establishment of a multi-level semantic decision threshold system;

[0089] Based on the controllability requirements of transmission in weak network scenarios, a set of strictly ordered semantic change thresholds can be predefined:

[0090]

[0091] Among them, it is lower than Changes are considered "semantic silence," exceeding The change indicates that it is not silent but insufficient to trigger a full difference, exceeding... Significant changes in the semantic structure necessitate the reconstruction of the reference state. These thresholds are not fixed constants and can be dynamically fine-tuned by the push client based on historical packet loss rates, RTT, or STScore feedback, while maintaining their hierarchical relationship to ensure the stability of the state machine.

[0092] Step 3: Decision using the four-level semantic differential state machine;

[0093] Optionally, it can be based on the current semantic distance. This maps the system state to four mutually exclusive semantic transport levels.

[0094] Level 0 (First Level): Semantic Silence Mode

[0095] Judgment conditions:

[0096]

[0097] The corresponding approach is to not send any semantic differential data, but only periodically send heartbeat packets and timestamps to maintain timing synchronization. This completely eliminates redundant transmissions when the broadcaster is stationary or undergoes minimal changes, freeing up bandwidth for weak networks.

[0098] Level 1 (Second Level): Sparse Semantic Difference Pattern

[0099] Judgment conditions:

[0100]

[0101] The corresponding processing method is to only send the filtered Top-K high-attention key difference and not update the reference semantic state. This allows for maintaining the continuity of actions at the generation end with minimal cost, even in scenarios involving micro-expressions or localized movements.

[0102] Level 2 (Third Level): Dense Semantic Difference Patterns (Dense Delta)

[0103] Triggering conditions:

[0104]

[0105] The corresponding processing method is as follows: send the complete skeleton difference set within the current semantic subspace; the difference data is quantized and compressed; and some reference state caches can be selectively refreshed. This can prevent error accumulation from causing generation drift when the action is constantly changing but no semantic jump occurs.

[0106] Level 3 (Fourth Level): Key Semantic Frame

[0107] Triggering conditions:

[0108]

[0109] The corresponding processing method is: send the complete semantic skeleton state. Simultaneously, identity calibration, generation constraint, or LoRA refresh trigger flags are sent. The constraint or LoRA refresh trigger flag is used to update the identity constraint weights in the generative model, ensuring that the appearance of the target object does not change significantly during video transmission. Then... Updated to the new reference state This allows the entire system to re-anchor a consistency benchmark when significant changes occur in semantic structure (such as turning the head or making large facial expressions).

[0110] Step 4: State transition suppression mechanism for weak network sensing;

[0111] To prevent frequent oscillations of the state machine under high packet loss conditions, a state transition suppression window can be introduced:

[0112]

[0113] That is, between any two state upgrades (such as Level 1 → Level 2), a minimum time interval must be satisfied:

[0114]

[0115] The purpose of this mechanism is to avoid triggering unnecessary high-level transmissions due to single-frame noise or short-term jitter, thereby ensuring overall stability under weak network conditions.

[0116] As an optional implementation, after receiving the semantic data set sent by the push terminal and the data volume level corresponding to the semantic data set, the method further includes: when the semantic data set is compressed into a data packet, determining the data packet template corresponding to the data packet based on the data volume level, wherein the data packet template includes a set of fields of the data packet; and decoding the data packet based on the data packet template to obtain the semantic data set.

[0117] In some embodiments of this application, the encoding process at the push end and the decoding process at the generation end are as follows: Figure 5 As shown, it includes the following steps:

[0118] Step 1: Semantic Transport Level-Driven Packet Template Selection;

[0119] The push device will determine the semantic level of the current video frame. :

[0120]

[0121] Based on the level, the corresponding data packet template is selected to ensure that the data packet field sets of different levels are strictly controlled, low-level packets do not carry redundant information of high-level packets, and the decoding generator can complete the fast parsing branch based solely on the Mode field during decoding.

[0122] Step 2: Packet encapsulation;

[0123] All semantic data packets use the following basic structure:

[0124] | Header | Timestamp | Mode | Payload | The meanings of the fields are as follows:

[0125] Header (2 Bytes): Packet type identifier and version number, used for protocol compatibility management; Timestamp (4 Bytes): Semantic timestamp, used for out-of-order reordering and time alignment; Mode (1 Byte): Semantic differential level identifier, corresponding to Level 0 to Level 3; Payload (Variable): Semantic data payload strongly correlated with Mode.

[0126] Optionally, the payload structure for each semantic data packet level is as follows:

[0127] (1) Level 0: Semantic Silence Package

[0128] Payload structure:

[0129] | Heartbeat |

[0130] It does not contain any semantic skeleton data and is only used to maintain connections and time synchronization. The data volume typically meets the following requirements:

[0131]

[0132] (2) Level 1: Sparse Semantic Differential Packet

[0133] Payload structure: |TopK_Index_List|Quantized_Delta|, where: TopK_Index_List:

[0134] Records the Top-K keypoint indices filtered out by module one; Quantized_Delta: the quantization difference value of the corresponding keypoint. The data volume range is approximately:

[0135]

[0136] (3) Level 2: Dense Semantic Differential Packet

[0137] Payload structure: | Full_Semantic_Index | Quantized_Delta |

[0138] The data packets at this level contain information that covers the complete skeleton set of the current semantic subspace, and do not carry identity or reference state reset information.

[0139] The data volume range is approximately:

[0140]

[0141] (4) Level 3: Key semantic frame packets

[0142] Payload structure:

[0143] | Full_Semantic_Data | Reference_Reset_Flag | Identity_Calibration_Flag |

[0144] This level of data packet carries the complete semantic state (S_t), explicitly declares reference state updates, and can trigger identity calibration or LoRA refresh.

[0145] The data volume range is approximately:

[0146]

[0147] Optionally, during encoding, fixed-point quantization encoding is uniformly used for all differential data involved in Level 1 to Level 3:

[0148]

[0149] in, Original semantic difference value; The quantization step size is dynamically determined by the attention weights in Module 1. The quantized data is encoded using Int16 or Int8, significantly reducing the transmission size.

[0150] Step 3: Decoding and semantic reconstruction at the generation end;

[0151] After receiving the semantic packets, the decoder rearranges them according to the Timestamp, then parses the Mode field and selects the corresponding parsing path. It then performs inverse quantization on the differential data.

[0152]

[0153] The difference is then superimposed onto the current reference semantic state. This yields the semantic state used to predict or reconstruct the first target video frame. For Level 3 packets, the semantic state carried in the packet is used to update... .

[0154] The above process ensures that even if some Level 1 / Level 2 packets are lost in the middle, the image transmission process can still automatically converge as long as Level 3 packets arrive periodically.

[0155] As an optional implementation, the generation process is not forcibly blocked in the following cases:

[0156] Level 1 / Level 2 packets are missing; timestamps are slightly out of order; payload is incomplete but headers are resolvable.

[0157] At this point, the generator will trigger a semantic prediction compensation process, or maintain the previous stable semantic state. Simultaneously, it will report the weak network status to the push server for subsequent level determination.

[0158] Step S204: Determine the generation method of the first target video frame based on the data volume level, and call the generation model of the generation end to generate the first target video frame using the generation method. The generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and calling the generation model to predict the first target video frame using the semantic data set as input.

[0159] The method for generating the first target video frame based on the data volume level includes: when the data volume level is the first data volume level, the generation method is to call the generation model to predict and generate the first target video frame based on the semantic data set; when the data volume level is the second data volume level, the generation method is to call the generation model to reconstruct the first target video frame based on the semantic data set.

[0160] As an optional implementation, before calling the generation model of the generation end to generate the first target video frame, the method further includes: determining the identity lock weight of the target object, wherein the identity lock weight is used to impose a continuous identity constraint on the feature space of the generation model; and adding the identity lock weight to the generation model.

[0161] Optionally, LoRA weights can be used as an explicit Identity Lock embedded in the generative model, and an identity consistency metric independent of the generative model can be introduced. Additionally, the identity consistency score can be used as a feedback signal to drive the semantic state machine and adjust the generation parameters. The complete process is as follows: Figure 6 As shown, it includes the following steps:

[0162] Step 1: Live Session-Level LoRA Identity Lock Injection

[0163] During the initialization phase of the live stream session, a unique identity LoRA weight set is assigned to each streamer. Its size is typically controlled within the range of 1–3 MB. LoRA weights are injected into the generative model. In the specified substructure, such as: intermediate layer Cross-Attention, local Self-Attention, or identity-sensitive feature projection layer.

[0164] The injected generative model can be represented as:

[0165]

[0166] It is important to note that LoRA is not a simple style transfer, but rather imposes a persistent identity constraint on the feature space of the generative model with a very low number of parameters, without compromising the original model's temporal generation capabilities.

[0167] Step 2: Generate frame identity feature encoding;

[0168] For each frame of image output from the generator Through a lightweight facial identity encoder Extract its identity embedding vector:

[0169]

[0170] Meanwhile, the generator has already cached the broadcaster's reference identity vector at the start of the session:

[0171]

[0172] in These can be standard face images from the registration phase or historical high-confidence keyframes. This approach decouples identity consistency assessment from the generation model, avoids model self-verification, and thus ensures the objectivity of the feedback signal.

[0173] Step 3: Calculate the identity consistency score (IdScore);

[0174] An identity consistency score is defined based on the similarity between the generated frame and the reference identity vector:

[0175]

[0176] Where CosSim is the cosine similarity function, and its value range is... It can also be normalized to .

[0177] This metric possesses the following engineering characteristics: it is independent of generation resolution and semantic transmission level; it is naturally robust to changes in expression and posture; and it is highly sensitive to identity drift and facial distortion.

[0178] Step 4: Identity consistency threshold determination;

[0179] The generation end presets an identity consistency security threshold. The identity lock is considered at risk of failure when the following conditions are met:

[0180]

[0181] threshold It can be configured according to the live streaming scenario and the type of generated model, but remains stable within a single session to avoid frequent triggering. This decision does not directly interrupt generation, but rather serves as a trigger source for closed-loop control, entering the subsequent adjustment process.

[0182] Step 5: Identity closed-loop control triggering and linkage adjustment;

[0183] When the identity consistency failure condition is met, the following set of linked operations can be triggered in order of priority:

[0184] LoRA refresh trigger: Re-inject the currently generated model. Or increase its injection strength coefficient:

[0185] Semantic state machine upgrade request: Request the push client to enter Level 3 (critical semantic frame mode) to obtain a new high-confidence semantic reference state.

[0186] Generation constraint weight adjustment: Increase the identity-guidance weight during the generation process to suppress randomness diffusion. These actions are not executed blindly in parallel, but rather based on the current... The network status and the most recent recovery effect are selectively triggered to avoid generation-end oscillations.

[0187] Step 6: Identity restoration confirmation and closed-loop convergence;

[0188] In subsequent frames after the adjustment is triggered, the generator continuously monitors the identity consistency score, and when the following conditions are met:

[0189]

[0190] Once the identity lock is successfully restored, the generating end will:

[0191] The temporary enhanced LoRA weights are removed, allowing the semantic state machine to degrade to normal operating level, and the current stable frame is used as a new high-confidence reference candidate. This mechanism ensures that closed-loop control has triggering, feedback, and exit conditions, avoiding long-term over-constraint on generation quality.

[0192] In some embodiments of this application, the step of calling the generation model of the generation end to generate the first target video frame includes: determining semantic optical flow, wherein the semantic optical flow is used to reflect the semantic position change information of key points in a preset semantic skeleton space; predicting the target semantic position of key points at the target time based on the semantic optical flow; determining the predicted semantic information based on the target semantic position of key points; and generating the first target video frame based on the predicted semantic information.

[0193] Optionally, if the semantic data set does not contain semantic information, the semantic optical flow of key points can be determined based on the historical semantic sequence. If the semantic data set contains semantic difference information, the semantic optical flow can be determined based on the semantic difference information, which is used to reflect the semantic differences between the second target video frame and the reference video frame. The historical semantic sequence includes the semantic information of video frames generated by a multi-frame generation model arranged in time sequence.

[0194] Optionally, the step of generating a first target video frame based on the predicted semantic information includes: determining the difference information between the predicted semantic information and the reference semantic information, wherein the reference semantic information is the semantic information in the historical semantic sequence; determining the deviation evaluation index corresponding to the predicted semantic information based on the difference information, wherein the deviation evaluation index is used to indicate the risk probability that the video frame generated based on the predicted semantic information is an abnormal frame, and the larger the deviation evaluation index, the greater the risk probability; and generating the first target video frame based on the predicted semantic information when the deviation evaluation index is not greater than a preset threshold.

[0195] Optionally, in a weak network environment, when semantic differential data is delayed, lost, or determined to be of a low level (such as Level 0 / Level 1) by Module 2 and not sent, the generator can use historical semantic skeleton states to predict future states in the semantic space, thereby maintaining the continuity and stability of the generation-driven process. The specific prediction process is as follows: Figure 7 As shown, it includes the following steps:

[0196] Step 1: Building a historical semantic skeleton cache;

[0197] The generator can maintain a semantic skeleton cache queue of the most recent frames:

[0198]

[0199] The semantic state of each frame comes from one of the following three sources:

[0200] The normally received semantic differential reconstruction results, Level 3 key semantic frames, and predicted semantic states are used to model the system. This ensures that even when semantic packets are missing for extended periods, the system still has a continuous semantic history available for modeling.

[0201] Step 2: Define the semantic skeleton optical flow;

[0202] In the semantic skeleton space, for each key point Define its first-order semantic optical flow:

[0203]

[0204] in , These represent the semantic positions of the key point in the two historical frames, respectively.

[0205] Unlike pixel optical flow, this optical flow directly reflects the movement trend itself and is not affected by rendering noise, lighting changes, or other factors.

[0206] Step 3: Next-moment semantic skeleton prediction;

[0207] Based on the calculated semantic optical flow, the skeleton state at the next time step is linearly predicted:

[0208]

[0209] By combining the prediction results of all key points, we obtain the predicted semantic state:

[0210]

[0211] Optionally, the above predictions are typically used only within a short time window to avoid long-term cumulative errors.

[0212] Step 4: Assess the reliability of the prediction results;

[0213] To avoid misusing prediction results in scenarios with sudden action changes, a bias evaluation can be introduced into the predicted semantic state:

[0214]

[0215] When the following conditions are met:

[0216]

[0217] If the current prediction is considered to be within an acceptable range and can be used as a generation driver, then the prediction risk is deemed too high, and the prediction is paused while waiting for new real semantic input or requesting a Level 3 keyframe.

[0218] In some embodiments of this application, the triggering conditions for the above semantic prediction process include: the current The continuous semantic packet loss time exceeds the set threshold. It was determined that there was a risk to the generation stability, but it was not immediately upgraded to Level 3.

[0219] Under the above conditions, the generator will prioritize using... Replace the missing true semantic state.

[0220] Step 6: Linking semantic prediction with generation stability constraints;

[0221] Predicting semantic states is not used unconditionally; its output must be subject to stability constraints.

[0222] When the generated stability index satisfies:

[0223]

[0224] Video frames can be generated based on predicted semantics, where the formula above... This represents the ascension stability index evaluated at the generation end. This is a preset threshold.

[0225] When the following conditions are met:

[0226]

[0227] Even if semantics are still missing, it will limit the prediction range or directly freeze the semantic state:

[0228]

[0229] This ensures that no uncontrollable jitter or deformation is introduced.

[0230] Step 7: Reference state update strategy for predicted semantics;

[0231] In prediction mode, the generator will not immediately... Upgraded to reference status Updates are only allowed if all of the following conditions are met:

[0232]

[0233] This design avoids contaminating long-term semantic benchmarks with prediction results.

[0234] Step S206: Determine the stability evaluation result corresponding to the first target video frame, and adjust the generation parameters of the generation model based on the stability evaluation result;

[0235] In the technical solution provided in step S206, determining the stability evaluation result corresponding to the first target video frame includes: determining the motion intensity evaluation result corresponding to the first target video frame; determining the temporal smoothness evaluation result corresponding to the first target video frame; determining the identity consistency evaluation result of the first target video frame; and performing a weighted summation of the motion intensity evaluation result, the temporal smoothness evaluation result, and the identity consistency evaluation result to obtain the stability evaluation result.

[0236] In some embodiments of this application, determining the identity consistency evaluation result of the first target video frame includes: extracting the identity embedding vector of the first target video frame; determining the similarity between the identity embedding vector and a reference identity embedding vector; and determining the identity consistency evaluation result based on the similarity. The specific process for determining the identity consistency evaluation result can be referred to... Figure 6 The relevant descriptions will not be repeated here.

[0237] In some embodiments of this application, the following are also provided: Figure 8 The flowchart shown is a diagram of the stability assessment feedback process. Figure 8 As can be seen from this, the process includes the following steps:

[0238] Step 1: Motion intensity assessment (MotionScore calculation);

[0239] To determine the temporal consistency constraint strength required for the currently generated frame, the semantic-level action strength index is first calculated:

[0240]

[0241] Optionally, semantic information can be computed only for the skeleton and expression subspace:

[0242]

[0243] This indicator is used to distinguish between low-motion scenarios (such as static speaking) and high-motion scenarios (such as turning the head, making large facial expressions, or body movements).

[0244] Step 2: Time smoothness assessment (Smoothness calculation);

[0245] To quantify the consistency trend of the generated results over time, a time smoothness index can be introduced:

[0246]

[0247] in This represents a temporal difference metric based on optical flow, temporal features, or low-dimensional generated features. Direct pixel comparison is easily affected by noise, while temporal features or low-frequency projection can more accurately reflect whether a visual jump has occurred.

[0248] Step 3: Jointly generate the stability metric STScore;

[0249] A unified generation stability metric is defined among identity consistency, action intensity, and temporal smoothness:

[0250]

[0251] in: These are the weighting coefficients. The maximum value is usually taken to ensure the highest priority for the identity.

[0252] This metric possesses the following engineering characteristics: it does not depend on whether the semantic packet has arrived, it can be calculated independently at the generation end, and it has natural fault tolerance for short-term semantic missing.

[0253] Step 4: Generate stability threshold decision;

[0254] The system defines and generates stability and security thresholds. When the following conditions are met:

[0255]

[0256] This means that the current generation process was determined to have a risk of temporal instability, requiring intervention and adjustment. This judgment is consistent with... Figure 5 The identity threshold decisions are independent of each other, but can be triggered simultaneously, thus forming a multi-indicator coordinated control.

[0257] Step 5: Generate an adaptive parameter adjustment strategy;

[0258] When a stability risk is triggered, it can be determined based on the current semantic level. Perform the following actions according to priority:

[0259] Random noise suppression: reducing the magnitude of noise injection in the generative model. Temporal Conditioning: Increases the weight of historical frames or historical semantic states in the generation process; Generation Steps and Sampling Path Convergence: In weak network or high packet loss conditions, fixes the number of inference steps to avoid random sampling path drift; Semantic Level Backtracking Request: If the STScore is not recovered within consecutive (M) frames, requests are made to Module 2 to upgrade to Level 3. .

[0260] Step 6: Stability recovery assessment and adjustment release;

[0261] In consecutive frames after the adjustment takes effect, stability indicators are continuously monitored until the following conditions are met:

[0262]

[0263] This involves determining whether the generation process has re-entered a stable region and gradually restoring the noise injection amplitude; reducing the weight of the time condition; and allowing the semantic state machine to return to its normal differential level. This process employs a gradual release to avoid secondary oscillations caused by a one-time release.

[0264] Step S208: Invoke the adjusted generation model to generate video frames after the first target video frame.

[0265] In some embodiments of this application, considering that the generation device may be a mobile device with relatively weak computing power, a reasoning scheduling and hardware adaptation mechanism for weak computing power, low power consumption, and weak network scenarios can be constructed for the generation model. This ensures that the edge AIGC generation process has predictable latency, stable computing power usage, and controllable power consumption boundaries while maintaining semantic consistency, identity stability, and temporal continuity. This allows the generation model to run continuously without crashing or degrading under conditions of fluctuating computing power, NPU resource contention, and unstable semantic input in long-term live streaming scenarios. The specific scheduling process is as follows: Figure 9 As shown, it includes the following steps:

[0266] Step 1: Generate an inference path lock;

[0267] On devices with low computing power, to avoid uncontrollable computational load due to random sampling paths, the basic inference path of the generated model can be locked during session initialization:

[0268]

[0269] in: Typically, 3 to 6 steps are taken, and this number remains constant throughout the entire live session. A fixed number of inference steps directly determines the upper bound of the generation latency, providing hard latency guarantees for real-time systems.

[0270] Step 2: Allocate computing power budget based on semantic level;

[0271] The computing power budget for the generating end can be allocated hierarchically based on the current semantic level Mode_t:

[0272]

[0273] Where:

[0274] Level 0 / Level 1:

[0275]

[0276] Level 2:

[0277]

[0278] Level 3:

[0279]

[0280] The computing power budget can be reflected in: NPU scheduling priority, number of parallel operators, and upper limit of attention resolution, etc.

[0281] This ensures that computational power consumption is only allowed to increase when there is a "real semantic change".

[0282] Step 3: Resolution and frequency degradation of the LoRA computation path;

[0283] To address the LoRA identity lock injected into the generative model, the following weak computational power adaptive strategy can be introduced:

[0284] First, the LoRA Attention Map is reduced in resolution:

[0285] Where (r > 1) represents the downsampling factor.

[0286] Secondly, in non-Level 3 states, the frequency of LoRA's participation in computation is reduced, and it is only fully enabled when a keyframe or identity risk is triggered. Through this method, the identity lock will persist, but its computational intensity can be dynamically reduced.

[0287] Step 4: Randomness suppression scheduling based on stability indices;

[0288] Based on the generation stability index, when the following conditions are met:

[0289]

[0290] Randomness can be actively generated through convergence:

[0291]

[0292] It also restricts sources of computing power instability, including: prohibiting dynamic step adjustments, freezing high-cost random sampling branches, and forcing the use of cached features or historical latent variables. This directly maps "generation stability issues" to deterministic behavior on the computing power side.

[0293] Step 5: Joint degradation strategy for weak computing power and weak network;

[0294] When simultaneously detected:

[0295]

[0296] If the current situation is determined to be within a risk range of both weak network and weak computing power, the following joint degradation strategy will be implemented:

[0297] By freezing semantic-driven operations and using only the prediction results from Module 5, and pausing unnecessary LoRA refreshes, the generation frequency is limited to within a device-safe threshold. This strategy ensures that no crashes or identity distortions occur even in the event of steady-state degradation.

[0298] Step 6: Conditions for restoring computing power and releasing scheduling;

[0299] When the generator detects that the following conditions are met consecutively:

[0300]

[0301] The original computational resolution, normal attention engagement, and default noise injection and sampling strategies of LoRA can be gradually restored. Furthermore, the restoration process can be implemented in stages to avoid a new round of instability caused by sudden increases in computing power.

[0302] In some embodiments of this application, the following are also provided: Figure 10 The video transmission process is shown below. Figure 10 As can be seen, the push end first extracts the semantic skeleton and quantizes attention on the input raw video frame to construct the corresponding semantic state. Then, through a multi-level semantic differential state machine, it determines the transmission level (Level 0 / 1 / 2 / 3) based on the intensity of semantic changes, and encodes the semantic differential data of the corresponding level into data packets, which are then transmitted to the generation end through a weak network.

[0303] Upon receiving data packets, the generator first decodes them to reconstruct the current semantic state. If low-level transmission (Level 0 / 1) or semantic packet loss occurs, semantic optical flow prediction and packet loss compensation are used to predict and generate driving semantics based on historical semantic states and semantic difference information. Then, a LoRA identity lock weight specific to the broadcaster is injected to maintain the consistency of the person's identity. Afterward, the generator performs random and controllable generation based on the driving semantics, outputs video frames, and performs identity consistency evaluation (IdScore) and generation stability evaluation (STScore) on the generated frames. Finally, the evaluation results are fed back to the transmission and generation stages. At the same time, NPU computing power is scheduled according to the evaluation results and transmission level to dynamically adjust generation constraints and computing power allocation. Steady-state degradation is achieved in scenarios with weak network and weak computing power, ensuring the continuous and stable operation of the live broadcast.

[0304] By using a semantic data set sent by the receiving push terminal and the corresponding data volume level, the generation method of the first target video frame is determined based on the data volume level. The generation model of the generation terminal is then called to generate the first target video frame using the generation method. The generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and calling the generation model to predict the first target video frame using the semantic data set as input. The stability evaluation result corresponding to the first target video frame is determined, and the generation parameters of the generation model are adjusted based on the stability evaluation result. By using a semantic data set sent by the receiving push terminal instead of complete video data, the amount of data that needs to be transmitted during video transmission is reduced, thereby reducing the amount of data transmitted during live streaming. This achieves the technical effect of data transmission even in weak network environments, thus solving the technical problem of potential transmission failure in weak network environments caused by transmitting complete video frames in related technologies.

[0305] This application embodiment also provides a method such as Figure 11 The video transmission system shown. From Figure 11As can be seen, the system includes a push-end device 110 and a generation-end device 112. The push-end device 110 is used to determine the semantic data set of the acquired video frames and the corresponding data volume level. The generation-end device 112 is used to receive the semantic data set sent by the push-end device and the corresponding data volume level; determine the generation method of the first target video frame based on the data volume level, and call the generation model to generate the first target video frame using the generation method. The generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and calling the generation model to predict and generate the first target video frame based on the semantic data set; determine the stability evaluation result corresponding to the first target video frame, and adjust the generation parameters of the generation model based on the stability evaluation result; and call the adjusted generation model to generate video frames after the first target video frame.

[0306] Optionally, the push device 110 also includes a semantic skeleton feature extraction and adaptive attention quantization module, and a multi-level semantic differential decision state machine and weak network transmission control module. The semantic skeleton feature extraction and adaptive attention quantization module is used in extremely weak network environments to transform the most decisive visual information regarding a person's identity, facial expressions, and posture in consecutive video frames into controllable, quantifiable, and differential semantic skeleton states, which serve as the sole upstream input for the subsequent semantic differential decision state machine and the generation-end rendering. The multi-level semantic differential decision state machine and weak network transmission control module is used in extremely weak network environments to dynamically determine the semantic transmission level, data packet format, and transmission frequency for the current frame based on the actual change amplitude between adjacent semantic states, thereby compressing the transmission overhead to a range acceptable to the network while ensuring generation quality.

[0307] Optionally, the generation device 112 also includes a LoRA-based identity locking and consistency verification closed-loop control module, an edge-side controllable generation and generation stability adaptive adjustment module, a semantic skeleton space-based optical flow prediction and packet loss compensation module, a semantic differential data packet structure and encoding / decoding module for weak network scenarios, and an edge-side AIGC inference NPU adaptation and weak computing power scheduling module. Among these:

[0308] The LoRA-based identity locking and consistency verification closed-loop control module ensures that, without relying on pixel-level video transmission, the identity features of the person generated remain stable, unwavering, and unaffected by semantic noise, even under conditions such as long-term live streaming, weak network fluctuations, semantic packet loss, and frequent state machine switching.

[0309] The edge-controlled generation and generation stability adaptive adjustment module is used to ensure that the generated video does not flicker, jump, or diverge in the time dimension, and always maintains perceptible continuity, under the conditions of unstable semantic driving frequency, frequent switching of differential levels, and dynamic intervention of identity closed loop.

[0310] The optical flow prediction and packet loss compensation module based on semantic skeleton space is used in weak network environments to predict future states in the semantic space by utilizing historical semantic skeleton states when semantic differential data is delayed, lost, or judged as low-level and not sent, thereby maintaining the continuity and stability of generation drive.

[0311] The semantic differential data packet structure and encoding / decoding module in weak network scenarios is used to decode the semantic differential results generated by the streaming end into a structured semantic state under the constraint of semantic transmission level judgment. It also has fault-tolerant parsing capabilities, providing reliable data support for generation drive.

[0312] The NPU adaptation and weak computing power scheduling module for edge AIGC inference is used to ensure that the edge AIGC generation process has predictable latency, stable computing power usage, and controllable power consumption boundaries, while guaranteeing semantic consistency, identity stability, and time continuity, so as to ensure that the generated model can run continuously for a long time on devices with weak computing power.

[0313] This application provides a video transmission device. Figure 12 This is a schematic diagram of the device. From Figure 12 As can be seen from the diagram, the device includes: a first processing module 120, used to receive a semantic data set sent by the push terminal, and a data volume level corresponding to the semantic data set; a second processing module 122, used to determine the generation method of the first target video frame based on the data volume level, and call the generation model of the generation terminal to generate the first target video frame using the generation method, wherein the generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and using the semantic data set as input to call the generation model to predict and obtain the first target video frame; a third processing module 124, used to determine the stability evaluation result corresponding to the first target video frame, and adjust the generation parameters of the generation model based on the stability evaluation result; and a fourth processing module 126, used to call the adjusted generation model to generate video frames after the first target video frame.

[0314] In some embodiments of this application, after receiving the semantic data set sent by the push terminal and the data volume level corresponding to the semantic data set, the first processing module 120 is further configured to: determine the data packet template corresponding to the data packet based on the data volume level when the semantic data set is compressed into a data packet, wherein the data packet template includes a set of fields of the data packet; and decode the data packet based on the data packet template to obtain the semantic data set.

[0315] In some embodiments of this application, the data volume level includes a first data volume level and a second data volume level; the step of the second processing module 122 determining the generation method of the first target video frame based on the data volume level includes: when the data volume level is the first level, the second level, or the third level, determining the generation method as calling the generation model to predict and generate the first target video frame based on the semantic data set; when the data volume level is the fourth level, determining the generation method as calling the generation model to reconstruct the first target video frame based on the semantic data set.

[0316] The first data volume level includes a first level, a second level, and a third level. The first level indicates that the semantic data set does not contain semantic information. The second level indicates that the semantic data set contains the first semantic difference information between the key points in the second target video frame and the corresponding key points in the reference video frame. The third level indicates that the semantic data set contains the second semantic difference information between the second target video frame and the reference video frame. Both the second target video frame and the reference video frame are video frames collected by the push terminal, and the collection time of the reference video frame is earlier than that of the second target video frame. The second data volume level includes a fourth level, where the fourth level indicates that the semantic data set contains the semantic information of the second target video frame.

[0317] In some embodiments of this application, before the generation model of the generation end is called to generate the first target video frame, the second processing module 122 is further configured to: determine the identity lock weight of the target object, wherein the identity lock weight is used to apply a continuous identity constraint to the feature space of the generation model; and add the identity lock weight to the generation model.

[0318] In some embodiments of this application, the step of the second processing module 122 calling the generation model of the generation end to generate the first target video frame includes: determining semantic optical flow, wherein the semantic optical flow is used to reflect the semantic position change information of key points in a preset semantic skeleton space; predicting the target semantic position of key points at the target time based on the semantic optical flow; determining the predicted semantic information based on the target semantic position of key points; and generating the first target video frame based on the predicted semantic information.

[0319] In some embodiments of this application, the step of the second processing module 122 generating a first target video frame based on predicted semantic information includes: determining the difference information between the predicted semantic information and reference semantic information, wherein the reference semantic information is semantic information in a historical semantic sequence; determining a deviation evaluation index corresponding to the predicted semantic information based on the difference information, wherein the deviation evaluation index is used to indicate the risk probability that the video frame generated based on the predicted semantic information is an abnormal frame, and the larger the deviation evaluation index, the greater the risk probability; and generating a first target video frame based on the predicted semantic information when the deviation evaluation index is not greater than a preset threshold.

[0320] In some embodiments of this application, the step of the third processing module 124 in determining the stability evaluation result corresponding to the first target video frame includes: determining the motion intensity evaluation result corresponding to the first target video frame; determining the temporal smoothness evaluation result corresponding to the first target video frame; determining the identity consistency evaluation result of the first target video frame; and performing a weighted summation of the motion intensity evaluation result, the temporal smoothness evaluation result, and the identity consistency evaluation result to obtain the stability evaluation result.

[0321] In some embodiments of this application, the step of the third processing module 124 in determining the identity consistency evaluation result of the first target video frame includes: extracting the identity embedding vector of the first target video frame; determining the similarity between the identity embedding vector and the reference identity embedding vector; and determining the identity consistency evaluation result based on the similarity.

[0322] It should be noted that the various modules in the above video transmission can be program modules (such as a set of program instructions to implement a certain function) or hardware modules. For the latter, they can be represented in the following forms, but are not limited to these: each of the above modules is represented by a processor, or the functions of each of the above modules are implemented by a processor.

[0323] According to an embodiment of this application, a non-volatile storage medium is also provided, which stores a program. During program execution, the device containing the non-volatile storage medium executes the following video transmission method: receiving a semantic data set sent by a push end, and a data volume level corresponding to the semantic data set; determining the generation method of a first target video frame based on the data volume level, and calling the generation model of the generation end to generate the first target video frame using the generation method. The generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and using the semantic data set as input to call the generation model to predict and obtain the first target video frame; determining the stability evaluation result corresponding to the first target video frame, and adjusting the generation parameters of the generation model based on the stability evaluation result; and calling the adjusted generation model to generate video frames after the first target video frame.

[0324] According to an embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the following steps of a video transmission method: receiving a semantic data set sent by a push terminal, and a data volume level corresponding to the semantic data set; determining a generation method for a first target video frame based on the data volume level, and calling a generation model of a generation terminal to generate the first target video frame using the generation method, wherein the generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and using the semantic data set as input to call the generation model to predict and obtain the first target video frame; determining a stability evaluation result corresponding to the first target video frame, and adjusting the generation parameters of the generation model based on the stability evaluation result; and calling the adjusted generation model to generate video frames after the first target video frame.

[0325] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0326] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0327] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0328] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0329] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0330] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A video transmission method, characterized in that, include: Receive the semantic data set sent by the push terminal, and the data volume level corresponding to the semantic data set; The generation method of the first target video frame is determined based on the data volume level, and the generation model of the generation end is called to generate the first target video frame using the generation method. The generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and using the semantic data set as input, calling the generation model to predict the first target video frame. Determine the stability evaluation result corresponding to the first target video frame, and adjust the generation parameters of the generation model based on the stability evaluation result; The adjusted generation model is invoked to generate video frames following the first target video frame.

2. The video transmission method according to claim 1, characterized in that, The data volume levels include a first data volume level and a second data volume level; determining the generation method of the first target video frame based on the data volume levels includes: When the data volume level is the first data volume level, the generation method is determined to be to call the generation model to predict and generate the first target video frame based on the semantic data set; When the data volume level is the second data volume level, the generation method is determined to be to call the generation model to reconstruct the first target video frame based on the semantic data set.

3. The video transmission method according to claim 2, characterized in that, The first data volume level includes a first level, a second level, and a third level. The first level indicates that the semantic data set does not contain semantic information. The second level indicates that the semantic data set contains first semantic difference information between key points in the second target video frame and corresponding key points in the reference video frame. The third level indicates that the semantic data set contains second semantic difference information between the second target video frame and the reference video frame. Both the second target video frame and the reference video frame are video frames collected by the push terminal, and the collection time of the reference video frame is earlier than the collection time of the second target video frame. The second data volume level includes a fourth level, wherein the fourth level indicates that the semantic data set contains semantic information of the second target video frame.

4. The video transmission method according to claim 1, characterized in that, Before calling the generation model of the generation end to generate the first target video frame using the generation method, the method further includes: Determine the identity lock weight of the target object, wherein the identity lock weight is used to apply a persistent identity constraint to the feature space of the generative model; Add the identity lock weight to the generative model.

5. The video transmission method according to claim 1, characterized in that, The process of generating the first target video frame by calling the generation model of the generation end and using the generation method includes: Determine the semantic optical flow, wherein the semantic optical flow is used to reflect the semantic position change information of key points in a preset semantic skeleton space; Based on the semantic optical flow, predict the target semantic location of the key point at the target time; Based on the target semantic location of the key points, predictive semantic information is determined; The first target video frame is generated based on the predicted semantic information.

6. The video transmission method according to claim 5, characterized in that, Generating the first target video frame based on the predicted semantic information includes: Determine the difference information between the predicted semantic information and the reference semantic information, wherein the reference semantic information is semantic information in the historical semantic sequence; Based on the difference information, a deviation evaluation index corresponding to the predicted semantic information is determined, wherein the deviation evaluation index is used to indicate the risk probability that the video frame generated based on the predicted semantic information is an abnormal frame, and the larger the deviation evaluation index, the greater the risk probability. If the deviation evaluation index is not greater than a preset threshold, the first target video frame is generated based on the predicted semantic information.

7. The video transmission method according to claim 1, characterized in that, The stability evaluation result corresponding to the first target video frame is determined as follows: Determine the motion intensity evaluation result corresponding to the first target video frame; Determine the temporal smoothness evaluation result corresponding to the first target video frame; Determine the identity consistency evaluation result of the first target video frame; The stability evaluation result is obtained by weighted summation of the action intensity evaluation result, the time smoothness evaluation result, and the identity consistency evaluation result.

8. The video transmission method according to claim 7, characterized in that, The identity consistency evaluation result of the first target video frame is determined as follows: Extract the identity embedding vector of the first target video frame; Determine the similarity between the identity embedding vector and the reference identity embedding vector; The identity consistency evaluation result is determined based on the similarity.

9. The video transmission method according to claim 1, characterized in that, After receiving the semantic data set sent by the push terminal, and the data volume level corresponding to the semantic data set, the method further includes: When the semantic data set is compressed into a data packet, the data packet template corresponding to the data packet is determined according to the data volume level, wherein the data packet template includes the set of fields of the data packet; The semantic data set is obtained by decoding the data packet according to the data packet template.

10. A video transmission system, characterized in that, This includes the push end and the generation end, among which, The push terminal device is used to determine the semantic data set of the collected video frames, and the data volume level corresponding to the semantic data set; The generation device is configured to receive the semantic data set sent by the push terminal, and the data volume level corresponding to the semantic data set; determine the generation method of the first target video frame based on the data volume level, and call the generation model to generate the first target video frame using the generation method, wherein the generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and calling the generation model to predict and generate the first target video frame based on the semantic data set; determine the stability evaluation result corresponding to the first target video frame, and adjust the generation parameters of the generation model based on the stability evaluation result; and call the adjusted generation model to generate video frames after the first target video frame.

11. A video transmission device, characterized in that, include: The first processing module is used to receive a semantic data set sent by the push terminal, and the data volume level corresponding to the semantic data set; The second processing module is used to determine the generation method of the first target video frame based on the data volume level, and call the generation model of the generation end to generate the first target video frame using the generation method. The generation method includes calling the generation model to reconstruct the first target video frame based on the semantic data set, and using the semantic data set as input to call the generation model to predict the first target video frame. The third processing module is used to determine the stability evaluation result corresponding to the first target video frame, and adjust the generation parameters of the generation model according to the stability evaluation result; The fourth processing module is used to call the adjusted generation model to generate video frames after the first target video frame.

12. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device containing the non-volatile storage medium to perform the video transmission method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the video transmission method according to any one of claims 1 to 9.

Images