Intelligent video stream repairing method based on space-time entropy balance and cross-modal semantic reconstruction

Abstract

The invention discloses an intelligent video stream repairing method based on space-time entropy equalization and cross-modal semantic reconstruction, which comprises the following steps: receiving a video stream through a GB28181 protocol, and carrying out video frame decoding and format conversion to obtain a video frame sequence in a standard format; calculating a time entropy variable quantity based on continuous video frames in the video frame sequence, setting a video frame damage threshold value according to the video coding quantization parameter, and if the time entropy variable quantity is greater than the video frame damage threshold value, indicating that the video frames are damaged; for the video frame with the damage, calculating the spatial information entropy of the video frame, determining the area of a damage region according to the spatial information entropy of the video frame, and determining the damage level; and according to the damage level, selecting a corresponding repair strategy to repair the damaged video frame. According to the method, the time-space domain analysis technology and the cross-modal semantic comprehension capability are deeply fused, simple-to-complex progressive restoration is realized through a three-stage restoration architecture, and the restoration quality, the real-time performance and the compatibility are improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video processing and streaming media transmission technology, specifically, it relates to an intelligent video stream repair method based on spatiotemporal entropy balance and cross-modal semantic reconstruction. Background Technology

[0002] In the fields of video surveillance and streaming media transmission, video data is susceptible to interference from various factors during acquisition, transmission, and storage, leading to video quality degradation. Video restoration can improve video quality, ensure the stable operation of video surveillance systems, and enhance user experience.

[0003] However, current video restoration technologies in the fields of video surveillance and streaming media transmission face numerous challenges and limitations, which severely restrict the practical application effectiveness of video surveillance systems: (1) Currently, the mainstream video restoration methods mainly rely on simple frame copying and interpolation algorithms. These methods perform reasonably well when dealing with static scenes, but often produce obvious ghosting and trailing phenomena when encountering dynamic scenes. At the same time, traditional pixel-level restoration algorithms will destroy the texture details of the video, resulting in blurring and mosaic effects in the restored image, which seriously affects the readability and evidentiary value of the surveillance video; (2) Existing video transmission protocols such as GB28181 and RTSP mainly focus on the transmission and control of video data and lack native support for video restoration functions. The protocols do not define a negotiation mechanism for restoration parameters, nor do they reserve fields for quality feedback, which makes it difficult for the video restoration process to be deeply integrated with the transmission system and to achieve end-to-end optimization. (3) Traditional video restoration technology mainly relies on preset rules and fixed algorithms, lacking the ability to understand the semantics of video content. When dealing with complex scenes such as multiple people moving or rapidly changing backgrounds, the restoration results often show semantic inconsistencies, such as misaligned limbs or deformed objects. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a video stream intelligent repair method based on spatiotemporal entropy equilibrium and cross-modal semantic reconstruction. It deeply integrates spatiotemporal domain analysis technology and cross-modal semantic understanding capabilities, and achieves progressive repair from simple to complex through a three-level repair architecture, thereby improving repair quality, real-time performance and compatibility, and better meeting the actual needs of the video surveillance field.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: A video stream intelligent restoration method based on spatiotemporal entropy equalization and cross-modal semantic reconstruction includes the following steps: Step S1: Receive the video stream via the GB28181 protocol, perform video frame decoding and format conversion, and obtain a standard format video frame sequence; Step S2: Calculate the change in temporal entropy based on consecutive video frames in the video frame sequence, and set the video frame impairment threshold according to the video coding quantization parameters. If the change in temporal entropy is greater than the video frame impairment threshold, it indicates that there is impairment on the video frame. Step S3: For a damaged video frame, calculate the spatial information entropy of the video frame, and determine the area of ​​the damaged region and the damage level based on the spatial information entropy of the video frame. Step S4: Select the appropriate repair strategy based on the damage level to repair the damaged video frames.

[0006] Furthermore, the calculation process for the change in time entropy in step S2 is as follows:

[0007] in, This represents the change in temporal entropy of the video frame. This indicates the brightness of the video frame. This indicates the brightness of the previous frame. This represents the L2 norm.

[0008] Further, the calculation process of the spatial information entropy of the video frame in step S3 is as follows: convert the video frame into a grayscale image, calculate the statistical probability of the grayscale distribution of the grayscale image, and obtain the spatial information entropy of the video frame. :

[0009] in, This represents the number of pixels in a grayscale image. express index, Indicates the first grayscale image Statistical probability of grayscale distribution of individual pixels.

[0010] Furthermore, the specific process of calculating the damaged area and determining the damage level based on the spatial information entropy of the video frame in step S3 is as follows: i: Set an entropy threshold, find pixels with entropy values ​​lower than the threshold based on the spatial information entropy of the video frame, determine the damaged area, and calculate the area of ​​the damaged area; ii: Calculate the percentage of the damaged area of ​​the video frame based on the damaged area and the total area of ​​the video frame; iii: If the damaged area of ​​the video frame accounts for less than 5%, it indicates minor damage; if the damaged area of ​​the video frame accounts for between 5% and 20%, it indicates moderate damage; if the damaged area of ​​the video frame accounts for more than 20%, it indicates severe damage.

[0011] Furthermore, if the damage is minor, a repaired video frame is generated using the motion information from the first few frames of the video frame through weighted compensation:

[0012] in, This indicates the repaired video frame. This refers to the first few frames of the video. express index, Indicates the pixel position on the video frame. Indicates the first The motion displacement of each video frame and the pixel positions on the video frame. express The weighted compensation coefficient for each video frame.

[0013] Furthermore, if the damage is moderate, a binary mask is set according to the damaged area of ​​the video frame, and the video frame and the set binary mask are input into an image inpainting model based on a generative adversarial network to generate a repaired video frame.

[0014] Furthermore, in the case of severe damage, the environmental audio signal is extracted simultaneously, the Mel frequency cepstral coefficients of the environmental audio signal are calculated, and the audio semantic features are extracted using the CLIP model. The audio semantic features, along with the temporal entropy change and spatial information entropy of the video frame, are input into the LSTM model to generate a semantically guided repair vector. The video frame is used as the input to the diffusion model, and the semantically guided repair vector is used as the conditional input to the diffusion model. The repaired video frame is generated through inverse denoising.

[0015] Furthermore, it also includes: constructing a quality assessment function for video frame restoration, performing quality assessment on the restored video frames, and dynamically adjusting the restoration strategy based on the quality assessment results.

[0016] Furthermore, the quality evaluation function of the video frame restoration The construction process is as follows:

[0017] in, Indicating structural similarity of video frames, express Weighting coefficients; Indicates the peak signal-to-noise ratio of a video frame. express Weighting coefficients; This indicates that the perceptual quality of video frames fused using multiple methods is evaluated. express The weighting coefficients.

[0018] Furthermore, X-Repair-Params:<Level, QP, T_latency> is added to the SIP message header of the GB28181 protocol, where Level represents the impairment level, QP represents the video coding quantization parameter, and T_latency represents the maximum permissible latency.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) The video stream intelligent repair method based on spatiotemporal entropy balance and cross-modal semantic reconstruction of the present invention analyzes the damage level of video frames through spatiotemporal entropy, selects the corresponding repair algorithm according to the damage level to repair video frames, realizes the accurate allocation of repair resources, avoids over-repair or under-repair, significantly improves processing efficiency, and meets real-time requirements. (2) The video stream intelligent repair method based on spatiotemporal entropy balance and cross-modal semantic reconstruction of the present invention associates audio semantic features with video frame content features when repairing severely damaged video frames, generates repaired video frames that conform to scene semantics, improves the ability to repair large-area damage, and the generated picture conforms to scene logic, improving the video's understandability and evidentiary value. (3) The intelligent video stream repair method based on spatiotemporal entropy balance and cross-modal semantic reconstruction of the present invention constructs a quality assessment function for video frame repair by using the structural similarity of video frames, the peak signal-to-noise ratio of video frames and the perceptual quality of video frames fused by multiple video evaluation methods. This function can comprehensively evaluate the accuracy of video frame repair, thereby dynamically adjusting the repair strategy and optimizing resource allocation. (4) The video stream intelligent repair method based on spatiotemporal entropy balance and cross-modal semantic reconstruction of the present invention extends the GB28181 protocol fields. By negotiating the new repair parameters, the extended header and the device interact to inform the device of the current damage level. The device automatically adjusts the video encoding quantization parameters according to the level to avoid further damage caused by network congestion. The video repair process is coordinated with the transmission system to achieve end-to-end optimization. Attached Figure Description

[0020] Figure 1 This is a flowchart of the intelligent video stream repair method based on spatiotemporal entropy equalization and cross-modal semantic reconstruction of the present invention; Figure 2 This is a schematic diagram illustrating the process of quality assessment of repaired video frames according to the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.

[0022] like Figure 1 This is a flowchart of the intelligent video stream restoration method based on spatiotemporal entropy equalization and cross-modal semantic reconstruction of the present invention. The intelligent video stream restoration method includes the following steps: Step S1: Receive the video stream via the GB28181 protocol, perform video frame decoding and format conversion to obtain a standard format video frame sequence, and establish a video frame buffer queue to temporarily store the current video frame and its preceding and following frames, providing a data foundation for subsequent spatiotemporal entropy analysis and graded repair.

[0023] Step S2: Calculate the temporal entropy change based on consecutive video frames in the video frame sequence. This reflects the intensity of scene motion or abrupt changes. A video frame impairment threshold is set according to the video coding quantization parameters. If the temporal entropy change is greater than the video frame impairment threshold, it indicates that impairment exists on that video frame. The video frame impairment threshold in this invention... ,in, This represents the video encoding quantization parameter, which controls the degree of video compression. The larger the video encoding quantization parameter, the higher the compression ratio, but the lower the video quality. This represents the adaptive threshold coefficient, used to balance detection sensitivity and false positive rate.

[0024] The calculation process for the change in time entropy in this invention is as follows:

[0025] in, This represents the change in temporal entropy of the video frame. This indicates the brightness of the video frame. This indicates the brightness of the previous frame. This represents the L2 norm.

[0026] Step S3: For a damaged video frame, calculate the spatial information entropy of the video frame, and determine the area of ​​the damaged region and the damage level based on the spatial information entropy of the video frame; specifically: The video frame is converted to a grayscale image, and the statistical probability of the grayscale distribution is calculated to obtain the spatial information entropy of the video frame. This entropy is used to measure the intra-frame texture complexity and information content; a higher spatial information entropy value indicates more complex intra-frame texture and greater information content. The calculation process is as follows:

[0027] in, This represents the number of pixels in a grayscale image. express index, Represents the first [item] in the grayscale image. Statistical probability of grayscale distribution of individual pixels.

[0028] Set an entropy threshold, find pixels with entropy values ​​below the threshold based on the spatial information entropy of the video frame, determine the damaged area, and calculate the area of ​​the damaged area. The percentage of the video frame's damaged area is calculated based on the damaged area and the total area of ​​the video frame. If the damaged area of ​​a video frame is less than 5%, it indicates minor damage; if the damaged area of ​​a video frame is between 5% and 20%, it indicates moderate damage; if the damaged area of ​​a video frame is greater than 20%, it indicates severe damage.

[0029] Step S4: Select the appropriate repair strategy based on the damage level to repair the damaged video frames.

[0030] This invention presents a video stream intelligent repair method based on spatiotemporal entropy balance and cross-modal semantic reconstruction. By analyzing the damage level of video frames through spatiotemporal entropy, the method selects the appropriate repair algorithm to repair the video frames according to the damage level, thereby achieving precise allocation of repair resources, avoiding over-repair or under-repair, significantly improving processing efficiency, and meeting real-time requirements.

[0031] For minor damage, the repair targets are mainly block effects or subtle flaws caused by minor packet loss or compression. By using the motion information of the previous few frames of the video frame, a repaired video frame can be generated through weighted compensation, which can achieve pixel-level repair, thereby accurately repairing minor damage and preserving the original details.

[0032] The process of generating the repaired video frames is as follows:

[0033] in, This indicates the repaired video frame. This refers to the first few frames of the video. express index, This indicates the position of a pixel on the video frame. Indicates the first Each video frame and the motion displacement of the pixels within that video frame. express The weighted compensation coefficient for each video frame.

[0034] For moderate damage, the repair target is a small area of ​​missing or blurred content. A binary mask is set according to the damaged area of ​​the video frame. The video frame and the set binary mask are input into an image inpainting model based on generative adversarial networks to generate a repaired video frame. This can accurately locate the damaged area and achieve local repair.

[0035] For severe damage, the area to be repaired is one that cannot be recovered from previous and subsequent frames. Simultaneously, environmental audio signals are extracted, and the Mel-frequency cepstral coefficients of these signals are calculated. Audio semantic features are then extracted using a CLIP model. These audio semantic features, along with the temporal entropy change and spatial entropy of the video frame, are input into an LSTM model to generate a semantically guided repair vector. This video frame is used as the input to a diffusion model, and the semantically guided repair vector is used as the conditional input. Inverse denoising is then used to generate the repaired video frame. This invention associates audio semantic features with the content features of the video frame to generate repaired video frames that conform to the semantics of the scene, improving the ability to repair large-area damage. The generated images conform to the scene logic, enhancing the video's comprehensibility and evidentiary value. Step S5: Construct a quality assessment function for video frame restoration, assess the quality of the restored video frames, and dynamically adjust the restoration strategy based on the quality assessment results.

[0036] like Figure 2 This invention constructs a quality assessment function for video frame restoration by using structural similarity, peak signal-to-noise ratio (PSNR), and multi-method evaluation fusion of video frames to construct the perceived quality. Structural similarity measures the similarity between the restored and original frames in terms of brightness, contrast, and structural information. PSNR is an objective evaluation metric based on inter-pixel errors, reflecting the fidelity of the video frame; a higher PSNR indicates less distortion. The multi-method evaluation fusion of perceived quality is a comprehensive metric developed by Netflix, integrating results from multiple fundamental visual quality models to better assess the perceived quality of the restoration results. This quality assessment function comprehensively evaluates the accuracy of video frame restoration, allowing for dynamic adjustment of restoration strategies and optimization of resource allocation. The construction process is as follows:

[0037] in, Indicating structural similarity of video frames, express Weighting coefficients; Indicates the peak signal-to-noise ratio of a video frame. express Weighting coefficients; This indicates that the perceptual quality of video frames fused using multiple methods is evaluated. express The weighting coefficients.

[0038] If the quality assessment function for video frame restoration of mild and moderate damage is below 0.8, the corresponding damage level will be increased by one level; if the quality assessment function for video frame restoration under each restoration strategy is between 0.8 and 0.9, the corresponding damage level will remain unchanged; if the quality assessment function for video frame restoration of moderate and severe damage is greater than 0.9, the corresponding damage level will be decreased by one level.

[0039] In one technical solution of the present invention, X-Repair-Params:<Level, QP, T_latency> is added to the SIP message header of the GB28181 protocol. By negotiating the newly added repair parameters, the extended header interacts with the device to inform the device of the current damage level. The device automatically adjusts the video coding quantization parameters according to the level to avoid further damage caused by network congestion. The video repair process is coordinated with the transmission system to achieve end-to-end optimization. Here, Level represents the damage level, QP represents the video coding quantization parameters, and T_latency represents the maximum allowable latency.

[0040] In one technical solution of the present invention, the video coding quantization parameters can be dynamically adjusted: if the video is severely damaged, and To reduce the video bitrate by 15%-20% and increase the video encoding quantization parameters by 3-5, reduce the amount of data transmitted to alleviate link pressure; if the damage is minor, and This increases the video bitrate by 5%-10%, ensuring that the original details of the video are preserved.

[0041] This invention presents an intelligent video stream repair method based on spatiotemporal entropy equalization and cross-modal semantic reconstruction. It deeply integrates spatiotemporal domain analysis technology and cross-modal semantic understanding capabilities, achieving progressive repair from simple to complex through a three-level repair architecture. This ensures video repair latency is less than 25ms and improves repair quality, real-time performance, and compatibility, better meeting the practical needs of the video surveillance field. In one technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program that enables a computer to execute the intelligent video stream repair method based on spatiotemporal entropy equalization and cross-modal semantic reconstruction of the present invention.

[0042] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent video stream repair method based on spatiotemporal entropy equalization and cross-modal semantic reconstruction of the present invention.

[0043] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0044] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0045] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A video stream intelligent repairing method based on spatiotemporal entropy equalization and cross-modal semantic reconstruction, characterized in that, Comprising the following steps: Step S1: receiving a video stream through the GB28181 protocol, decoding and format converting the video frames to obtain a sequence of video frames in a standard format; Step S2: calculating a time entropy change quantity based on consecutive video frames in the sequence of video frames, and setting a video frame damage threshold according to a video encoding quantization parameter, wherein if the time entropy change quantity is greater than the video frame damage threshold, it indicates that there is damage on the video frame; Step S3: for the video frame with damage, calculating the spatial information entropy of the video frame, determining the area of the damage region according to the spatial information entropy of the video frame, and determining the damage level; Step S4: selecting a corresponding repair strategy according to the damage level to repair the damaged video frame.

2. The video stream intelligent repairing method based on space-time entropy equalization and cross-modal semantic reconstruction according to claim 1, characterized in that, The calculation process of the time entropy change quantity in step S2 is as follows: wherein, denotes a temporal entropy variation of the video frame, denotes a luminance of the video frame, denotes a luminance of a previous frame, denotes a two-norm.

3. The video stream intelligent repairing method based on space-time entropy equalization and cross-modal semantic reconstruction according to claim 1, characterized in that, The calculation process of the spatial information entropy of the video frame in step S3 is: converting the video frame into a gray scale image, calculating the gray scale distribution statistical probability of the gray scale image, and obtaining the spatial information entropy of the video frame : wherein, represents the number of pixel points in a gray scale image, represents an index of represents the gray scale distribution statistical probability of the pixel point in the gray scale image.

4. The video stream intelligent repairing method based on space-time entropy equalization and cross-modal semantic reconstruction according to claim 1, characterized in that, The specific process of calculating the damage area according to the spatial information entropy of the video frame and determining the damage level in step S3 is as follows: i: set an entropy threshold, find the pixel points below the entropy threshold according to the spatial information entropy of the video frame, determine the damage region, and calculate the area of the damage region; ii: calculate the damage area ratio of the video frame according to the damage area and the total area of the video frame; iii: if the damage area ratio of the video frame is less than 5%, it indicates slight damage; if the damage area ratio of the video frame is between 5% and 20%, it indicates moderate damage; if the damage area ratio of the video frame is greater than 20%, it indicates severe damage.

5. The method of claim 4, wherein, If it is slight damage, use the motion information of the previous several frames of the video frame to generate a repaired video frame through weighted compensation: wherein, represents a repaired video frame, represents a number of previous frames of the video frame, represents an index of represents a pixel position on the video frame, represents a motion displacement of the th video frame and the pixel position on the video frame, represents a weighted compensation coefficient of video frames.​ 6. The method of claim 4, wherein, If it is moderate damage, set a binary mask according to the damage region of the video frame, input the video frame and the set binary mask into an image repair model based on a generative adversarial network to generate a repaired video frame.

7. The method of claim 4, wherein, If it is severe damage, extract the environmental audio signal synchronously, calculate the mel frequency cepstral coefficient of the environmental audio signal, extract the audio semantic feature using the CLIP model; input the audio semantic feature, the time entropy change quantity and the spatial information entropy of the video frame into an LSTM model to generate a semantic guided repair vector, use the video frame as the input of the diffusion model, and use the semantic guided repair vector as the conditional input of the diffusion model, and generate a repaired video frame through reverse denoising.

8. The video stream intelligent repairing method based on space-time entropy equalization and cross-modal semantic reconstruction according to claim 1, characterized in that, Further comprising: Building a quality evaluation function for video frame repair, evaluating the quality of the repaired video frame, and dynamically adjusting the repair strategy according to the quality evaluation result.

9. The method of claim 8, wherein, Quality assessment function of the video frame repair The construction process is as follows: wherein, denotes the structural similarity of a video frame, denotes a weight coefficient of denotes the peak signal-to-noise ratio of a video frame, denotes a weight coefficient of denotes the perceptual quality of a video frame of a video multi-method assessment fusion, denotes a weight coefficient of 10. The method of claim 1, wherein, Add X-Repair-Params:〈Level, QP, T_latency〉 in the SIP message header of the GB28181 protocol, wherein Level represents the damage level, QP represents the video encoding quantization parameter, and T_latency represents the maximum allowed delay.

Images