Low-delay high-definition panoramic multi-video stream fusion method and device and medium

Abstract

The invention provides a low-delay high-definition panoramic multi-video stream fusion method and device and a medium, and belongs to the technical field of video transmission. The method comprises the following steps: collecting video data, extracting key frames through a semantic compression strategy, and maintaining a local data queue; constructing a transmission time delay model at an edge side, introducing a time delay deviation index to evaluate the transmission consistency of different video streams, constructing a decoding packet loss probability model, and establishing a relationship between a compression ratio and a packet loss probability; constructing an optimization problem aiming at minimizing the weighted sum of the total time delay and the time delay deviation, and combining the bandwidth resource constraint and the long-term packet loss rate constraint; the optimization problem is converted into a real-time optimization target; carrying out edge bandwidth resource allocation by adopting an SAC algorithm; and dynamically adjusting an end-side video compression ratio based on a bandwidth-oriented UCB algorithm, and dynamically selecting an adaptive scene according to an edge-side allocation bandwidth. According to the invention, the balance between low time delay and low packet loss is realized, and the stability and impression quality of panoramic multi-video stream fusion are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a low-latency high-definition panoramic multi-video stream fusion method, device and medium, belonging to the technical field of video transmission. BACKGROUND

[0002] In the distribution network field operation, the panoramic video stream fusion technology plays a crucial role in improving the operation safety and ensuring the real-time safety monitoring capability. With the continuous access of multi-camera high-definition video terminals, edge computing nodes and wireless transmission networks, the operating environment of the panoramic fusion system is becoming increasingly complex, and higher requirements are put forward for the end-side compression rate adjustment, bandwidth resource allocation and dynamic control of the transmission link. Therefore, there is an urgent need for a fusion method that can accurately depict the coupling relationship of multi-video streams and realize low latency and low packet loss optimization while ensuring real-time, to cope with the uncertainty brought by network fluctuations and data redundancy, and improve the smoothness and stability of the overall monitoring picture.

[0003] In the existing low-latency high-definition panoramic multi-video stream fusion technology, the synchronization alignment and content splicing between video sources are the core links of multi-stream processing and final rendering effect, which have been widely applied in real-time monitoring, emergency command and immersive remote operation scenes. Its typical characteristics include inter-frame consistency, delay compensation and key frame quality maintenance, which can effectively reflect the balance of video stream in terms of latency and picture quality. However, due to the significant differences in camera code streams, strong scene dynamics and bandwidth fluctuations in transmission links, the existing methods have two shortcomings when processing multi-video streams: first, the differences in transmission latency of different video streams are ignored, which limits the system performance to the slowest stream, and easily causes stuttering, misalignment and even picture distortion; second, excessive reliance on high compression rate to reduce bandwidth demand, although it can reduce the average latency, it significantly increases the risk of packet loss, causing key frame loss and panoramic quality degradation. In addition, traditional multi-video stream fusion models mostly use hierarchical optimization architecture, simply adjusting the compression rate at the end side or allocating bandwidth at the edge side, due to the non-convexity and multiple constraints of the optimization problem, often resulting in large latency deviation, uncontrollable packet loss and performance trapped in suboptimal solution, making it difficult to meet the dual demands of low latency and high quality.

[0004] A panoramic video fusion video quality optimization method in the prior art (application number CN202411606070.1) excessively relies on static regional distribution index coding in realizing panoramic video fusion quality optimization, resulting in low dynamic integration efficiency of multi-camera video sequences, especially in high-resolution scenarios, which is prone to inter-frame jitter and color distortion, and cannot realize seamless real-time fusion effect. At the same time, the double-channel optimization mechanism of the technology fails to fully consider the interactive influence of foreground and background, and although the quality evaluation index system covers resolution and contrast, it ignores the quantification of noise interference and illumination change, resulting in poor adaptability of the optimization process to complex environments, and the overall clarity and user immersion of the fused video are difficult to reach the expected level. In addition, the existing panoramic multi-video stream fusion method lacks systematic joint modeling and fails to simultaneously depict the coupling relationship between end-side compression rate, edge bandwidth allocation, transmission bandwidth constraint and packet loss rate, resulting in inaccurate calculation of delay and delay deviation, and it is difficult to meet the low-delay high-reliability transmission demand. SUMMARY

[0005] The purpose of the present application is to provide a low-latency high-definition panoramic multi-video stream fusion method, device and medium, which breaks through the limitation of single compression rate or single bandwidth allocation optimization through end-edge collaborative optimization, realizes the balance of low latency and low packet loss, and improves the stability and visual quality of panoramic multi-video stream fusion.

[0006] To achieve the above purpose, the present application realizes the following technical solutions: A low-latency high-definition panoramic multi-video stream fusion method, comprising the following steps: The end side collects video data through a video collection terminal and extracts key frames through a semantic compression strategy, and each video collection terminal maintains a local data queue to reflect data accumulation; A transmission delay model is constructed on the edge side, and a delay deviation index is introduced to evaluate the transmission consistency of different video streams, a decoding packet loss rate probability model is constructed, and the relationship between compression rate and packet loss rate is established; An optimization problem is constructed to minimize the weighted sum of total delay and delay deviation, and bandwidth resource constraints and long-term packet loss rate constraints are combined; The optimization problem is converted into a real-time optimization target by constructing a virtual queue; The SAC algorithm is used for edge bandwidth resource allocation; The bandwidth-oriented UCB algorithm is used to dynamically adjust the video compression rate on the end side, and the adaptive scene is dynamically selected according to the edge side bandwidth allocation.

[0007] Preferably, the local data queue on the end side is updated at a time interval The difference formula between the collection amount and the transmission amount is: , , wherein, denotes a local data queue, denotes a compression rate, is the amount of raw video data collected by the video collection terminal in the i-th time slot, is the amount of raw video data collected by the video collection terminal in the i-th time slot, is the amount of raw video data collected by the video collection terminal in the i-th time slot, is the amount of actual transmitted data, denotes the time slot length, is the transmission rate between the i-th time slot terminal and the edge, is the transmission rate between the i-th time slot terminal and the edge, is the transmission rate between the i-th time slot terminal and the edge, is the transmission rate between the i-th time slot terminal and the edge, is the transmission rate between the i-th time slot terminal and the edge, is the transmission rate between the i-th time slot terminal and the edge, , wherein, is the bandwidth of the video collection terminal, is the channel gain, is the transmission power of the video collection terminal, is the transmission power of the video collection terminal, is the noise spectral density, is the noise spectral density, is the interference power.

[0008] Preferably, the transmission delay model is: , , wherein, is the data transmission delay of the i-th time slot video collection terminal, is the average data collection amount of the video collection terminal at the end of the i-th time slot, is the average data collection amount of the video collection terminal at the end of the i-th time slot, is the average data collection amount of the video collection terminal at the end of the i-th time slot, is the average data collection amount of the video collection terminal at the end of the i-th time slot, is the average data collection amount of the video collection terminal at the end of the i-th time slot, is the average data collection amount of the video collection terminal at the end of the i-th time slot, is the average data collection amount of the video collection terminal at the end of the i-th time slot, is the average data collection amount of the video collection terminal at the end of the i-th time slot, is the delay deviation, is the variance of the delay of each video stream and the average delay: , wherein, is a set of video collection terminals, is the i-th video collection terminal, is the i-th video collection terminal, is the average delay of the i-th time slot, is the average delay of the i-th time slot, is the average delay of the i-th time slot, Data transmission delay of a video collection terminal; The decoding packet loss rate probability model is: Wherein, is Slot terminal Packet loss rate probability of data decoding, 、 、 、 is a packet loss rate parameter, which is obtained by least square fitting according to historical data, is a natural constant.

[0009] Preferably, the optimization problem of minimizing the weighted sum of total delay and delay deviation is the target, and the bandwidth resource constraint and the long-term packet loss rate constraint are combined, as follows: , Wherein, is the weighting coefficient of total delay and delay deviation, is the bandwidth resource constraint, is the total bandwidth, is the compression rate selection constraint, is the long-term packet loss rate constraint, is the packet loss rate threshold, is the total number of time slots divided by the total optimization period.

[0010] Preferably, the virtual queue is expressed as: , When the virtual queue reaches a stable length, the long-term constraint can be met, and the converted real-time optimization target is: , In the formula, indicates the weight of the original optimization target.

[0011] Preferably, the state space of the SAC algorithm-based side bandwidth resource allocation method is expressed as: , The action space is: , The reward function is Set as the optimization target P2, and adopt the maximum entropy objective function.

[0012] Preferably, the SAC algorithm adopts an experience replay learning mechanism based on multi-dimensional performance differentiation driving, including: Edge side is each training sample Additional multi-dimensional performance labels, the training sample with labels is expressed as: , Wherein, is the traditional experience sample data, is the current state, is the current action, is the current reward, is the next state, is the time delay label of the sample, is the time delay deviation label of the sample, is the packet loss rate label of the sample; The edge side establishes the corresponding experience pool according to the time delay, time delay deviation and packet loss rate, and stores the training sample in the corresponding experience pool when the performance index of the training sample is greater than the threshold value; The comprehensive sampling weight of the training sample based on the label and the experience pool is calculated : , , , , , , , Wherein, is the dynamic sampling weight of the time delay, is the time delay label domain weight, is the dynamic sampling weight of the time delay deviation, is the time delay deviation label domain weight, is the dynamic sampling weight of the packet loss rate, is the packet loss rate label domain weight, is the maximum sample time delay in the experience pool, is the maximum time delay deviation in the experience pool, is the maximum packet loss rate in the experience pool, is the sliding average value representing the time delay of the time slot, is the sliding average value representing the time delay deviation of the time slot, is the sliding average value representing the packet loss rate of the time slot; After obtaining the combined sampling weights of all training samples, the softmax function is used for normalization to map the weights of each sample to the same probability space, thereby obtaining the sampling probability of each training sample.

[0013] Preferably, the end-side dynamic video compression rate adjustment method based on the bandwidth-oriented UCB algorithm includes: The bandwidth is divided into three scenarios, and an independent UCB algorithm is set for each scenario; The terminal selects the corresponding scenario based on the allocated bandwidth, and the scenario allocation method is as follows: , in, For video capture terminal bandwidth, For low-bandwidth scenarios, For high-bandwidth scenarios, the threshold is set. Calculate the preference value for each compression ratio action, and the preference value is calculated as follows: , in, For the terminal exist Selecting actions in a scene up to The preference value of the time slot, For the terminal exist Selecting actions in a scene up to Historical average reward for time slots For the terminal exist Selecting actions in a scene up to Number of times to select a time slot Explore weights for the algorithm; The terminal selects the action corresponding to the maximum preference value. Execute the action and update the corresponding statistics, including the historical average reward and number of selections for the action: , .

[0014] The advantages of this invention are as follows: By constructing a panoramic multi-video stream model based on end-edge collaborative optimization, this invention accurately characterizes the latency differences between different video streams, effectively avoiding the system performance degradation caused by differences in the transmission latency of individual video streams. By introducing latency deviation as a consistency indicator, the smoothness of panoramic image stitching is ensured, reducing stuttering, misalignment, and image distortion, significantly improving the overall smoothness and stability of the monitoring image.

[0015] The application effectively reduces data redundancy and reduces the demand for transmission bandwidth by dynamically adjusting the video compression rate on the terminal side. At the same time, the SAC algorithm is used to adaptively allocate bandwidth resources on the edge side to ensure efficient use of bandwidth. By constructing a relationship model between the compression rate and the decoding packet loss probability, and proposing a long-term packet loss constraint mechanism, the method further reduces the transmission delay while ensuring video quality.

[0016] In view of the limitations of single compression rate or single bandwidth allocation optimization in traditional methods, the application proposes a panoramic multi-video stream fusion method based on terminal-side compression rate optimization and edge-side bandwidth allocation coordination. By constructing a video information packet loss rate virtual queue, the packet loss rate and long-term delay constraint are converted into an optimization objective that can be solved in real time, realizing the dual guarantee of low delay and low packet loss.

[0017] The application introduces a multi-dimensional performance differentiated driving experience replay learning mechanism to dynamically sample the samples, realizing intelligent training sample selection and utilization. This mechanism ensures that the training process can focus on the current key performance bottleneck, improving the adaptability of the system in complex environments.

[0018] The application achieves significant results in improving picture smoothness and stability, optimizing bandwidth resource utilization, achieving low delay and low packet loss balance, enhancing the adaptability of the system to complex environments, and improving the panoramic video fusion viewing quality. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application.

[0020] Fig. 1 The application method implementation framework schematic diagram.

[0021] Fig. 2 The application method flowchart schematic diagram. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0023] Embodiment 1 As Figs. 1-2To improve the quality and stability of the power distribution network field operation panoramic multi-video stream fusion, enhance the operation safety and reliability, the application provides a low-latency high-definition panoramic multi-video stream fusion method, which regards the multi-camera end side as a data collection source, and regards the edge node as a fusion processing center, realizes the minimization of the transmission bandwidth constraint-based time delay and time delay deviation and the control of the packet loss rate by jointly modeling the end side compression strategy and the edge resource allocation, depicts the multi-dimensional relationship among the data volume, compression ratio, bandwidth allocation and decoding success rate, avoids the complexity of traditional frame-by-frame calculation, and ensures the low-latency fusion and high reliability of the high-definition video stream.

[0024] Specifically, the following steps are included: S1: The end side collects video data through a video collection terminal, extracts key frames through a semantic compression strategy, and each video collection terminal maintains a local data queue to reflect data accumulation; S2: A transmission time delay model is constructed on the edge side, a time delay deviation index is introduced to evaluate the transmission consistency of different video streams, a decoding packet loss rate probability model is constructed, and the relationship between the compression ratio and the packet loss rate is established; S3: An optimization problem with the minimum total time delay and time delay deviation weighted sum as the target is constructed, and the bandwidth resource constraint and the long-term packet loss rate constraint are combined; S4: The optimization problem is converted into a real-time optimization target by constructing a virtual queue; S5: The SAC algorithm is used for edge bandwidth resource allocation; S6: The bandwidth-oriented UCB algorithm is used to dynamically adjust the end side video compression ratio, and the edge side allocated bandwidth is used to dynamically select the adaptive scene.

[0025] As a refinement of the above embodiment, the step S1 includes a plurality of video collection terminals on the end side, which are responsible for collecting high-definition video data, and these data need to be uploaded to the edge node in real time for splicing and rendering. The total optimization period of the system is divided into equal time slots, denoted as a set , and the length of each time slot is . Considering video collection terminals, denoted as a set .

[0026] The end side compression is the key to reducing the data volume, which directly affects the transmission load and the quality trade-off. The application adopts a semantic compression strategy to extract key frame features and remove redundant information. The video collection terminal has a compression ratio in time slots, which is represented by [0, 1], and the larger the value, the higher the compression degree.

[0027] Each video collection terminal maintains a local data queue , reflecting the local data acquisition backlog. In time slot The difference between the amount of acquisition and transmission is considered, expressed as: , Where, is the amount of raw video data collected by the video acquisition terminal in the first time slot. is the actual amount of data transmitted.

[0028] Considering the transmission limit between the terminal and the edge, the actual data transmission amount of the video acquisition terminal is expressed as , Where, represents the transmission rate between the terminal and the edge in the first time slot.

[0029] The edge node is responsible for bandwidth allocation and panoramic video fusion processing. The transmission model is constructed based on Shannon's law. Let the total bandwidth be , the bandwidth allocated to the video acquisition terminal be , and the transmission rate of the video acquisition terminal be: , Where, is the channel gain, is the transmission power of the video acquisition terminal , is the noise spectral density, is the interference power.

[0030] As a refinement of the above embodiment, the data transmission delay of the video acquisition terminal in step S2 can be characterized by its local data queue length and average acquisition data amount, expressed as: , Where, is the average data acquisition amount of the video acquisition terminal at the end of the cutoff time slot.

[0031] In the panoramic multi-video stream fusion scene, due to the differences in data volume, compression ratio, network bandwidth occupation and packet loss of different video streams, if only the average delay is concerned, the transmission delay of individual video stream is likely to be significantly higher than that of other streams, which further leads to the lag, misalignment or even distortion of the panoramic picture during splicing. To depict this inconsistency, the present application introduces the delay deviation as an evaluation index, which is defined as the variance between the delay of each video stream in the system and the average delay thereof. By minimizing the delay deviation, the transmission consistency of different video streams can be effectively improved, and the individual video stream can be avoided to become a system bottleneck, so as to improve the fluency and stability of panoramic video fusion, and meet the transmission requirements of low delay and high reliability. The delay deviation is expressed as: , wherein, is the average delay of the time slot.

[0032] As a refinement of the above embodiment, in the step S2, the edge node decodes the compressed video to restore the original video stream for panoramic fusion. When the compression ratio is larger, the packet loss probability is larger, and the packet loss probability of the data decoding is defined as time slot terminal data decoding packet loss probability is expressed as , wherein, is the packet loss parameter, which can be obtained by least square fitting according to historical data.

[0033] As a refinement of the above embodiment, in the step S3, the optimization problem of minimizing the weighted sum of the total delay and the delay deviation as the target, and the bandwidth resource constraint and the long-term packet loss rate constraint are combined, which is as follows: , wherein, is the weighting coefficient of the total delay sum and the delay deviation. is the bandwidth resource constraint, is the total bandwidth, is the compression ratio selection constraint, is the long-term packet loss rate constraint, is the packet loss rate threshold, is the total number of time slots divided by the total optimization period.

[0034] ​The system of the application systematically integrates key links such as data acquisition, compression processing, transmission bandwidth allocation and decoding. In the modeling process, the difference of the data stream of each camera is fully considered, and a time delay deviation model is introduced to describe the time delay difference between different video streams, so as to ensure the stability and consistency of the global picture. At the same time, a decoding packet loss model is established to quantify the risk of packet loss under high compression rate conditions and control it through a long-term constraint mechanism, thereby effectively avoiding key frame loss and panorama image quality degradation.

[0035] As a refinement of the above embodiment, there is a coupling relationship between the long-term stability constraint and the short-term strategy in step S4, and it is difficult to directly solve the optimization problem. The application converts the long-term constraint into a form that can be processed in real time by constructing a virtual queue of video information packet loss rate, thereby decoupling the optimization of short-term decision. Video acquisition terminal Virtual queue The evolution mode is represented as: , When the virtual queue reaches stability, the long-term constraint can be satisfied, so the optimization problem can be converted to , In the formula, represents the weight of the original optimization target.

[0036] As a refinement of the above embodiment, the SAC algorithm is used to solve the edge side bandwidth resource allocation problem in step S5, and the state space of the edge side bandwidth resource allocation method based on the SAC algorithm is represented as: , The action space is: , The reward function is Set the optimization target P2 as: .

[0037] In the policy optimization process, SAC uses a maximum entropy objective function to output the conditional probability distribution of the bandwidth allocation strategy through a policy network (Actor), and combines a double Q-value network (Critic) to improve the balance between exploration and utilization of the strategy.

[0038] In order to enhance the optimization ability of SAC in a dynamic network environment, the application further proposes an experience replay learning mechanism based on multi-dimensional performance differentiation driving.

[0039] Including: (1) The edge side is each training sample An additional multi-dimensional performance label is added, and the training sample with the label is represented as: , wherein, is the traditional experience sample data, is the current state, is the current action, is the current reward, is the next state, is the time delay label of the sample, is the time delay deviation label of the sample, is the packet loss rate label of the sample; (2) The edge side establishes a corresponding experience pool according to the time delay, time delay deviation and packet loss rate, and when a performance index of a training sample is greater than a threshold value, the sample is stored in the corresponding experience pool, so as to improve the sample experience utilization rate and simultaneously consider the training of multiple performance indexes; the edge side calculates the time delay, time delay deviation and packet loss rate label domain weight of the sample respectively, and is represented as: , , , wherein, is the maximum sample time delay in the experience pool, is the maximum time delay deviation in the experience pool, is the maximum packet loss rate in the experience pool, and the greater the label domain weight of the sample is, the better the performance of the sample in the index is (3) In the SAC algorithm training of the present application, the dynamic weight of each index is calculated according to the recent performance, the sample is drawn from each experience pool according to the dynamic weight, and the differential experience playback is performed. In order to unify the order of magnitude of different indexes, the values of the three performance indexes are normalized, and the mode is to divide the current value of the performance index by the historical maximum value. represents the sliding average value of the time slot index , which represents the normalized average performance of the index in the recent period, so as to reflect the dynamic change trend of the index. The dynamic sampling weight is represented as: , , , wherein, is the dynamic sampling weight of the time delay, is the dynamic sampling weight of the time delay deviation, is the dynamic sampling weight of the packet loss rate, is the the sliding average value of the slot delay, denotes the sliding average value of the slot delay deviation, denotes the sliding average value of the slot packet loss rate. The sliding average value is calculated by a fixed window sliding average method, that is, the latest slots are selected as the observation window, the indicators in the window are normalized and the arithmetic average value is taken.

[0040] When is small, it indicates that the recent delay performance indicator performs well, the dynamic sampling weight of the delay indicator is reduced, and the probability of extracting the delay sample is reduced, so as to shift the training focus to other relatively weak or urgently optimized performance indicators.

[0041] (4) Calculate the comprehensive sampling weight of the training sample based on the attached label and the experience pool : , wherein, is the dynamic sampling weight of the delay, is the delay label domain weight, is the dynamic sampling weight of the delay deviation, is the delay deviation label domain weight, is the dynamic sampling weight of the packet loss rate, is the packet loss rate label domain weight, is the maximum sample delay in the experience pool, is the maximum delay deviation in the experience pool, is the maximum packet loss rate in the experience pool, denotes the sliding average value of the slot delay, denotes the sliding average value of the slot delay deviation, denotes the sliding average value of the slot packet loss rate.

[0042] After obtaining the comprehensive sampling weight of all training samples, the softmax function is used for normalization processing, and the sample weight is mapped to the same probability space, so as to obtain the sampling probability of each training sample.

[0043] An adaptive allocation method of sideband bandwidth resources based on SAC algorithm is proposed, and through the experience replay learning mechanism based on multi-dimensional performance differentiation driving, dynamic sampling is performed on the samples, intelligent training sample selection and utilization are realized, and the training process is focused on the current key performance bottleneck.

[0044] As a refinement of the above embodiments, in step S6, to differentiate and adapt to the end-side video compression requirements under different bandwidth scenarios, this invention proposes an end-side dynamic video compression rate adjustment method based on the bandwidth-oriented UCB algorithm. This method sets three scenarios... These correspond to low bandwidth, medium bandwidth, and high bandwidth, respectively. Under different bandwidth conditions, the system automatically selects the matching UCB algorithm for solving the problem, thereby dynamically adjusting the video compression rate under different scenarios. The scenario division method is represented as follows: , In the formula, , These are the bandwidth scene thresholds.

[0045] in, For video capture terminal bandwidth, For low-bandwidth scenarios, The threshold is set for high-bandwidth scenarios; the threshold is preset based on the system's available bandwidth and service quality requirements, thereby enabling adaptive selection for different scenarios.

[0046] Calculate the preference value for each compression ratio action, and the preference value is calculated as follows: , in, For the terminal exist Selecting actions in a scene up to The preference value of the time slot, For the terminal exist Selecting actions in a scene up to Historical average reward for time slots For the terminal exist Selecting actions in a scene up to Number of times to select a time slot Explore weights for the algorithm; The terminal selects the action corresponding to the maximum preference value. Execute the action and update the corresponding statistics, including the historical average reward and number of selections for the action: , .

[0047] By constructing multiple sets of UCB scenarios corresponding to different bandwidth conditions, a dynamic selection of an adaptive scenario is performed according to edge bandwidth allocation, so as to improve the delay performance and transmission efficiency of video streams under limited bandwidth. Through end-edge collaborative optimization, the method breaks through the limitations of single compression rate or single bandwidth allocation optimization, realizes the balance of low delay and low packet loss, and improves the stability and viewing quality of panoramic multi-video stream fusion.

[0048] Embodiment 2 The embodiments of the present disclosure also provide a low-latency high-definition panoramic multi-video stream fusion device, comprising a processor and a memory. Optionally, the device can also comprise a communication interface and a bus. Wherein, the processor, the communication interface and the memory can complete mutual communication through the bus. The communication interface can be used for information transmission. The processor can call the logical instructions in the memory to execute the low-latency high-definition panoramic multi-video stream fusion method of the above-mentioned embodiments.

[0049] In addition, the logical instructions in the memory described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0050] The memory as a computer readable storage medium can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiments of the present disclosure. The processor executes the program instructions / modules stored in the memory to perform function applications and data processing, that is, to implement the low-latency high-definition panoramic multi-video stream fusion method in the above-mentioned embodiments.

[0051] The memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory.

[0052] The embodiments of the present disclosure provide a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are set to execute the above-mentioned low-latency high-definition panoramic multi-video stream fusion method.

[0053] The above-mentioned computer readable storage medium can be a transitory computer readable storage medium, or a non-transitory computer readable storage medium.

[0054] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes one or more instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, and can also be a transitory storage medium.

[0055] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and is not used to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, and for those skilled in the art, the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A low-latency, high-definition panoramic multi-video stream fusion method, characterized in that, Includes the following steps: The device collects video data through video capture terminals and extracts key frames through semantic compression strategies. Each video capture terminal maintains a local data queue to reflect data backlog. A transmission delay model is constructed at the edge, and a delay deviation index is introduced to evaluate the transmission consistency of different video streams. A decoding packet loss rate probability model is constructed, and the relationship between compression rate and packet loss rate is established. We construct an optimization problem with the objective of minimizing the weighted sum of total latency and latency deviation, and combine bandwidth resource constraints with long-term packet loss rate constraints. By constructing a virtual queue, the optimization problem is transformed into a real-time optimization objective; The SAC algorithm is used for edge bandwidth resource allocation; The bandwidth-oriented UCB algorithm dynamically adjusts the video compression rate on the edge side and dynamically selects the appropriate scenario based on the bandwidth allocated on the edge side.

2. The low-latency high-definition panoramic multi-video stream fusion method according to claim 1, characterized in that, The local data queue on the endpoint is in the time slot The evolution of the difference between the acquisition amount and the transmission amount is calculated using the following formula: ， ， in, Indicates a local data queue. Indicates compression ratio. For the first Time-slot video acquisition terminal The amount of raw video data collected. This represents the actual amount of data transmitted. Indicates the time slot length. No. Time slot terminal Transmission rate between the edge and the edge; The first Time slot terminal The transmission rate between the edge and the device is: ， in, For video capture terminal bandwidth, For channel gain, For video capture terminal Transmission power, For noise spectral density, This represents the interference power.

3. The low-latency high-definition panoramic multi-video stream fusion method according to claim 2, characterized in that, The transmission delay model is as follows: ， ， in, for Time-slot video acquisition terminal Data transmission latency, For the deadline Video capture terminal at the end of time slot Average data collection volume No. Time-slot video acquisition terminal Average data collection volume; The time delay deviation The variance of the latency of each video stream compared to the average latency: ， in, A collection of video capture terminals. For the first One video capture terminal, for Average time delay of time slot For the first Data transmission latency of a video capture terminal; The decoding packet loss rate probability model is as follows: in, for Time slot terminal The probability of packet loss during data decoding. , , , The packet loss rate parameter was obtained by fitting historical data using the least squares method. It is a natural constant.

4. The low-latency high-definition panoramic multi-video stream fusion method according to claim 3, characterized in that, The optimization problem, which aims to minimize the weighted sum of total latency and latency deviation, and is combined with bandwidth resource constraints and long-term packet loss rate constraints, is as follows: ， in, This is the total delay and the tradeoff factor between the delay and the deviation. Due to bandwidth resource constraints, Total bandwidth Choose constraints for compression ratio. To constrain long-term packet loss rate, The packet loss rate threshold. The total number of time slots divided for the total optimization cycle.

5. The low-latency high-definition panoramic multi-video stream fusion method according to claim 4, characterized in that, The virtual queue Represented as: ， When virtual queue Once stability is achieved, the long-term constraints are satisfied, and the transformed real-time optimization objective is: ， In the formula, The weights represent the original optimization objective.

6. The low-latency high-definition panoramic multi-video stream fusion method according to claim 5, characterized in that, The state space of the side bandwidth resource allocation method based on the SAC algorithm Represented as: ， The motion space is: , The reward function is Set the optimization target as P2. The maximum entropy objective function is adopted.

7. The low-latency high-definition panoramic multi-video stream fusion method according to claim 6, characterized in that, The SAC algorithm employs an experience replay learning mechanism driven by multidimensional performance differentiation, including: The edge side is for each training sample Additional multidimensional performance labels are added, and the labeled training samples are represented as follows: ， in, For traditional empirical sample data, This is the current state. For the current action, For the current reward, For the next state, For the time delay label of the sample, The time delay deviation label for the sample. The packet loss rate label for the sample; On the edge side, corresponding experience pools are established based on latency, latency deviation, and packet loss rate. When a certain performance index of a training sample exceeds a threshold, it is stored in the corresponding experience pool. Training samples are calculated based on attached labels and experience pools. Comprehensive sampling weight : ， ， ， ， ， ， ， in, For the dynamic sampling weight of the time delay, For the time-delay label field weight, The dynamic sampling weights for the time delay bias, For the time delay deviation label field weight, The dynamic sampling weights for packet loss rate For packet loss rate label domain weight, The maximum sample latency in the experience pool. The maximum time delay deviation in the experience pool. The maximum packet loss rate in the experience pool. To indicate The moving average of the time slot delay, express The moving average of the time slot delay deviation, express The moving average of the time slot packet loss rate; After obtaining the combined sampling weights of all training samples, the softmax function is used for normalization to map the weights of each sample to the same probability space, thereby obtaining the sampling probability of each training sample.

8. The low-latency high-definition panoramic multi-video stream fusion method according to any one of claims 1-7, characterized in that, The end-side dynamic video compression rate adjustment method based on the bandwidth-guided UCB algorithm includes: The bandwidth is divided into three scenarios, and an independent UCB algorithm is set for each scenario; The terminal selects the corresponding scenario based on the allocated bandwidth, and the scenario allocation method is as follows: ， in, For video capture terminal bandwidth, For low-bandwidth scenarios, For high-bandwidth scenarios, the threshold is set. Calculate the preference value for each compression ratio action, and the preference value is calculated as follows: ， in, For the terminal exist Selecting actions in a scene up to The preference value of the time slot, For the terminal exist Selecting actions in a scene up to Historical average reward for time slots For the terminal exist Selecting actions in a scene up to Number of times to select a time slot Explore weights for the algorithm; The terminal selects the action corresponding to the maximum preference value. Execute the action and update the corresponding statistics, including the historical average reward and number of selections for the action: ， 。 9. A low-latency, high-definition panoramic multi-video stream fusion device, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the low-latency high-definition panoramic multi-video stream fusion method as described in any one of claims 1-8 when running the program instructions.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the low-latency high-definition panoramic multi-video stream fusion method as described in any one of claims 1-8.

Images