Attention-based multi-channel video low-delay adaptive method, medium and equipment

Abstract

The invention discloses an attention-based multi-channel video low-delay adaptive method, a medium and equipment. The method comprises the following steps: initializing system resources; receiving video stream data input of multiple cameras; monitoring and analyzing the attention quality and the network transmission effect of the video stream data of each path of current camera, and solving a constraint problem according to the attention quality and the network transmission effect of the video stream data of each path of camera so as to optimize resource allocation parameters; and adjusting the system resources of the video stream data of each path of camera according to the optimized resource allocation parameters so as to reduce the video transmission delay. According to the method, the resource allocation parameters of the video stream data are optimized by solving the constraint problem, the video quality of important scenes and events can be ensured, and the effectiveness of monitoring and remote operation is improved. Through intelligent resource allocation, the bandwidth and computing resource requirements are reduced while the performance is ensured.

Description

Technical Field

[0001] The present application relates to the field of video stream data processing, and specifically to an attention-based multi-channel video low-latency adaptive method, medium and device. Background Art

[0002] In modern video surveillance and remote operation systems, low-latency and high-quality transmission of multi-channel videos is a key challenge. Especially in the fields of industry, security, marine, and medical care, the real-time and quality requirements of video transmission are extremely high. However, traditional video transmission systems face problems such as bandwidth limitations, high latency, single transmission strategy, dynamic scene changes, and network environment. The simultaneous transmission of multiple channels of high-definition video is limited by network bandwidth, which poses a huge challenge to network bandwidth. There is a trade-off between real-time requirements and video quality, and low latency is usually at the expense of video quality. The importance of content captured by different cameras varies, but traditional systems often adopt a unified and single transmission strategy. The adaptability of multi-channel high-definition video transmission to dynamic scenes cannot be solved. The complexity and importance of the scene change over time, and fixed transmission parameters are difficult to adapt to this dynamic nature. Fluctuations in network conditions can lead to unstable video transmission quality. Summary of the invention

[0003] In view of the above problems, the present application provides an attention-based multi-channel video low-latency adaptive method, medium and device to solve the problems that the existing video transmission strategy cannot adapt to the needs of dynamic scenes, the video transmission quality is low and unstable.

[0004] To achieve the above objectives, in a first aspect, the present application provides an attention-based multi-channel video low-latency adaptive method, the method comprising the following steps:

[0005] Initialize system resources;

[0006] Receive video stream data input from multiple cameras;

[0007] Monitor and analyze the attention quality and network transmission effect of the current video stream data of each camera, and solve the constraint problem according to the attention quality and network transmission effect of the video stream data of each camera to optimize the resource allocation parameters;

[0008] The system resources of the video stream data of each camera are adjusted according to the optimized resource allocation parameters.

[0009] Further, the initializing system resources includes:

[0010] Load system configuration parameters, initialize network connection and communication protocol, and set initial encoding parameters;

[0011] Establish connections with multiple cameras and configure the initial encoding parameters of each camera.

[0012] Furthermore, the video stream data includes a plurality of video frame images, and monitoring and analyzing the attention quality and network transmission effect of the current video stream data of each camera includes:

[0013] Use the first neural network model to process the decoded video stream data, generate an attention map of each frame of the video stream image, and obtain the area of ​​the attention region through binarization processing;

[0014] It also measures the real-time bandwidth usage of each video stream data, calculates the end-to-end transmission delay, and records the network jitter and packet loss rate.

[0015] Furthermore, the decoded video stream data is processed using the first neural network model to generate an attention map of each frame of the video stream image, including:

[0016] The attention density function A(x, y) is used to represent the attention value of each pixel (x, y) on the video frame image. The calculation formula is as follows: Get the attention map of each frame of the video stream image;

[0017] The method further comprises:

[0018] The pixel values ​​of the attention map of the video stream image are scaled to the range [0, 1], and the mean and standard deviation normalization is applied to the color channel of each pixel.

[0019] Furthermore, the area of ​​the attention region is obtained through binarization processing, including:

[0020] Determine whether the attention value of each pixel point of the video frame image is greater than the preset attention threshold. If so, assign the pixel point a value of 1, otherwise assign the pixel point a value of 0, where 1 represents a high attention area and 0 represents a low attention area;

[0021] Apply dilation or erosion operations to smooth the binarization result, identify and mark continuous high-attention areas, calculate the sum of all pixel points representing the high-attention areas, and obtain the area of ​​the attention area.

[0022] Furthermore, solving constraint problems based on the attention quality of the video stream data of each camera and the network transmission effect to optimize resource allocation parameters includes:

[0023] Input the attention quality and network transmission effect of the video stream data of each camera into the trained second neural network model, predict the attention quality and network transmission effect of the current video stream data in a subsequent preset time period based on the trained second neural network model, and output optimized resource allocation parameters based on the predicted attention quality and network transmission effect;

[0024] The second neural network model is trained according to the following method:

[0025] Acquire sample historical data, where the sample historical data includes attention quality and network transmission effect corresponding to the sample video stream data;

[0026] The sample historical data is input into the second neural network model to be trained, and the model parameters are iteratively updated according to the prediction results to obtain the trained second neural network model.

[0027] Furthermore, the constraint problem is designed as follows:

[0028] Define the following function:

[0029] Entropy function:

[0030]

[0031] Similarity constraint function:

[0032]

[0033] Total resource constraint function:

[0034]

[0035] Delay constraint function:

[0036]

[0037] Where n is the number of cameras, B i is the network transmission resource allocated to the i-th camera, P i is the proportion of the attention area of ​​the i-th camera, B total is the total number of available network transmission resources, L max is the maximum allowed delay, k is a constant;

[0038] The objective function of the constraint problem is designed as follows:

[0039] J=H(B)-λD KL (P|B);

[0040]

[0041] For all i,

[0042] For all i, B i ≥0;

[0043] Among them, λ is a trade-off parameter used to adjust the balance between entropy maximization and distribution similarity;

[0044] Expand this objective function:

[0045]

[0046] The last item is a constant;

[0047] The solution to the constraint problem includes:

[0048] The constrained problem is solved by the Lagrange multiplier method.

[0049] Furthermore, the resource allocation parameters include any one or more of the resolution, frame rate and compression level of the video stream data.

[0050] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the attention-based multi-channel video low-latency adaptive method as described in the first aspect of the present application.

[0051] In a third aspect, the present application provides an electronic device comprising a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the attention-based multi-channel video low-latency adaptive method as described in the first aspect of the present application is implemented.

[0052] Different from the prior art, the attention-based multi-channel video low-latency adaptive method, medium and device described in the above technical solution include: initializing system resources; receiving video stream data input from multiple cameras; monitoring and analyzing the attention quality and network transmission effect of the current video stream data of each camera, solving constraint problems according to the attention quality and network transmission effect of the video stream data of each camera to optimize resource allocation parameters; adjusting the system resources of the video stream data of each camera according to the optimized resource allocation parameters. This application can ensure the video quality of important scenes and events and improve the effectiveness of monitoring and remote operation by solving constraint problems to optimize the resource allocation parameters of video stream data. Through intelligent resource allocation, bandwidth and computing resource requirements are reduced while ensuring performance.

[0053] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present invention and other related contents, and shall not be considered as limiting the present application.

[0055] In the drawings of the specification:

[0056] Figure 1 This is a flowchart of the attention-based multi-channel video low-latency adaptive method involved in the first exemplary embodiment of the present application;

[0057] Figure 2 This is a flow chart of an attention-based multi-channel video low-latency adaptive method involved in a second exemplary embodiment of the present application;

[0058] Figure 3 This is a flowchart of an attention-based multi-channel video low-latency adaptive method according to a third exemplary embodiment of the present application;

[0059] Figure 4 This is a module diagram of an attention-based multi-channel video low-latency adaptive system involved in an exemplary embodiment of the present application;

[0060] Figure 5 A flowchart of network transmission delay estimation involved in an exemplary embodiment of the present application;

[0061] Figure 6 A flowchart of a discrete multi-objective bandwidth allocation optimization algorithm according to an exemplary embodiment of the present application;

[0062] Figure 7 A schematic diagram of a module of an electronic device according to a first exemplary embodiment of the present application;

[0063] Reference numerals:

[0064] 10. Electronic equipment;

[0065] 101. Processor;

[0066] 102. Storage medium. DETAILED DESCRIPTION

[0067] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0068] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0069] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0070] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0071] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0072] Without further limitations, in this application, the words "include", "comprises", "has" or other similar open-ended expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0073] In this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0074] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0075] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0076] In the first aspect, Figure 1 and Figure 4 As shown, in Figure 1 The present application provides an attention-based multi-channel video low-latency adaptive method, the method comprising the following steps:

[0077] Step S101: Initialize system resources;

[0078] Step S102: receiving video stream data input from multiple cameras;

[0079] Step S103: monitoring and analyzing the attention quality and network transmission effect of the video stream data of each camera at present, solving the constraint problem according to the attention quality and network transmission effect of the video stream data of each camera, so as to optimize the resource allocation parameters;

[0080] Step S104: adjusting the system resources of the video stream data of each camera according to the optimized resource allocation parameters.

[0081] In step S101, Figure 2 As shown, in some embodiments, the initializing system resources includes:

[0082] Step S201: Load system configuration parameters, initialize network connection and communication protocol, and set initial encoding parameters;

[0083] Step S202: Establish connections with multiple cameras and configure initial encoding parameters for each camera.

[0084] Preferably, the initial encoding parameters include any one or more of the resolution, frame rate and compression level of the video stream data.

[0085] In other embodiments, initializing system resources also includes loading a neural network model and initializing a data storage system. Loading the neural network model includes loading a pre-trained DINOv2 model and initializing model parameters and computing resources. Initializing the data storage system includes setting up a database (or file system) to store historical data and initializing a data structure to record attention quality and network performance indicators.

[0086] In step S102, after receiving the encoded video stream data collected by multiple cameras, a suitable decoder may be used to decode each video stream, and the decoded video frames may be cached for processing.

[0087] In step S103, Figure 3 As shown, the video stream data includes multiple video frame images, and monitoring and analyzing the attention quality and network transmission effect of the current video stream data of each camera includes:

[0088] Step S301: using the first neural network model to process the decoded video stream data, generating an attention map of each frame of the video stream image, and obtaining the area of ​​the attention region through binarization processing;

[0089] Step S302: measure the real-time bandwidth usage of each video stream data, calculate the end-to-end transmission delay, and record the network jitter and packet loss rate.

[0090] Specifically, the DINOv2 model can be used to process the decoded video frames, generate an attention map for each frame, and apply binarization processing to obtain the attention area S_i. In addition, the real-time bandwidth usage of each video stream can be measured, the end-to-end transmission delay can be calculated, and the network jitter and packet loss rate can be recorded. The attention quality data can be sampled at a preset frequency (such as every second or every 5 seconds), the network performance data at the corresponding time point can be recorded synchronously, and the sampled data can be stored as a multidimensional input vector, which records the timestamp, attention quality, network performance indicators, etc. At the same time, for the collected data, real-time charts can be generated to display the attention quality and network performance trends, and an operation interface can be provided for manual monitoring and intervention.

[0091] In step S103, based on the current network status and the importance of each video channel, the constraint problem is solved to optimize the resource allocation parameters. The output of the resource optimization configuration is the following three parameters: resolution (Resolution), represented by the letter R; frame rate (Frame Rate), represented by the letter f, compression level / quality parameter (Compression Level), represented by the letter q. The solving process specifically includes the following steps:

[0092] (1) Data preprocessing

[0093] Specifically, it retrieves historical data from the storage system over a recent period of time and performs data cleaning and standardization.

[0094] (2) Initial stage optimization

[0095] Specifically, the objective function calculation module is implemented, an improved genetic algorithm or other heuristic algorithm is designed and implemented, and the algorithm is used to solve the local optimal video encoding parameters, and the optimization results are stored as training data.

[0096] In the initial stage of system operation, a greedy heuristic algorithm is used to solve the local optimal video encoding parameters to provide training data for machine learning. The specific calculation formula is as follows:

[0097]

[0098] Among them, f(D delay，i ) is a nonlinear penalty function for delay.

[0099] (3) Neural network training and optimization

[0100] Specifically, it is to design a neural network structure (such as LSTM or Transformer) suitable for the problem, train the neural network using accumulated historical data, and regularly update the model to adapt to new data patterns.

[0101] (4) Real-time parameter optimization

[0102] Specifically, the trained neural network is used to predict the attention distribution and network status in the short term in the future, and the optimal coding parameter combination is calculated based on the prediction results.

[0103] (5) Decision output

[0104] Generate new encoding parameters for each video stream: resolution (R), frame rate (f), compression level (q), perform rationality checks to ensure that the parameters are within the allowed range, and prepare parameter update instructions.

[0105] In step S104, the system resources of the video stream data of each camera are adjusted according to the optimized resource allocation parameters, specifically including:

[0106] (1) Parameter update preparation

[0107] Specifically, for cameras whose parameters need to be changed, the optimized encoding parameters are converted into an instruction format acceptable to the camera, and an independent update command is generated for each camera.

[0108] (2) Camera communication

[0109] Specifically, it establishes a secure connection with a specific IPC camera, sends parameter update instructions, and receives and verifies the camera's confirmation response.

[0110] (3) Application of Coding Parameters

[0111] Specifically, the camera receives the new parameters and reconfigures the encoder, and uses the new parameters for video encoding.

[0112] (4) Performance Monitoring

[0113] Specifically, it monitors the encoding quality under the new parameters in real time, measures the encoding delay and resource usage, and records the performance comparison data before and after the parameter changes.

[0114] (5) Feedback loop

[0115] Specifically, it collects short-term performance data after the new parameters are applied, and feeds this data back to the optimization module for the next round of optimization.

[0116] (6) Exception handling

[0117] Specifically, it sets monitoring thresholds, detects abnormal situations after parameter updates, implements a rollback mechanism, and restores to the last stable configuration when serious problems occur.

[0118] (7) Logging

[0119] Specifically, it records the content, time and effect of each parameter update in detail, and generates a performance report for subsequent analysis and system improvement.

[0120] The video frame attention area map (attention-map) involved in the attention quality assessment of the method of the present application mainly covers the following identification features and categories, including equipment and structure features, container features, vehicle features, personnel features, dangerous area features, cargo status features, safety equipment features, operating status features, potential danger features, communication and control features, and maintenance-related features.

[0121] For the scenario of port container operation, more details need to be considered to ensure safety and comprehensiveness. For example:

[0122] (1) Equipment and structure identification features

[0123] Gantry crane structure: 'crane', 'gantry crane', 'bridge', 'steel structure' categories;

[0124] Lifting device: 'hook', 'spreader', 'lifting device' categories;

[0125] Control room: 'cabin', 'control room' category;

[0126] (2) Container identification features:

[0127] In addition to 'container ship' and 'storage tank', 'cargo container' and 'freight container' are also considered;

[0128] Classification by container status: 'stacked containers', 'single container', 'open top container';

[0129] Container size: Identify '20-foot container', '40-foot container'.

[0130] (3) Extended vehicle identification features:

[0131] In addition to 'car' and 'truck', add 'forklift', 'reach stacker', 'terminaltractor', 'straddle carrier' categories;

[0132] Specially added large port machinery categories: 'rubber tyred gantry crane', 'rail mounted gantry crane'.

[0133] (4) Personnel safety features:

[0134] Personnel categories: 'worker', 'dock worker', 'crane operator', 'pedestrian';

[0135] Safety equipment categories: 'hard hat', 'safety vest', 'safety boots'.

[0136] (5) Hazardous area identification features:

[0137] Ground markings: 'road marking', 'safety line', 'warning sign' categories;

[0138] Aerial work area: 'elevated platform', 'ladder', 'scaffolding' categories.

[0139] (6) Cargo status characteristics:

[0140] In addition to containers, it includes 'bulk cargo', 'liquid cargo' and 'break bulk cargo' categories;

[0141] Cargo movement status: 'lifted cargo', 'suspended load', 'moving cargo' categories.

[0142] (7) Safety equipment features:

[0143] Fire fighting equipment: 'fire extinguisher', 'fire hydrant', 'sprinkler system' categories;

[0144] Emergency equipment: 'life buoy', 'emergency exit', 'first aid kit' categories.

[0145] (8) Operation status identification features:

[0146] Lifting process: 'lifting operation', 'lowering operation' categories;

[0147] Horizontal movement: 'trolley movement', 'gantry movement' categories.

[0148] (9) Potentially hazardous characteristics:

[0149] Obstacles: 'obstacle', 'debris', 'uneven surface' categories;

[0150] Electrical hazards: 'power line', 'electrical cabinet', 'transformer' categories.

[0151] (10) Communication and control system characteristics:

[0152] Traffic lights: 'traffic light', 'warning light', 'indicator' categories;

[0153] Control panel: 'control panel', 'display screen', 'button' categories.

[0154] (11)Maintenance related features:

[0155] Maintenance area: 'maintenance area', 'repair shop' categories;

[0156] Tools and equipment: 'tool box', 'spare parts', 'maintenance vehicle' categories.

[0157] For the scenario of port container operation, DINOv2 is used to extract attention maps to ensure security and comprehensiveness. DINOv2 has the following advantages:

[0158] (1) Multi-scale feature recognition capability

[0159] DINOv2’s multi-scale feature learning capability is well suited for port environments. It can simultaneously capture large structures (e.g., gantry cranes, container stacks) and small objects (e.g., workers, safety equipment). This is critical for comprehensive recognition of a wide range of elements, from large machinery to personal protective equipment.

[0160] (2) High resolution support

[0161] DINOv2 supports higher resolution input images, which is very beneficial for detailed analysis of port scenes. The high resolution capability helps to identify details such as safety signs, container numbers, workers' safety equipment, etc.

[0162] (3) Powerful semantic understanding

[0163] DINOv2 demonstrates strong semantic understanding capabilities, which are important for distinguishing and identifying multiple elements in a port environment. For example, it can better distinguish between different types of containers, vehicles, and machinery, such as distinguishing between regular containers and open-top containers, or identifying different types of port vehicles.

[0164] (4) Improved self-attention mechanism

[0165] DINOv2’s improved self-attention mechanism helps generate more refined and accurate attention maps. This is particularly useful for identifying dangerous areas, operating states, and potential risk points. For example, it can more accurately focus on areas where lifting operations are ongoing or aerial work platforms.

[0166] (5) Strong generalization ability

[0167] Because it is trained on a larger and more diverse dataset, DINOv2 performs better when faced with complex port environments. This generalization ability helps it recognize a variety of different equipment, structures, and situations, even specific port elements that the model was not directly exposed to during training.

[0168] (6) Accurate feature representation

[0169] DINOv2 is able to generate more precise and meaningful feature representations. This is important for identifying subtle safety-related details, such as whether a worker is wearing safety gear correctly, or identifying safety markings and warning signs on the ground.

[0170] (7) Computational efficiency

[0171] Despite its powerful capabilities, DINOv2 remains computationally efficient when generating attention maps, which is critical for real-time monitoring of port operations to quickly identify potential safety hazards and anomalies.

[0172] (8) Adapt to complex scenarios

[0173] The port environment is complex and ever-changing, and DINOv2 is designed to better adapt to this complexity. It can handle multiple concerns at the same time, such as equipment identification, personnel safety, and hazardous area identification, providing comprehensive scene understanding.

[0174] (9) Advantages of unlabeled learning

[0175] As a self-supervised learning model, DINOv2 can produce meaningful attention maps without the need for large amounts of labeled data. This is particularly beneficial in port environments, where acquiring large amounts of labeled data can be time-consuming and expensive.

[0176] (10) Dynamic scene adaptability

[0177] DINOv2 is better able to adapt to dynamically changing scenarios, which is highly consistent with the characteristics of the port environment. It can effectively capture scene changes under different operating conditions, such as lifting processes, vehicle movement, etc. In short, these characteristics of DINOv2 make it an ideal choice for monitoring port container operation scenarios, and it can comprehensively and accurately identify various safety-related elements, thereby improving the safety and efficiency of overall operations.

[0178] In some embodiments, processing the decoded video stream data using the first neural network model to generate an attention map for each frame of the video stream image includes:

[0179] The attention density function A(x, y) is used to represent the attention value of each pixel (x, y) on the video frame image. The calculation formula is as follows: Get the attention map of each frame of the video stream image;

[0180] The method further comprises:

[0181] The pixel values ​​of the attention map of the video stream image are scaled to the range [0, 1], and the mean and standard deviation normalization is applied to the color channel of each pixel.

[0182] The normalization calculation formula is as follows: normalized_image = (original_image / 255.0-mean) / std;

[0183] mean = [0.485, 0.456, 0.406] (corresponding to R, G, B channels respectively);

[0184] std = [0.229, 0.224, 0.225].

[0185] The attention density function proposed above makes the pixel value distribution of different images more consistent, which helps the generalization ability and training stability of the model.

[0186] Furthermore, the area of ​​the attention region is obtained through binarization processing, including:

[0187] Determine whether the attention value of each pixel point of the video frame image is greater than the preset attention threshold. If so, assign the pixel point a value of 1, otherwise assign the pixel point a value of 0, where 1 represents a high attention area and 0 represents a low attention area;

[0188] Apply dilation or erosion operations to smooth the binarization result, identify and mark continuous high-attention areas, calculate the sum of all pixel points representing the high-attention areas, and obtain the area of ​​the attention area.

[0189] When outputting the attention region, the specific output is a two-dimensional matrix (tensor), not an image. The size of this matrix is ​​usually [num_patches x num_patches], where num_patches is the number of patches into which the image is divided. For example, for a 518x518 input image, using a 14x14 patch size, you will get a 37x37 attention matrix. Each element represents the attention weight of one patch to another patch. The 37x37 matrix is ​​then enlarged to 518x518 through bilinear interpolation to make it consistent with the original image size.

[0190] Binarization is the process of converting continuous attention values ​​into binary (0 or 1). This is very useful for highlighting the most important areas. At this stage, we define the attention region R as all points where the integrated attention model exceeds a certain threshold θ:

[0191] R={(x,y)|A(x,y)>θ}:

[0192] The threshold θ can be set empirically (for example, the value θ=0.01) or dynamically adjusted through an adaptive algorithm.

[0193] The area of ​​the attention region can be calculated by the following integral:

[0194]

[0195] In a discrete digital image, this can be simplified to pixel counting:

[0196]

[0197] To compare between images of different sizes, we define the normalized attention score:

[0198]

[0199] Where W and H are the width and height of the image respectively.

[0200] The binary Attention Map output provides a concise way to represent the areas that the model considers most important, facilitating further analysis and decision making. It can help quickly identify areas that require manual review or additional attention, and is particularly suitable for alarm triggering mechanisms in automated monitoring systems.

[0201] In some embodiments, for monitoring scenarios, the ViT-Base model can be used to extract features. ViT-Base has the following advantages:

[0202] Balancing performance and efficiency: ViT-Base provides a good balance of performance and computational efficiency. This is an important consideration for real-time monitoring.

[0203] Sufficient feature complexity: Captures detailed features of port scenes, such as containers, vehicles, and people.

[0204] Moderate resource requirements: Compared with ViT-Large, ViT-Base has lower resource requirements and is suitable for long-running monitoring systems.

[0205] Real-time performance: On most modern GPUs, ViT-Base can achieve near real-time processing speeds.

[0206] In some embodiments, solving constraint problems based on the attention quality of the video stream data of each camera and the network transmission effect to optimize resource allocation parameters includes:

[0207] Input the attention quality and network transmission effect of the video stream data of each camera into the trained second neural network model, predict the attention quality and network transmission effect of the current video stream data in a subsequent preset time period based on the trained second neural network model, and output optimized resource allocation parameters based on the predicted attention quality and network transmission effect;

[0208] The second neural network model is trained according to the following method:

[0209] Acquire sample historical data, where the sample historical data includes attention quality and network transmission effect corresponding to the sample video stream data;

[0210] The sample historical data is input into the second neural network model to be trained, and the model parameters are iteratively updated according to the prediction results to obtain the trained second neural network model.

[0211] It is known that the average area of ​​attention area of ​​each camera in a specific time period is S_i.

[0212]

[0213] Now we need to allocate network transmission resources as much as possible while ensuring low latency, so that the distribution of B is as similar as possible to P. We use the maximum entropy principle to construct this objective function. The maximum entropy principle ensures that known constraints are met while making no assumptions about unknown information.

[0214] The constraint problem is designed as follows:

[0215] Define the following function:

[0216] Entropy function:

[0217]

[0218] Similarity constraint function:

[0219]

[0220] Total resource constraint function:

[0221]

[0222] Delay constraint function:

[0223]

[0224] Where n is the number of cameras, B i is the network transmission resource allocated to the i-th camera, P i is the proportion of the attention area of ​​the i-th camera, B total is the total number of available network transmission resources, L max is the maximum allowed delay, k is a constant;

[0225] The objective function of the constraint problem is designed as follows:

[0226] J=H(B)-λD KL (P|B);

[0227]

[0228] For all i,

[0229] For all i, B i ≥0;

[0230] Among them, λ is a trade-off parameter used to adjust the balance between entropy maximization and distribution similarity;

[0231] Expand this objective function:

[0232]

[0233] The last item is a constant;

[0234] This optimization problem can be solved by Lagrange multiplier method or other numerical optimization methods. Solving this problem will give a resource allocation B that is as close to the attention distribution P as possible while meeting the low latency requirement.

[0235] The characteristics of this objective function are: it encourages the resource allocation B to be as uniform as possible (by maximizing entropy). At the same time, it also encourages the distribution of B to be close to the distribution of P (by minimizing KL divergence), which ensures that the total resource constraints and delay constraints are met. By adjusting the λ parameter, a balance can be achieved between the uniformity of resource allocation and the similarity to the attention distribution. Larger λ values ​​will make the allocation closer to the attention distribution, while smaller λ values ​​will make the allocation more uniform.

[0236] For n cameras, we define the following objective function:

[0237]

[0238] Frame rate f∈{20, 25, 30, 50, 60};

[0239] Quality factor q∈[0.2, 1];

[0240] f(D delay，i ) is a nonlinear penalty function for delay:

[0241]

[0242] Among them, B i is the bit rate of the i-th video actually measured;

[0243] D delay，i : Processing delay of the i-th video (milliseconds);

[0244] R rsholution，i : The resolution of the i-th video;

[0245] D max_delay : Maximum allowed processing delay;

[0246] M max_memory : Maximum allowed memory usage.

[0247] In order to come up with a suitable range of values ​​for the λ parameter and a recommended default value, we need to consider several factors:

[0248] The role of λ: λ balances the two goals of maximizing entropy (even distribution of resources) and minimizing KL divergence (similar resource allocation to attention distribution).

[0249] The magnitude of entropy and KL divergence: These two quantities usually have different orders of magnitude, which affects the choice of λ.

[0250] Specific requirements of the system: Different application scenarios may require different degrees of uniformity and similarity.

[0251] Taking these factors into consideration, the following method is recommended to determine the value range and default value of λ:

[0252] (1) Recommended value range:

[0253] Based on experience, the value range of λ can be set to 0.1 to 0.9, that is, 0.1≤λ≤0.9;

[0254] Default value: As a starting point, the default value of λ can be set to 0.5. This value strikes a balance between maximizing entropy and minimizing KL divergence, that is, the default λ = 0.5.

[0255] For highly dynamic scenes (where the attention distribution changes frequently), a smaller λ value (such as 0.3-0.4) may be required to maintain a certain uniformity. For relatively static scenes, a larger λ value (such as 0.6-0.7) can be used to better match the attention distribution.

[0256] like Figure 5 As shown in FIG. 1 , the detailed steps of using the timestamp information in the RTSP / RTP protocol to estimate the network transmission delay in this application include:

[0257] (1) Preparation

[0258] a. Environment settings

[0259] Install necessary libraries such as FFmpeg and related development tools on your Windows or Linux system.

[0260] Make sure the RTSP stream address and access rights are available.

[0261] b. Apply RTSP / RTP protocol

[0262] RTSP (Real Time Streaming Protocol) is used to control the streaming media server.

[0263] RTP (Real-time Transport Protocol) is used to transmit real-time data.

[0264] RTP contains timestamp information, which is the key to our latency estimation.

[0265] (2) Connect to RTSP stream

[0266] a. Establish a connection

[0267] Use an RTSP URL to connect to the camera's video stream.

[0268] Initializes the network connection and necessary buffers.

[0269] b. Negotiate media format

[0270] Negotiate video encoding, resolution and other parameters with the server.

[0271] Make sure the client can correctly parse the received packets.

[0272] (3) Receiving and parsing RTP packets

[0273] a. Receive data packets

[0274] Continue to receive RTP packets from the network.

[0275] Each packet typically contains a video frame or a portion of a frame.

[0276] b. Parsing RTP header

[0277] Extract timestamp information from each RTP packet.

[0278] RTP timestamps are usually in units of a specific clock frequency (such as 90kHz).

[0279] (4) Timestamp processing

[0280] a. Timestamp conversion

[0281] Convert RTP timestamp to real time (seconds or milliseconds).

[0282] Conversion formula: actual time = RTP timestamp / clock frequency

[0283] b. System time record

[0284] The current system time is recorded when each RTP packet is received.

[0285] Use a high-precision timer to get millisecond accuracy or better.

[0286] (5) Estimating clock offset (handling time asynchrony issues)

[0287] a. Initial sampling

[0288] Before starting the main measurement loop, a series of RTP timestamps and the corresponding system time are collected.

[0289] It is recommended to collect 100-1000 samples, depending on the required accuracy and the time available.

[0290] b. Calculate the offset

[0291] For each sample, the difference between the system time and the RTP timestamp is calculated.

[0292] Use the median of these differences as the estimated clock offset.

[0293] The median is more resistant to outliers than the mean.

[0294] (6) Network delay calculation

[0295] a. Setting a benchmark

[0296] Record the RTP timestamp and system time of the first received packet as a reference point.

[0297] b. Continuous measurement

[0298] For each subsequent packet:

[0299] Calculate the RTP timestamp difference relative to the reference point.

[0300] Calculates the difference in system time relative to a reference point.

[0301] Network delay = (system time difference - RTP timestamp difference - clock offset)

[0302] c.Unit conversion

[0303] Make sure all time calculations are converted to the same units (such as milliseconds).

[0304] (7) Data processing and analysis

[0305] a. Sliding window average

[0306] A sliding window of the most recent N measurements is used to smooth the delay estimate.

[0307] This helps reduce the impact of momentary fluctuations.

[0308] b. Outlier Detection

[0309] Implement a simple outlier detection mechanism to ignore obviously unreasonable delay values.

[0310] Statistical methods such as the 3-sigma rule or the interquartile range can be used.

[0311] c. Periodic reporting

[0312] The average network latency is calculated and reported periodically (e.g., every second).

[0313] Statistics such as minimum, maximum, and standard deviation can be reported simultaneously.

[0314] (8) Continuous monitoring and adjustment

[0315] a. Long-term operation considerations

[0316] Implement a mechanism to re-estimate the clock offset periodically, since long runs may cause the offset to change.

[0317] b. Network condition adaptation

[0318] Dynamically adjust measurement parameters, such as sampling rate, based on changing network conditions.

[0319] c. Error handling

[0320] Implement a robust error handling mechanism to cope with network outages, packet loss, etc.

[0321] (9) Result display and storage

[0322] a. Real-time display

[0323] Implement a simple user interface to display estimated network latency in real time.

[0324] b. Data Recording

[0325] Save measurement results to a log file or database for later analysis.

[0326] c. Alarm mechanism

[0327] Set thresholds to trigger alerts when network latency exceeds a certain value.

[0328] The constraint problem solved in this application adopts the benchmark algorithm greedy heuristic search algorithm, with R resolution，i , f i ,q i As an unknown number, B i It is determined by the following expression: logB=δ·log(R)+αlogf+γlogq+ζ, i=1, 2, 3, . . . , n.

[0329] Resolution

[0330] Frame Rate

[0331] Compression level or its quality parameter (e.g., the inverse of the quantization parameter) q

[0332]

[0333] f(D dellay，i ) is a nonlinear penalty function for delay:

[0334]

[0335] In the initialization phase, the following expression is used

[0336] logB=δ·log(R)+αlogf+γlogq+log(k·C) (7);

[0337] Through at least 4 sets of data, δ, α, γ, ζ,

[0338] logB=δ·log(R)+αlog+γlogq+ζ (8).

[0339] like Figure 6 As shown, in some embodiments, the present application uses an improved genetic algorithm combined with local search to solve the constraint problem, and the algorithm steps are as follows:

[0340] (1) Initialization

[0341] a. Parameter settings

[0342] Specifically, it includes: defining the population size N, setting the maximum number of iterations G_max, determining the number of elite individuals E, setting the tournament size T, and initializing the mutation rate μ_0.

[0343] b. Population initialization

[0344] N individuals are randomly generated, each of which contains the configuration of n cameras. Each camera configuration contains three parameters: resolution R, frame rate f, and quality factor q. The parameter values ​​are randomly selected from a predefined discrete set.

[0345] (2) Fitness evaluation

[0346] a. Bandwidth calculation

[0347] The bandwidth requirement for each camera configuration is calculated using a log-linear model: log B = δ·log(R)+α·log(f)+γ·log(q)+ζ;

[0348] b. Objective function calculation

[0349] Calculate entropy (uniformity measure): -Σ(B_i / B_total*log(B_i / B_total));

[0350] Calculate KL divergence (similarity to attention distribution): λ*Σ(P_i*log(B_i / B_total))-λ*Σ(P_i*log(P_i));

[0351] Estimate latency and calculate latency penalties.

[0352] c. Fitness calculation

[0353] Combining the above three indicators, the fitness value of each individual is calculated.

[0354] (3) Select operation

[0355] Using the tournament selection strategy, T individuals are randomly selected from the population, and the individual with the highest fitness is selected as the parent.

[0356] (4) Crossover operation

[0357] Using the single-point crossover method, a crossover point is randomly selected and the gene sequences of the two parents after that point are exchanged.

[0358] (5) Mutation Operation

[0359] For each gene position, a mutation is performed with probability μ. When mutation occurs, a parameter (R, f or q) is randomly selected and a new value is randomly selected from its discrete value set.

[0360] (6) Local Search

[0361] A local search is performed on each newly generated offspring, traversing all possible discrete value combinations of each camera configuration, and if a better configuration is found, the individual is updated.

[0362] (7) Elite Retention Strategy

[0363] Keep the E individuals with the highest fitness in the current population and copy these elite individuals directly to the next generation population.

[0364] (8) Adaptive mutation and diversity maintenance

[0365] a. Diversity calculation

[0366] Calculate diversity metrics for the population, such as average genetic distance.

[0367] b. Adaptive mutation rate

[0368] The mutation rate is dynamically adjusted according to the current generation and population diversity.

[0369] (9) Termination condition check

[0370] Check whether the maximum number of iterations G_max is reached, or check whether other termination conditions are met, such as fitness convergence.

[0371] (10) Result output

[0372] Output the optimal individual and its corresponding camera configuration, calculate and display various indicators of the optimal solution (entropy, KL divergence, delay, etc.).

[0373] In this application, the above algorithm can also be optimized from the following aspects:

[0374] Parameter tuning: It is recommended to use methods such as grid search or Bayesian optimization to optimize algorithm parameters.

[0375] Parallel computing: Consider using parallel computing techniques to speed up the fitness evaluation and local search process.

[0376] Initialization strategy: Consider using heuristic methods to generate part of the initial population to provide a better starting point.

[0377] Constraint handling: Other constraint handling techniques can be explored, such as feasibility rules or repair operations.

[0378] Multiple runs: Due to the random nature of the algorithm, it is recommended to run it multiple times and select the best result.

[0379] Convergence analysis: Monitor population diversity and the evolution of the optimal solution during implementation so as to adjust the strategy or terminate the algorithm in a timely manner.

[0380] In this way, the algorithm is able to effectively search for the optimal bandwidth allocation scheme in a complex discrete solution space while balancing multiple optimization objectives and constraints.

[0381] In this application, we define the reward multiplication scoring function required for reinforcement learning to score the overall effect after a period of time (for example, after 1 minute). For n cameras, we define the following objective function:

[0382]

[0383] In this application, a neural network for prediction and decision scheduling of time series data will be designed, with the goal of making the decision result achieve the highest performance after 1 minute. The input of the neural network is (F,n,K), where K=4, which is used to represent the measured R,f,q, and B, n represents n cameras, and F represents F frames sampled in the past. The output dimension of the neural network is: (n,4), which represents the optimized R,f,q and predicted B.

[0384] Each structure and layer of this neural network is as follows:

[0385] (1) Prediction Module

[0386] a. Temporal Encoder (LSTM / GRU)

[0387] Structure: multi-layer bidirectional LSTM or GRU;

[0388] Input dimension: (F,n,K), F is the time step, n is the number of cameras, K is the number of features (R,f,q,B);

[0389] Output dimension: (n,H), where H is the hidden state dimension.

[0390] Specifically, the number of layers is two to three. Increasing the number of layers can capture more complex time dependencies. The value range of the hidden state dimension H is 64-256, which can be adjusted according to the complexity of the problem. The dropout rate is 0.1-0.3, which can prevent overfitting.

[0391] The temporal encoder can capture the long-term dependencies of the time series and encode historical information. LSTM / GRU can effectively process long sequence data and avoid the gradient disappearance / explosion problem.

[0392] b. Attention Mechanism

[0393] Structure: Multi-head self-attention layer;

[0394] Input / output dimensions: (n,H);

[0395] Specifically, the number of attention heads ranges from 4 to 8. Increasing the number of heads can focus on different feature subspaces; the dimension of each head is H / number of heads. The self-attention mechanism can capture the mutual influence and relationship between different cameras. The self-attention mechanism can dynamically weigh the importance of different cameras and improve the expressiveness of the model.

[0396] c. Prediction Head

[0397] Structure: Multilayer Perceptron (MLP);

[0398] Input dimension: (n,H);

[0399] Output dimension: (n,2), predicting P and D_delay respectively.

[0400] Specifically, the number of hidden layers can be one or two, and the number of neurons in the hidden layer is in descending order from H to 2, such as [H,H / 2,2]. The activation function can be ReLU or LeakyReLU. The prediction head can map the encoded features to specific prediction values, and MLP can learn complex nonlinear mappings, which is suitable for the final prediction task.

[0401] (2) Decision-making module

[0402] a. State Encoder

[0403] Structure: fully connected layer;

[0404] Input dimension: (n,H+2), including temporal encoding and prediction results;

[0405] Output dimension: (n,H'), where H' is the new hidden state dimension;

[0406] Specifically, H' is usually close to H, such as H'=H or H'=H*1.5, and the activation function can be ReLU or Tanh. The state encoder can fuse historical information and prediction results, provide comprehensive features for decision-making, and integrate information from different sources into a unified representation space to facilitate subsequent decision-making.

[0407] b. Decision network

[0408] Structure: Multilayer Perceptron (MLP);

[0409] Input dimension: (n,H');

[0410] Output dimension: (n,3), corresponding to the discrete selection probability of R,f,q.

[0411] Specifically, the number of hidden layers is two to three, the number of neurons in the hidden layer decreases, such as [H', H' / 2, H' / 4, 3], the activation function is ReLU or LeakyReLU, and the last layer uses Softmax. The decision network generates decision probabilities based on the encoded state. MLP can learn complex decision rules, and Softmax output ensures that the probability sum is 1.

[0412] c. Gumbel-Softmax layer

[0413] Input / output dimensions: (n,3);

[0414] Specifically, the temperature parameter τ starts from 1 and gradually decreases to close to 0 as the training progresses. The Gumbel-Softmax layer implements a differentiable approximation of discrete decisions during the training process, making the discrete decision process differentiable and facilitating back-propagation and end-to-end training.

[0415] The design of the neural network in this application fully considers the timing of the problem, the discrete decision-making characteristics, and the needs of multi-objective optimization. By combining timing prediction and decision optimization, and adopting various advanced training techniques, the network can effectively learn complex bandwidth allocation strategies to maximize the performance index S for the next 1 minute. The design of each component has its specific role, and together they form a powerful and flexible end-to-end optimization framework.

[0416] In some embodiments, the steps of changing the resource allocation parameters of the camera (modifying the resolution (R), frame rate (f) and compression level / quality parameter (q) of the IPC camera on the client) are as follows:

[0417] 1. Network connection establishment

[0418] a. Make sure the client device and the camera are on the same network or can access each other.

[0419] b. Obtain the camera's IP address, port number, and any necessary network configuration information.

[0420] c. Use an appropriate network protocol (such as TCP / IP) to establish a connection with the camera.

[0421] 2. Identity Authentication

[0422] a. Prepare the camera's access credentials (user name and password).

[0423] b. Perform identity authentication according to the authentication methods supported by the camera, including:

[0424] Basic Authentication: Send base64 encoded username and password.

[0425] Digest authentication: Generates an authentication response using the nonce value and credentials provided by the camera.

[0426] Token authentication: First obtain an access token, and then use the token in subsequent requests.

[0427] c. Handle possible authentication errors, such as invalid credentials, account lockout, etc.

[0428] 3. Parameter modification

[0429] For each parameter (R, f, q), perform the following steps:

[0430] a. Resolution (R) modification:

[0431] Query the list of resolutions supported by the camera and select the target resolution (such as 1920x1080). Construct an appropriate command or request, specifying the new width and height. Send the request to the camera, wait for and parse the camera's response to confirm whether the change was successful.

[0432] b. Frame rate (f) modification:

[0433] Confirm the frame rate range supported by the camera, select the target frame rate (such as 30fps), construct a frame rate modification request, send the request to the camera, and verify the camera's response to ensure that the frame rate has been updated.

[0434] c.Compression level / quality parameter (q) modification:

[0435] Understand the range and meaning of the camera's quality parameter (which may be a numeric value from 0 to 100 or a specific enumeration value) and select an appropriate quality value. Create a quality parameter modification request and send the request to the camera. Check the camera's confirmation response.

[0436] 4. Change verification

[0437] a. Immediately after applying each parameter change, query the camera for the current settings.

[0438] b. Compare the returned value with the expected setting to ensure that the changes were applied correctly.

[0439] c. If some changes do not take effect, consider retrying or notifying the user.

[0440] 5. Error handling and exception management

[0441] a. Implement network timeout handling and retry when the connection or request times out.

[0442] b. Capture and handle possible network errors, such as disconnection, packet loss, etc.

[0443] c. Parse the error code and message returned by the camera and take appropriate action (such as retry, roll back changes, or notify the user).

[0444] d. Handle invalid parameter settings, such as requesting an unsupported resolution or frame rate.

[0445] 6. Save configuration

[0446] a. Check if the camera requires additional steps to make the changes permanent.

[0447] b. If necessary, send a configuration save command.

[0448] c. Confirm that the configuration has been saved successfully. You may need to wait for the camera to complete the saving process.

[0449] 7. Flow Management

[0450] a. Note that some parameter changes may cause the current video stream to be interrupted.

[0451] b. Be prepared to re-establish the logic for video streaming.

[0452] c. After parameter changes, it may be necessary to reinitialize the RTSP stream or other video delivery mechanism.

[0453] By following these detailed steps, client applications can safely and efficiently modify key video parameters of IPC cameras while ensuring operational reliability and system stability.

[0454] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the attention-based multi-channel video low-latency adaptive method as described in the first aspect of the present invention.

[0455] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0456] The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface memory can be a disk memory or a tape memory.

[0457] The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The computer-readable storage medium described in the embodiments of the present invention is intended to include these and any other suitable types of memory.

[0458] like Figure 7 As shown, in a third aspect, the present invention provides an electronic device 10, comprising a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, the attention-based multi-channel video low-latency adaptive method as described in the first aspect of the present invention is implemented.

[0459] In some embodiments, the processor can be implemented by software, hardware, firmware or a combination thereof, and can use circuits, single or multiple application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), central processing units (CPU), controllers, microcontrollers, and at least one of microprocessors, so that the processor can execute some or all of the steps in the message push method described in the various embodiments of the present application, or any combination of the steps therein.

[0460] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A low-latency adaptive method for multi-channel videos based on attention, characterized in that: The method comprises the following steps: Initialize system resources; Receive video stream data input from multiple cameras; Monitor and analyze the attention quality and network transmission effect of the current video stream data of each camera, and solve the constraint problem according to the attention quality and network transmission effect of the video stream data of each camera to optimize the resource allocation parameters; The system resources of the video stream data of each camera are adjusted according to the optimized resource allocation parameters.

2. The attention-based multi-channel video low-delay adaptive method according to claim 1, characterized in that: The initialization system resources include: Load system configuration parameters, initialize network connection and communication protocol, and set initial encoding parameters; Establish connections with multiple cameras and configure the initial encoding parameters of each camera.

3. The attention-based multi-channel video low-delay adaptive method as claimed in claim 1, characterized in that: The video stream data includes multiple video frame images, and monitoring and analyzing the attention quality and network transmission effect of the current video stream data of each camera includes: Use the first neural network model to process the decoded video stream data, generate an attention map of each frame of the video stream image, and obtain the area of ​​the attention region through binarization processing; It also measures the real-time bandwidth usage of each video stream data, calculates the end-to-end transmission delay, and records the network jitter and packet loss rate.

4. The attention-based multi-channel video low-delay adaptive method as claimed in claim 3, characterized in that: The decoded video stream data is processed using the first neural network model to generate an attention map for each frame of the video stream image, including: The attention density function A(x, y) is used to represent the attention value of each pixel (x, y) on the video frame image. The calculation formula is as follows: Get the attention map of each frame of the video stream image; The method further comprises: The pixel values ​​of the attention map of the video stream image are scaled to the range [0, 1], and the mean and standard deviation normalization is applied to the color channel of each pixel.

5. The attention-based multi-channel video low-delay adaptive method as claimed in claim 3, characterized in that: The area of ​​the attention region obtained by binarization includes: Determine whether the attention value of each pixel point in the video frame image is greater than the preset attention threshold. If so, assign the pixel point a value of 1, otherwise assign the pixel point a value of 0, where 1 represents a high attention area and 0 represents a low attention area; Apply dilation or erosion operations to smooth the binarization result, identify and mark continuous high-attention areas, calculate the sum of all pixel points representing the high-attention areas, and obtain the area of ​​the attention area.

6. The attention-based multi-channel video low-delay adaptive method as claimed in claim 1, characterized in that: Solving constraint problems based on the attention quality of the video stream data of each camera and the network transmission effect to optimize resource allocation parameters includes: Input the attention quality and network transmission effect of the video stream data of each camera into the trained second neural network model, predict the attention quality and network transmission effect of the current video stream data in a subsequent preset time period based on the trained second neural network model, and output optimized resource allocation parameters based on the predicted attention quality and network transmission effect; The second neural network model is trained according to the following method: Acquire sample historical data, where the sample historical data includes attention quality and network transmission effect corresponding to the sample video stream data; The sample historical data is input into the second neural network model to be trained, and the model parameters are iteratively updated according to the prediction results to obtain the trained second neural network model.

7. The attention-based multi-channel video low-delay adaptive method as claimed in claim 1, characterized in that: The constraint problem is designed as follows: Define the following function: Entropy function: Similarity constraint function: Total resource constraint function: Delay constraint function: Where n is the number of cameras, B i is the network transmission resource allocated to the i-th camera, P i is the proportion of the attention area of ​​the i-th camera, B total is the total number of available network transmission resources, L max is the maximum allowed delay, k is a constant; The objective function of the constraint problem is designed as follows: J=H(B)-λD KL (P|B); For all i, For all i, B i ≥0; Among them, λ is a trade-off parameter used to adjust the balance between entropy maximization and distribution similarity; Expand this objective function: The last item is a constant; The solution to the constraint problem includes: The constrained problem is solved by the Lagrange multiplier method.

8. The attention-based multi-channel video low-delay adaptive method as claimed in claim 1, characterized in that: The resource allocation parameters include any one or more of the resolution, frame rate and compression level of the video stream data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the attention-based multi-channel video low-latency adaptive method as described in any one of claim 18 is implemented.

10. An electronic device, characterized in that: It includes a processor and a storage medium, on which a computer program is stored. When the computer program is executed by the processor, the attention-based multi-channel video low-latency adaptive method as described in any one of claim 18 is implemented.

Images